Why does my simple cpp-netlib program take so long to compile?

Question

I recently started learning cpp-netlib and am testing out one of netlibs example programs

#include <boost/network/protocol/http/client.hpp>
#include <iostream>
int main()
{
    using namespace boost::network;
    http::client client;
    http::client::request request("http://www.boost.org");
    request << header("Connection", "close");
    http::client::response response = client.get(request);
    std::cout << body(response) << std::endl;
    return 0;
}

After many hours of research I found that the proper command I need to use to compile my program is

clang++ -std=c++11 -I /usr/local/Cellar/openssl/1.0.2e/include test.cpp -L /usr/local/Cellar/openssl/1.0.2e/lib -lboost_system-mt -lboost_thread-mt -lcppnetlib-client-connections -lcppnetlib-uri -lcppnetlib-server-parsers -lssl -lcrypto

Here is a link to an older question that I posted detailing how I came about finding everything I needed for this program to compile cpp-Netlib with Boost Mac OSX seg fault when using HTTP Client body

I find that it takes about 15 seconds to compile and was wondering if there was a way for me to speed up this process? Is it really this slow to compile this code or is the linker taking a good amount of time to go and get all these libraries and if so can I speed this up?

MacBook Air (11-inch, Mid 2013), 1.3 GHz Intel Core i5, 4 GB 1600 MHz DDR3, Intel HD Graphics 5000 1536 MB — Jem4687, Dec 05 '15 at 00:35
Wild guess, but `boost` is a big library and there's no telling how many other files/objects this "simple" program is *really* using when `boost` starts to include its own set of headers, and linking everything up. A faster CPU + disk drive might help, due to the amount of code processing + I/O necessary during the build process. If using a regular HDD (not SSD), you should have at least a 7200 RPM drive. — code_dredd, Dec 05 '15 at 00:35
I think MacBook Air's have SSDs don't they? However the CPU isn't really up to the job I don't think. — trojanfoe, Dec 05 '15 at 00:36
@Jem4687: Since yours has SSD, I'd blame the CPU. A 1.3GHz CPU is fairly slow by today's standards. — code_dredd, Dec 05 '15 at 00:38
Run "sudo opensnoop" in another Terminal window, and while that's running, compile your file, and you can see all of the other files that the compiler is loading in while it compiles. — Jeremy Friesner, Dec 05 '15 at 03:56
The absolute minimum system configurations that we give to programmers who must have portable development platforms has twice the performance of your CPU. Your CPU is designed to use as little power as possible -- performance is not a priority. — David Schwartz, Dec 05 '15 at 10:07

sehe · Accepted Answer · 2015-12-07T21:57:15.513

I seem to remember cpp-netlib uses Spirit Qi grammarts for URL parsing e.g.

network/uri/accessors.hpp
network/uri/uri.ipp

In this case the slowdown seems to be the key_value_sequence parser in accessors.hpp

These are very template heavy and take considerable time to compile, depending only slightly on the compiler used (MSVC being worst, in my experience).

You could prevent inclusion of these headers. At the very least only include them in the translation units (cpps) that require it; don't ever make it get caught up in your "common" header dependencies. That would mean the compiler has to recompile that stuff on each iteration (even using precompiled headers the cost would likely be significant).

Depending on your compiler version, these could help:

disable debug information (-g0)
optimize for size (-Os)
define BOOST_SPIRIT_USE_PHOENIX_V3 (default since ~1.58)
define things like FUSION_MAX_VECTOR_SIZE to smaller numbers (default: 10)

Really, if you use c++14 capable clang, I'd be interested in testing a patch to use Spirit X3 instead of Qi.

At least replacing this Qi parser:

#include <boost/spirit/include/qi.hpp>

// ...
namespace details {
template <typename Map>
struct key_value_sequence : spirit::qi::grammar<uri::const_iterator, Map()> {
  typedef typename Map::key_type key_type;
  typedef typename Map::mapped_type mapped_type;
  typedef std::pair<key_type, mapped_type> pair_type;

  key_value_sequence() : key_value_sequence::base_type(query) {
    query = pair >> *((spirit::qi::lit(';') | '&') >> pair);
    pair = key >> -('=' >> value);
    key =
        spirit::qi::char_("a-zA-Z_") >> *spirit::qi::char_("-+.~a-zA-Z_0-9/%");
    value = +spirit::qi::char_("-+.~a-zA-Z_0-9/%");
  }

  spirit::qi::rule<uri::const_iterator, Map()> query;
  spirit::qi::rule<uri::const_iterator, pair_type()> pair;
  spirit::qi::rule<uri::const_iterator, key_type()> key;
  spirit::qi::rule<uri::const_iterator, mapped_type()> value;
};
}  // namespace details

template <class Map>
inline Map &query_map(const uri &uri_, Map &map) {
  const uri::string_type range = uri_.query();
  details::key_value_sequence<Map> parser;
  spirit::qi::parse(boost::begin(range), boost::end(range), parser, map);
  return map;
}

With this X3 variant:

#include <boost/spirit/home/x3.hpp>

// ...
namespace details {
    namespace kvs_parser {
        namespace x3 = boost::spirit::x3;

        static auto const key    = x3::char_("a-zA-Z_") >> *x3::char_("-+.~a-zA-Z_0-9/%");
        static auto const value  = +x3::char_("-+.~a-zA-Z_0-9/%");

        template <typename Map, typename K = typename Map::key_type, typename V = typename Map::mapped_type, typename Pair = std::pair<K, V> >
        static auto const pair   = x3::rule<struct kvs_pair, Pair> {} 
                                = key >> -('=' >> value);

        template <typename Map>
        static auto const query  = pair<Map> >> *((x3::lit(';') | '&') >> pair<Map>);
    }
}  // namespace details

template <class Map>
inline Map &query_map(const uri &uri_, Map &map) {
    const uri::string_type range = uri_.query();
    spirit::x3::parse(boost::begin(range), boost::end(range), details::kvs_parser::query<Map>, map);
    return map;
}

Reduces compilation time from ~8s down to ~5s on my system

BIG FAT WARNING The X3 code is untested (I don't even know what uses it, I just "blindly" translated to X3 to the best of my knowledge)

How would I go about making sure they pieces aren't compiled everytime I change my main program? Could I do this using a makefile? — Jem4687, Dec 05 '15 at 11:44
@Jem4687 that's what all build tools to: they follow up the dependency graph. Your job is to minimize the dependencies. In practice that is simply not including the stuff that hurts in many TUs — sehe, Dec 05 '15 at 11:55

Why does my simple cpp-netlib program take so long to compile?

1 Answers1

Linked