Internal compiler error, while using boost spirit x3

Question

I am currently implementing expressions and the operator hierarchy for my DSL, using boost spirit X3.

I think my parser is semanticaly correct, but when I try to compile it, while compiling, gcc and clang having HUGE memory footprint, compiles for an infinite amount of time, and then quits with "g++: internal compiler error: Killed (program cc1plus)".

I tried to minimize the code arround the expression parser, but its somehow not that trivial

Here is a demo.

Can someone tell me what I am doing wrong here, or is this a bug?

EDIT: I think the problem is somwhere there:

auto const idx = as<ast::Operation>(helper::idxaccess_op > expression > ']');
auto const func = as<ast::Operation>(helper::func_call_op > expression%',' > ')');
auto const data = as<ast::Operation>(helper::access_op > expression);

auto const func_call_expr_def =
    primary_expr >> *(idx|func|data);

if I change (idx|func|data) to (idx|func), it also compiles forever and uses up to 16GB of ram, but gcc is able to compile it, and the parser works how it should work.

EDIT II: Please have a look at the link above there is my example which causes the error.

Answer 1

I have done a bunch of changes on top of your source files. Please check those at http://melpon.org/wandbox/permlink/sY2CQkXiMiLoS1BM

The changes are:

1. Changed the AST. It did not seem correct. Mainly the part adding 'Unary' to the variant.
2. Corrected the grammar. This is based on what I really could make of the grammar from your tests. I may be wrong, I have only tried making the first test case working. Better run a diff tool to compare my changes against yours, especially in 'parser.hpp'

NOTE: If my changes are not correct as per your requirements, the I would suggest you to enable debug 'BOOST_SPIRIT_X3_DEBUG' and trace it. Can't stress enough how much BOOST_SPIRIT_X3_DEBUG is useful in such cases.

parser.hpp

#include <boost/spirit/home/x3.hpp>
#include "ast.hpp"
#include "operators.hpp"
namespace parser{
    namespace x3 = boost::spirit::x3;


    template<typename T>
    auto as = [](auto p) { return x3::rule<struct _, T>{} = as_parser(p); };


    typedef x3::rule<struct multiplicative_expr_class, ast::Expression> multiplicative_expr_type;
    typedef x3::rule<struct expression_class, ast::Expression> expression_type;
    typedef x3::rule<struct primary_expr_class, ast::Operand> primary_expr_type;
    typedef x3::rule<struct func_call_call_class, ast::Expression> func_call_expr_type;
    typedef x3::rule<struct unary_expr_class, ast::Operand> unary_expr_type;



    const primary_expr_type primary_expr = "primary_expr";    
    const func_call_expr_type func_call_expr = "func_call_expr";
    const expression_type expression = "expression";
    const multiplicative_expr_type multiplicative_expr = "multiplicative_expr";
    const unary_expr_type unary_expr = "unary_expr";


    auto const primary_expr_def =
        +(x3::alpha | x3::char_('.'))
        | ( "(" > expression > ")" );

    auto const idx = as<ast::Operation>(helper::idxaccess_op > primary_expr > ']');
    auto const func = as<ast::Operation>(helper::func_call_op > primary_expr%',' > ')');
    auto const data = as<ast::Operation>(helper::access_op > expression);

    auto const func_call_expr_def =
        primary_expr >> *( idx | func | data );

    auto const unary_expr_def =
              func_call_expr
                      | as<ast::Operation>(helper::unary_op > func_call_expr);

    auto const multiplicative_expr_def =
        primary_expr >>  *(idx | func);

    auto const expression_def = multiplicative_expr_def;


BOOST_SPIRIT_DEFINE(primary_expr,
                    func_call_expr,
                    multiplicative_expr,
                    unary_expr,
                    expression);

}

ast.hpp

#include <boost/spirit/home/x3.hpp>
#include <boost/spirit/home/x3/support/ast/variant.hpp>
#include <boost/fusion/include/adapt_struct.hpp>

#include <vector>

namespace ast{
    namespace x3 = boost::spirit::x3;

    enum class operator_t
    {
      _eq_,       // ==
      _ne_,       // !=
      _lt_,       // <
      _gt_,       // >
      _le_,       // <=
      _ge_,       // >=
      _add_,      // +
      _sub_,      // -
      _mul_,      // *
      _div_,      // /
      _mod_,      // %
      _pos_,      // unary +
      _neg_,      // unary -
      _not_,      // unary !
      _size_,     // unary #
      _bitnot_,   // unary ~
      _bitand_,   // &
      _bitor_,    // |
      _bitxor_,   // ^
      _shl_,      // <<
      _shr_,      // >>
      _and_,      // &&
      _or_,       // ||
      _idx_,      // []
      _apply_,    // ()
      _access_    // .
    };

    struct nil{};
    struct Expression;

    typedef x3::variant<
            nil,
            std::string,
            x3::forward_ast<Expression>
            //std::vector<Expression> //adding this as an operand for function parameter
    > Operand;

    struct Operation {
        operator_t operator_;
        Operand operand_;
    };

    struct Expression {
        Operand first_;
        std::vector<Operation> rest_;
    };
}
BOOST_FUSION_ADAPT_STRUCT(ast::Operation, operator_, operand_)
BOOST_FUSION_ADAPT_STRUCT(ast::Expression, first_, rest_)

Answer 2

The problem has something to do with the template instantiation depth. To avoid the internal compiler error, and to get acceptable compiletimes, for my parser I had to implement something like a template firewall, which is implemented as a custom parser, which calls a non templated function in the parse expression.

struct OptimizedExpressionParser : x3::parser<OptimizedExpressionParser> { 
    using attribute_type = ast::Expression;
    static bool const has_attribute = true;

    //parse fnction, which calls "non-templated"-firewall function
    template <typename Iter, typename Ctx, typename Attribute>
    bool parse(Iter& iFirst, const Iter& iLast, const Ctx& iCtx, x3::unused_type, Attribute& oAttr) const {
      ast::Expression a;
      return parse(iFirst, iLast, iCtx, x3::unused, a);
    }

    //parse fnction, which calls "non-templated"-firewall function
    template <typename Iter, typename Ctx>
    bool parse(Iter& iFirst, const Iter& iLast, const Ctx& iCtx, x3::unused_type, ast::Expression& oAttr) const {
      if (iFirst != iLast) {
        return parse_expression(iFirst, iLast, oAttr);
      }
      return false;
    }
   private:

   //"non-template"- parse function
   //of cause this is a template function, but the parser isnt a template argument
   template <typename Iter>
   bool parse_expression(Iter& iFirst, const Iter& iLast, ast::Expression& oAst) const {
      return x3::parse(iFirst, iLast, expression_impl, oAst);

  } 
  };

in this code expression_impl is the old "heavy" expression parser, the new "firewalled" expression parser is this:

auto const expression = OptimizedExpressionParser{};

wherever I would like to use an expression in another parser, or for parsing I now use the expression object from my OptimizedExpressionParser parser. this decrease ram usage, compiletimes and also the size of the resulting binary(which was about 1.6 Gb without the template firewall) To see how this works out with my example have a look here.

To be honest, I would have never solved this problem on my own, the idea and most of the code comes from here, I only changed it a bit to work out with my given example.