How to use Boost Spirit with variant with more than 20 types?

Question

I'm parsing a quite complex grammar with Boost Spirit and I'm facing a problem with a variant that have more than 20 types (21 here):

namespace eddic { namespace ast {

typedef boost::mpl::vector<
            Integer,
            IntegerSuffix,
            Float,
            Litteral,
            VariableValue,
            DereferenceValue,
            Expression,
            Unary,
            Null,
            True,
            False,
            ArrayValue,
            FunctionCall,
            MemberFunctionCall,
            Cast,
            BuiltinOperator,
            Assignment,
            SuffixOperation,
            PrefixOperation,
            Ternary
        > types_initial;

typedef boost::mpl::push_back<types_initial, New>::type types;
typedef boost::make_variant_over<types>::type Value;

}}

Boost Spirit does not recognize the last type (eddic::ast::New) added with push_back. When I parse something that have this element, it fails with this error:

eddic: /usr/include/boost/variant/detail/visitation_impl.hpp:264: typename Visitor::result_type boost::detail::variant::visitation_impl(int, int, Visitor&, VoidPtrCV, mpl_::false_, NoBackupFlag, Which*, step0*) [with Which = mpl_::int_<0>; step0 = boost::detail::variant::visitation_impl_step, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0l>, boost::mpl::v_iter, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 21l> >; Visitor = boost::variant, boost::mpl::vector, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Null, eddic::ast::True, eddic::ast::False, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred >, 0> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>::assigner; VoidPtrCV = const void*; NoBackupFlag = boost::variant, boost::mpl::vector, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Null, eddic::ast::True, eddic::ast::False, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred >, 0> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>::has_fallback_type_; typename Visitor::result_type = void; mpl_::false_ = mpl_::bool_]: Assertion `!"Boost.Variant internal error: 'which' out of range."' failed.

If I swap two elements (Ternary and New for example), New is correctly recognized, but not Ternary. It is only the last element that fails.

I already tried using push_front or vector21, but it doesn't change anything, it's always the last element that have problem. In my opinion it comes from the fact that Spirit uses a variant internally before copying it to my variant_over type.

Is there a workaround to this problem ?

I could probably reduce the number to 20, but the problem is that I will certainly have more than that in the future.

Thanks a lot for any idea

Any reason not to mix with classic OO? Roll all operations in a single AST type (polymorphically), all function calls likewise etc. It almost looks as though you are including details at the tokenizer level in your ast (IntegerSuffix - really?) — sehe, Jul 24 '12 at 20:48
I'm not sure I do understand. You mean an inheritance tree ? Like Value being the base class of Integer ? — Baptiste Wicht, Jul 24 '12 at 22:27
I think you get the idea. The thing is, having a humongous variant is pretty much a code smell in my experience. — sehe, Jul 25 '12 at 00:23
@BaptisteWicht, How did your AST end up? I am designing my AST at the moment and will soon be experiencing the very same issue. — PeteUK, Feb 14 '13 at 12:56
@PeteUK I've been able to simplify it a bit (https://github.com/wichtounet/eddic/blob/develop/include/ast/Value.hpp). I still think it can be improved, especially regarding the constants and literals that could be merged probabled. However, I don't want to make the processing of the AST more complex. — Baptiste Wicht, Feb 14 '13 at 23:39
@BaptisteWicht, you still have quite a few datatypes in there I see. I've created a similar [question](http://stackoverflow.com/questions/14878938/boost-spirit-and-abstract-syntax-tree-design) on AST design with Boost.Spirit. I think it would be messy to deal with the nodes polymorphically through an abstract base. Would make the visitation of the AST more complex as you say. The AST-related examples that come with Spirit (calc, mini_c, mini_xml) don't seem to have a group node that can contain a wide variety of nodes, so I don't know what to do. — PeteUK, Feb 15 '13 at 00:15

score 5 · Accepted Answer · answered Jul 24 '12 at 12:30

5

Define BOOST_MPL_LIMIT_VECTOR_SIZE to whatever new limit you want, but going this high is usually a hint at design problems so it might be worth to give it some consideration.

answered Jul 24 '12 at 12:30

Ylisar

4,293
21
27

I just tried it, but it didn't fixed the problem. IMO, the problem does not comes from the mpl::vector. I also tried boost::mpl::vector21 and the problem is the same. – Baptiste Wicht Jul 24 '12 at 12:55
1

It would seem it's related to `BOOST_MPL_LIMIT_LIST_SIZE` instead, perhaps you'll need to increase both to a new limit. – Ylisar Jul 24 '12 at 13:00
1

Sorry, but it would seem you also need to define `BOOST_MPL_CFG_NO_PREPROCESSED_HEADERS`, tried it out in a playground and this seems to seal the deal :P . One also needs to stick to increments of 10. – Ylisar Jul 24 '12 at 13:15
I did it ;) Because without that I add problem with other parts of Boost. But that didn't solve the problem. – Baptiste Wicht Jul 24 '12 at 13:16
1

That's weird, because the assert fires due to `BOOST_VARIANT_VISITATION_UNROLLING_LIMIT` being to low, which is set to `BOOST_VARIANT_LIMIT_TYPES`, which in turn is set to `BOOST_MPL_LIMIT_LIST_SIZE`. Are you certain that your defines are done before any includes which reasonably could resolve to something inside `mpl`? – Ylisar Jul 24 '12 at 14:00
You were right, I had forgotten an include in my lexer :( Thanks a lot. The only problem is that now my compile times are much worse than before but at least it works. – Baptiste Wicht Jul 24 '12 at 14:18
I cut up the variadic macro into a series of small macros, it sucked a lot but works – Carbon Dec 04 '17 at 22:52

How to use Boost Spirit with variant with more than 20 types?

1 Answers1

Linked