X3, how to populate a more complex AST?

Question

Trying to generate an AST like the employee example that has more than just the employee. In my current mindset, the RExpressions example isn't helping me. The example I have doesn't compile, but I went as far as I understood in adding teams, departments, and corporations to the employee example.

My problem is in understanding how to add the different structs into a variant and add the variant to phrase_parse, if that's the idea.

In this example, there can be several of the same lines following each other. So wondering if that's what requires making an AST recursive.

#include <boost/config/warning_disable.hpp>       
#include <boost/spirit/home/x3.hpp>
#include <boost/fusion/include/adapt_struct.hpp>   
#include <boost/fusion/include/io.hpp>

#include <iostream>
#include <string>
#include <complex>

namespace client { namespace ast
{
    struct employee;
    struct team;
    struct department;
    struct corporation;

    typedef x3::variant<
        employee,
        team,
        department,
        corporation
     > var_types;

    struct employee
    {
        int age;
        std::string surname;
        std::string forename;
        double salary;
    };

    struct team 
    {
        std::string name;
        int num_employees;  
    };

    struct department
    { 
        std::string name;
        int num_teams;  
        double budget;
    };

    struct corporation
    { 
        std::string name;
        int num_depts;
    };
}}

BOOST_FUSION_ADAPT_STRUCT(client::ast::employee, age, surname, forename, salary)
BOOST_FUSION_ADAPT_STRUCT(client::ast::team, name, num_employees)
BOOST_FUSION_ADAPT_STRUCT(client::ast::department, name, num_teams, budget)
BOOST_FUSION_ADAPT_STRUCT(client::ast::corporation, name, num_depts)

namespace client
{
    namespace parser
    {
        namespace x3 = boost::spirit::x3;
        namespace ascii = boost::spirit::x3::ascii;

        using x3::int_;
        using x3::lit;
        using x3::double_;
        using x3::lexeme;
        using ascii::char_;
        using x3::eol;
        using x3::blank;
        using x3::skip;

        x3::rule<class employee, ast::employee> const employee = "employee";
        auto const employee_def = int_ >> *(char_ - eol) >> *(char_ - eol) >> double_;
        BOOST_SPIRIT_DEFINE(employee);

        x3::rule<class team, ast::team> const team = "team";
        auto const team_def = *(char_ - eol) >> int_;
        BOOST_SPIRIT_DEFINE(team);

        x3::rule<class department, ast::department> const department = "department"; 
        auto const department_def = *(char_ - eol) >> int_ >> double_;
        BOOST_SPIRIT_DEFINE(department);

        x3::rule<class corporation, ast::corporation> const corporation = "corporation";
        auto const corporation_def = *(char_ - eol) >> int_;
        BOOST_SPIRIT_DEFINE(corporation);

        auto pemployee = skip(blank) [
            *(employee >> eol)
        ];  

        auto pteam = skip(blank) [
            *(team >> eol)
        ];  

        auto pdepartment = skip(blank) [
            *(department >> eol)
        ];  

        auto pcorporation = skip(blank) [
            *(corporation >> eol)
        ];  

        auto const input = pemployee >> pteam >> pdepartment >> pcorporation;
    }
}

int main()
{
    namespace x3 = boost::spirit::x3;
    using boost::spirit::x3::ascii::blank;
    using x3::char_;
    using client::parser::input;
    using client::ast::var_types;

    var_types types;

    std::istringstream iss("30 joe smith 100000.00\n20 mary jo 100000.00\n25 john doe 100000.00\nteamA 1\nteamB 1\nteamC 1\naccounting 1 100000.00\nengineering 2 200000.00\nAcmeCorp 3\n");

    boost::spirit::istream_iterator iter(iss >> std::noskipws), eof;

    bool ok = phrase_parse(iter, eof, input, x3::char_(' '), types);

    std::cout << "ok = " << ok << std::endl;

    return 0;
}

How do you think you can assign a whole bunch of employees, teams, departments and then corporations to a single `var_types`? What are you trying to achieve that the code clearly doesn't show — sehe, Jul 16 '17 at 11:17

sehe · Accepted Answer · 2017-07-16T12:04:15.403

Except for missing includes and namespace aliases, you should probably just make sure the bound attribute ref allows multiple entries, since the grammar matches multiple employees, teams, departments and corporations...:

std::vector<var_types> types;

Makes it compile for me.

DIRECT FIX

Assuming your ast is what you want (why?! it doesn't reflect the grammar) here's a working example.

Note

you need to use lexemes and limit the matching for strings to non-blanks otherwise you will always match till the end of the line
departments need to be ordered before teams, or you get "team" matched instead of departments
using the variant is a bit obscure, as you can see from the printing loop
as long as your rules are not recursive, no need for BOOST_SPIRIT_DEFINE
using phrase_parse is bogus because you override the skipper anyways. I like to have the skipper in the grammar definition for reasons of correctness and ease of use anyways.

Live On Coliru

#include <iostream>
#define BOOST_SPIRIT_X3_DEBUG
#include <boost/spirit/home/x3.hpp>
#include <boost/spirit/home/x3/support/ast/variant.hpp>
#include <boost/spirit/include/support_istream_iterator.hpp>
#include <boost/fusion/include/adapt_struct.hpp>
#include <boost/fusion/include/io.hpp>

namespace x3 = boost::spirit::x3;

namespace client { namespace ast {
    struct employee;
    struct team;
    struct department;
    struct corporation;

    typedef x3::variant<
        employee,
        team,
        department,
        corporation
    > var_types;

    struct employee
    {
        int age;
        std::string surname;
        std::string forename;
        double salary;
    };

    struct team
    {
        std::string name;
        int num_employees;
    };

    struct department
    {
        std::string name;
        int num_teams;
        double budget;
    };

    struct corporation
    {
        std::string name;
        int num_depts;
    };
}}

BOOST_FUSION_ADAPT_STRUCT(client::ast::employee, age, surname, forename, salary)
BOOST_FUSION_ADAPT_STRUCT(client::ast::team, name, num_employees)
BOOST_FUSION_ADAPT_STRUCT(client::ast::department, name, num_teams, budget)
BOOST_FUSION_ADAPT_STRUCT(client::ast::corporation, name, num_depts)

namespace client
{
    namespace parser
    {
        namespace ascii = boost::spirit::x3::ascii;

        using namespace x3;

        auto const string 
            = x3::rule<struct string_, std::string> {"string"}
            = lexeme[+graph];
        auto const employee
            = x3::rule<class employee, ast::employee>{"employee"}
            = int_ >> string >> string >> double_;

        auto const team
            = x3::rule<class team, ast::team>{"team"}
            = string >> int_;

        auto const department
            = x3::rule<class department, ast::department>{"department"}
            = string >> int_ >> double_;

        auto const corporation
            = x3::rule<class corporation, ast::corporation>{"corporation"}
            = string >> int_;

        auto any = employee|department|team|corporation;
        auto const input  = skip(blank) [ *(any >> eol) ];
    }
}

int main()
{
    namespace x3 = boost::spirit::x3;
    using boost::spirit::x3::ascii::blank;
    using x3::char_;
    using client::ast::var_types;

    std::vector<var_types> types;

    std::string const iss(R"(30 joe smith 100000.00
20 mary jo 100000.00
25 john doe 100000.00
teamA 1
teamB 1
teamC 1
accounting 1 100000.00
engineering 2 200000.00
AcmeCorp 3
)");

    auto iter = iss.begin(), eof = iss.end();

    bool ok = parse(iter, eof, client::parser::input, types);

    if (iter != eof) {
        std::cout << "Remaining unparsed: '" << std::string(iter, eof) << "'\n";
    }
    std::cout << "Parsed: " << (100.0 * std::distance(iss.begin(), iter) / iss.size()) << "%\n";
    std::cout << "ok = " << ok << std::endl;

    for (auto& item : types) {
        boost::apply_visitor([](auto& v) { std::cout << boost::fusion::as_deque(v) << "\n"; }, item);
    }
}

Prints

Parsed: 100%
ok = 1
(30 joe smith 100000)
(20 mary jo 100000)
(25 john doe 100000)
(teamA 1)
(teamB 1)
(teamC 1)
(accounting 1 100000)
(engineering 2 200000)
(AcmeCorp 3)

With a lot of debug information if enabled:

<employee>
  <try>30 joe smith 100000.</try>
  <string>
    <try> joe smith 100000.00</try>
    <success> smith 100000.00\n20 </success>
    <attributes>[j, o, e]</attributes>
  </string>
  <string>
    <try> smith 100000.00\n20 </try>
    <success> 100000.00\n20 mary j</success>
    <attributes>[s, m, i, t, h]</attributes>
  </string>
  <success>\n20 mary jo 100000.0</success>
  <attributes>[30, [j, o, e], [s, m, i, t, h], 100000]</attributes>
</employee>
<employee>
  <try>20 mary jo 100000.00</try>
  <string>
    <try> mary jo 100000.00\n2</try>
    <success> jo 100000.00\n25 joh</success>
    <attributes>[m, a, r, y]</attributes>
  </string>
  <string>
    <try> jo 100000.00\n25 joh</try>
    <success> 100000.00\n25 john d</success>
    <attributes>[j, o]</attributes>
  </string>
  <success>\n25 john doe 100000.</success>
  <attributes>[20, [m, a, r, y], [j, o], 100000]</attributes>
</employee>
<employee>
  <try>25 john doe 100000.0</try>
  <string>
    <try> john doe 100000.00\n</try>
    <success> doe 100000.00\nteamA</success>
    <attributes>[j, o, h, n]</attributes>
  </string>
  <string>
    <try> doe 100000.00\nteamA</try>
    <success> 100000.00\nteamA 1\nt</success>
    <attributes>[d, o, e]</attributes>
  </string>
  <success>\nteamA 1\nteamB 1\ntea</success>
  <attributes>[25, [j, o, h, n], [d, o, e], 100000]</attributes>
</employee>
<employee>
  <try>teamA 1\nteamB 1\nteam</try>
  <fail/>
</employee>
<department>
  <try>teamA 1\nteamB 1\nteam</try>
  <string>
    <try>teamA 1\nteamB 1\nteam</try>
    <success> 1\nteamB 1\nteamC 1\na</success>
    <attributes>[t, e, a, m, A]</attributes>
  </string>
  <fail/>
</department>
<team>
  <try>teamA 1\nteamB 1\nteam</try>
  <string>
    <try>teamA 1\nteamB 1\nteam</try>
    <success> 1\nteamB 1\nteamC 1\na</success>
    <attributes>[t, e, a, m, A]</attributes>
  </string>
  <success>\nteamB 1\nteamC 1\nacc</success>
  <attributes>[[t, e, a, m, A], 1]</attributes>
</team>
<employee>
  <try>teamB 1\nteamC 1\nacco</try>
  <fail/>
</employee>
<department>
  <try>teamB 1\nteamC 1\nacco</try>
  <string>
    <try>teamB 1\nteamC 1\nacco</try>
    <success> 1\nteamC 1\naccountin</success>
    <attributes>[t, e, a, m, B]</attributes>
  </string>
  <fail/>
</department>
<team>
  <try>teamB 1\nteamC 1\nacco</try>
  <string>
    <try>teamB 1\nteamC 1\nacco</try>
    <success> 1\nteamC 1\naccountin</success>
    <attributes>[t, e, a, m, B]</attributes>
  </string>
  <success>\nteamC 1\naccounting </success>
  <attributes>[[t, e, a, m, B], 1]</attributes>
</team>
<employee>
  <try>teamC 1\naccounting 1</try>
  <fail/>
</employee>
<department>
  <try>teamC 1\naccounting 1</try>
  <string>
    <try>teamC 1\naccounting 1</try>
    <success> 1\naccounting 1 1000</success>
    <attributes>[t, e, a, m, C]</attributes>
  </string>
  <fail/>
</department>
<team>
  <try>teamC 1\naccounting 1</try>
  <string>
    <try>teamC 1\naccounting 1</try>
    <success> 1\naccounting 1 1000</success>
    <attributes>[t, e, a, m, C]</attributes>
  </string>
  <success>\naccounting 1 100000</success>
  <attributes>[[t, e, a, m, C], 1]</attributes>
</team>
<employee>
  <try>accounting 1 100000.</try>
  <fail/>
</employee>
<department>
  <try>accounting 1 100000.</try>
  <string>
    <try>accounting 1 100000.</try>
    <success> 1 100000.00\nenginee</success>
    <attributes>[a, c, c, o, u, n, t, i, n, g]</attributes>
  </string>
  <success>\nengineering 2 20000</success>
  <attributes>[[a, c, c, o, u, n, t, i, n, g], 1, 100000]</attributes>
</department>
<employee>
  <try>engineering 2 200000</try>
  <fail/>
</employee>
<department>
  <try>engineering 2 200000</try>
  <string>
    <try>engineering 2 200000</try>
    <success> 2 200000.00\nAcmeCor</success>
    <attributes>[e, n, g, i, n, e, e, r, i, n, g]</attributes>
  </string>
  <success>\nAcmeCorp 3\n</success>
  <attributes>[[e, n, g, i, n, e, e, r, i, n, g], 2, 200000]</attributes>
</department>
<employee>
  <try>AcmeCorp 3\n</try>
  <fail/>
</employee>
<department>
  <try>AcmeCorp 3\n</try>
  <string>
    <try>AcmeCorp 3\n</try>
    <success> 3\n</success>
    <attributes>[A, c, m, e, C, o, r, p]</attributes>
  </string>
  <fail/>
</department>
<team>
  <try>AcmeCorp 3\n</try>
  <string>
    <try>AcmeCorp 3\n</try>
    <success> 3\n</success>
    <attributes>[A, c, m, e, C, o, r, p]</attributes>
  </string>
  <success>\n</success>
  <attributes>[[A, c, m, e, C, o, r, p], 3]</attributes>
</team>
<employee>
  <try></try>
  <fail/>
</employee>
<department>
  <try></try>
  <string>
    <try></try>
    <fail/>
  </string>
  <fail/>
</department>
<team>
  <try></try>
  <string>
    <try></try>
    <fail/>
  </string>
  <fail/>
</team>
<corporation>
  <try></try>
  <string>
    <try></try>
    <fail/>
  </string>
  <fail/>
</corporation>

AST FIX

If you rework your AST to mimick the grammar better:

namespace client { namespace ast {
    struct employee    { int age; std::string surname; std::string forename; double salary; };
    struct team        { std::string name; int num_employees;                               };
    struct department  { std::string name; int num_teams; double budget;                    };
    struct corporation { std::string name; int num_depts;                                   };

    struct input {
        std::vector<employee>    employees;
        std::vector<team>        teams;
        std::vector<department>  departments;
        std::vector<corporation> corporations;
    };
} }

Now, as it happens all the attribute coercion rules become redundant, and you can simply have this grammar:

namespace parser
{
    using namespace x3;

    static auto string      = lexeme[+graph];
    static auto employee    = int_ >> string >> string >> double_;
    static auto team        = string >> int_;
    static auto department  = string >> int_ >> double_;
    static auto corporation = string >> int_;

    auto const input  = skip(blank) [
           *(employee >> eol)
        >> *(team >> eol)
        >> *(department >> eol)
        >> *(corporation >> eol)
    ];
}

That's all. I prefer a helper to be more expressive with the line ends:

static auto lines = [](auto p) { return *(p >> eol); };
auto const input  = skip(blank) [
       lines(employee)
    >> lines(team)
    >> lines(department)
    >> lines(corporation)
];

Note

No more variant, printing is as simple as you'd expect:

for (auto& item : types.employees)    { std::cout << boost::fusion::as_deque(item) << "\n"; }
for (auto& item : types.teams)        { std::cout << boost::fusion::as_deque(item) << "\n"; }
for (auto& item : types.departments)  { std::cout << boost::fusion::as_deque(item) << "\n"; }
for (auto& item : types.corporations) { std::cout << boost::fusion::as_deque(item) << "\n"; }

No more ambiguity between team/department because they can only occur in fixed order

Live On Coliru

#include <iostream>
#include <boost/spirit/home/x3.hpp>
#include <boost/fusion/include/adapt_struct.hpp>
#include <boost/fusion/include/io.hpp>

namespace x3 = boost::spirit::x3;

namespace client { namespace ast {
    struct employee    { int age; std::string surname; std::string forename; double salary; };
    struct team        { std::string name; int num_employees;                               };
    struct department  { std::string name; int num_teams; double budget;                    };
    struct corporation { std::string name; int num_depts;                                   };

    struct input {
        std::vector<employee>    employees;
        std::vector<team>        teams;
        std::vector<department>  departments;
        std::vector<corporation> corporations;
    };
} }

BOOST_FUSION_ADAPT_STRUCT(client::ast::employee, age, surname, forename, salary)
BOOST_FUSION_ADAPT_STRUCT(client::ast::team, name, num_employees)
BOOST_FUSION_ADAPT_STRUCT(client::ast::department, name, num_teams, budget)
BOOST_FUSION_ADAPT_STRUCT(client::ast::corporation, name, num_depts)
BOOST_FUSION_ADAPT_STRUCT(client::ast::input, employees, teams, departments, corporations)

namespace client
{
    namespace parser
    {
        namespace ascii = boost::spirit::x3::ascii;

        using namespace x3;

        auto const string 
            //= x3::rule<struct string_, std::string> {"string"}
            = lexeme[+graph];

        auto const employee
            //= x3::rule<class employee, ast::employee>{"employee"}
            = int_ >> string >> string >> double_;

        auto const team
            //= x3::rule<class team, ast::team>{"team"}
            = string >> int_;

        auto const department
            //= x3::rule<class department, ast::department>{"department"}
            = string >> int_ >> double_;

        auto const corporation
            //= x3::rule<class corporation, ast::corporation>{"corporation"}
            = string >> int_;

        auto lines = [](auto p) { return *(p >> eol); };
        auto const input  
             //= x3::rule<struct _input, ast::input>{"input"} 
             = skip(blank) [
               lines(employee)
            >> lines(team)
            >> lines(department)
            >> lines(corporation)
        ];
    }
}

int main()
{
    namespace x3 = boost::spirit::x3;
    using boost::spirit::x3::ascii::blank;
    using x3::char_;

    std::string const iss(R"(30 joe smith 100000.00
20 mary jo 100000.00
25 john doe 100000.00
teamA 1
teamB 1
teamC 1
accounting 1 100000.00
engineering 2 200000.00
AcmeCorp 3
)");

    auto iter = iss.begin(), eof = iss.end();

    client::ast::input types;
    bool ok = parse(iter, eof, client::parser::input, types);

    if (iter != eof) {
        std::cout << "Remaining unparsed: '" << std::string(iter, eof) << "'\n";
    }
    std::cout << "Parsed: " << (100.0 * std::distance(iss.begin(), iter) / iss.size()) << "%\n";
    std::cout << "ok = " << ok << std::endl;

    for (auto& item : types.employees)    { std::cout << boost::fusion::as_deque(item) << "\n"; }
    for (auto& item : types.teams)        { std::cout << boost::fusion::as_deque(item) << "\n"; }
    for (auto& item : types.departments)  { std::cout << boost::fusion::as_deque(item) << "\n"; }
    for (auto& item : types.corporations) { std::cout << boost::fusion::as_deque(item) << "\n"; }
}

Prints

Parsed: 100%
ok = 1
(30 joe smith 100000)
(20 mary jo 100000)
(25 john doe 100000)
(teamA 1)
(teamB 1)
(teamC 1)
(accounting 1 100000)
(engineering 2 200000)
(AcmeCorp 3)

Added a [live demo](http://coliru.stacked-crooked.com/a/9f772975e68740f6) and some improvement notes. Also, added a version that is much briefer and simpler: [only 56 of code](http://coliru.stacked-crooked.com/a/d1152ac0a84da3bb) comparing to your original [131 lines of code](http://coliru.stacked-crooked.com/a/f256024bf4ff028c), and it works :) — sehe, Jul 16 '17 at 12:15

X3, how to populate a more complex AST?

1 Answers1

DIRECT FIX

AST FIX

Linked