How to use boost::tokenizer as a C++ class member?

Question

I am replacing RWCTokenizer with boost::tokenizer and I can't get it to compile as the code is currently designed. This is what its basically doing right now:

parse.h
    RWCTokenizer*           myTokenizer;

parse.C
function parse::A
    myTokenizer = new RWCTokenizer(myTokenLine);

function parse::B
    token =  ((*myTokenizer)());

function parse::C
    if(myTokenizer == EcDNull)
    {
        return ""; // end of tokens
    }

So basically they are setting up the parse in A, get tokens in B, and checking for the end in C.

Therefore I need to have iterator and tok variables declared in the include so they can be accessed by A, B, and C. But when I compile, the myTok says it needs 2 arguments so I don't know how to save that information. This is currently what I tried to do:

parse.h
    boost::tokenizer< boost::char_separator<char> > myTok;
    boost::tokenizer< boost::char_separator<char> >::iterator myTokenizer2;

parse.C
    function parse::A
    boost::char_separator<char> sep(" ");   // break on white space
    std::string myTokenLine_TMP = myTokenLine.data();
    boost::tokenizer< boost::char_separator<char> > tok(myTokenLine_TMP, sep);
    myTokenizer = tok.begin();
    myTok = tok;

    function parse::B
    RWCString token = *myTokenizer;
    myTokenizer++;

    function parse::C
    if (myTokenizer == myTok.end())
    {
        return "";   // end of tokens
    }

The only the examples I see online has the boost code in a c main function where its all together vs. a C++ class example. The only other thing right now I can think of is to declare the boost stuff a static at the top of the class which is something I really don't want to do.

If I can some how save the myTok.end() part I should be able to compile. But I think I need the hold structure because the line I am parse was passed into tok(myTokenLine_TMP, sep) and it should go out of scope when I leave the function so saving the end() part would be useless.

So how do you do it in a C++ class?

exactly the same as in a regular function. Create a `boost::tokenizer` data member like you would create an int data member. You might have to rework some of your code though — MivVG, Mar 23 '18 at 19:48
If I knew how to rework the code I would have done it, but I can't see anyway of doing it. If you know how to declare in the include and then assign in the function please post an example because I can't see anyway of doing it with the posted documentation boost has put out. — user3416126, Mar 26 '18 at 12:27

score 0 · Answer 1 · answered Mar 26 '18 at 15:22

0

To do it in a C++ class, you just need to make the tokenizer a member of your class, like

class MyClass
{
private:
     boost::tokenizer<boost::char_seperator<char>> m_tokenizer;
};

answered Mar 26 '18 at 15:22

MivVG

679
4
16

While that will declare it, you are not setting any variables and there are no functions that allow you to set the arguments except in the constructor call so this will not work. If you look at "what I tried" code example, this is what I initially tried and I was unable to set it to anything. I am thinking I will have to make a helper class that I can allocate which will allow me to do what I want. I am going to see if I can get that work and will post a working solution if it does. – user3416126 Mar 27 '18 at 18:36
May I ask how you it is possible that you cannot set anything to it. I don't know much about `boost::tokenizer`, but it still is a regular data member that can be accessed by all members functions of the class, so your "what I tried" example should be perfectly fine, as long as you store the token you get in the B function somewhere, preferably as a data member of your class. – MivVG Mar 29 '18 at 09:51
Because that person that coded the class did not intend it be used as a member variable. There is no assignment operators, no option to change the pattern as you parse (like strtok) or an function that allows to to set the input arguments outside the initial constructor. I have decided to use strtok and create a helper class using it to duplicate the Roguewave tokenizer class where I need it to be a member variable. Its working so I am fine with it. – user3416126 Mar 30 '18 at 14:11

score 0 · Answer 2 · answered Mar 30 '18 at 14:13

0

I have decided to coded a helper class that is using strtok vs. boost tokenizer that will allow me to duplicate the Roguewave tokenizer class.

answered Mar 30 '18 at 14:13

user3416126

148
11

How to use boost::tokenizer as a C++ class member?

2 Answers2