How can I use non-default delimiters when reading a text file with std::fstream?

Question

In my C++ code, I want to read from a text file (*.txt) and tokenize every entry. More specifically, I want to be able to read individual words from a file, such as "format", "stack", "Jason", "europe", etc.

I chose to use fstream to perform this task, and I do not know how to set it's delimiter to the ones I want to use (space, \n, as well as hyphens and even apostrophes as in "Mcdonal's"). I figured space and \n are the default delimiters, but hyphens are not, but I want to treat them as delimiters so that when parsing the file, I will get words in "blah blah xxx animal--cat" as simply "blah", "blah", "xxx", "animal", "cat".

That is, I want to be able to get two strings from "stack-overflow", "you're", etc, and still be able to maintain \n and space as delimiters at the same time.

Answer 1

An istream treats "white space" as delimiters. It uses a locale to tell it what characters are white space. A locale, in turn, includes a ctype facet that classifies character types. Such a facet could look something like this:

#include <locale>
#include <iostream>
#include <algorithm>
#include <iterator>
#include <vector>
#include <sstream>

class my_ctype : public
std::ctype<char>
{
    mask my_table[table_size];
public:
    my_ctype(size_t refs = 0)  
        : std::ctype<char>(&my_table[0], false, refs)
    {
        std::copy_n(classic_table(), table_size, my_table);
        my_table['-'] = (mask)space;
        my_table['\''] = (mask)space;
    }
};

And a little test program to show it works:

int main() {
    std::istringstream input("This is some input from McDonald's and Burger-King.");
    std::locale x(std::locale::classic(), new my_ctype);
    input.imbue(x);

    std::copy(std::istream_iterator<std::string>(input),
        std::istream_iterator<std::string>(),
        std::ostream_iterator<std::string>(std::cout, "\n"));

    return 0;
}

Result:

This
is
some
input
from
McDonald
s
and
Burger
King.

istream_iterator<string> uses >> to read the individual strings from the stream, so if you use them directly, you should get the same results. The parts you need to include are creating the locale and using imbue to make the stream use that locale.

Answer 2

You can use

istream::getline(char* buffer, steamsize maxchars, char delim)

although this only supports a single delimiter. To further split the lines on your different delimiters, you could use

char* strtok(char* inString, const char* delims)  

which takes multiple delimeters. When you use strtok you only need to pass it the address of your buffer the first time - after that just pass in a null and it will give you the next token from the last one it gave you, returning a null pointer when there are no more.

EDIT: A specific implementation would be something like

char buffer[120]; //this size is dependent on what you expect the file to contain
while (!myIstream.eofbit) //I may have forgotten the exact syntax of the end bit
{
    myIstream.getline(buffer, 120); //using default delimiter of \n
    char* tokBuffer;
    tokBuffer = strtok(buffer, "'- ");
    while (tokBuffer != null) {
        cout << "token is: " << tokBuffer << "\n";
        tokBuffer = strtok(null, "'- "); //I don't need to pass in the buffer again because it remembers the first time I called it
    }
}