Debug Assertion Failed File, tokenScanner, and text files

Question

I have written a program that processes text files one at a time and extract relevant information. My program works well with some of the text files and not others. There is no obvious difference between the files that run seamlessly through my program and those that don't.

As far as the problematic files are concerned:

1. the program opens the file
2. it reads in and processes a good chunk of the lines one at a time as it should

3. But then it reaches a problem line and gives the error message:

   "Debug Assertion Failed File: f:/dd/vctools/crt_bld/self_x86/src/isctype.c Line: 56 Expression: (unsigned)(c+1) <= 256"

When I enter the debugger mode the problem seems to arise from the "while(tokenScanner)" loop in my code below. I pulled up the content of the problem line being processed and compared that across a couple of problem files and I found that the Assertion Failure message pops up at </li> where the last token being processed is ">". It's not clear to me why this is a problem. This particular token in the original text file is contiguous with <li in the form </li><li. Therefore the scanner is having trouble half way throught this string. Any thoughts on why this is and how I can fix this? Any advice would be much appreciated!

Here is the relevant portion of my code:

#include <string>
#include <iostream>
#include <fstream> //to get data from files
#include "filelib.h"
#include "console.h"
#include "tokenScanner.h"
#include "vector.h"
#include "ctype.h"
#include "math.h"

using namespace std;


/*Prototype Function*/
void evaluate(string expression);
Vector<string> myVectorOfTokens; //will store the tokens
Vector<string> myFileNames;

/*Main Program*/
int main() {

    /*STEP1 : Creating a vector of the list of file names 
              to iterate over for processing*/
    ifstream infile; //declaring variable to refer to file list
    string catchFile = promptUserForFile(infile, "Input file:");
    string line; //corresponds to the lines in the master file containing the list files
    while(getline(infile, line)){
        myFileNames.add(line);
    }

    /* STEP 2: Iterating over the file names contained in the vector*/
    int countFileOpened=0; //keeps track of number of opened files

    for (int i=1; i< myFileNames.size(); i++){
        myVectorOfTokens.clear(); //resetting the vector of tokens for each new file

        string fileName;
        string line2;
        ifstream inFile;
        fileName= myFileNames[i];

        inFile.open(fileName.c_str());   //open file convert c_str

        if (inFile){
            while(getline(inFile, line2)){
                evaluate(line2);
            }
        }

        inFile.close();
        countFileOpened++;
    }
    return 0;
}

/*Function for Extracting the Biographer Name*/
void evaluate(string line){
    /*Creating a Vector of Tokens From the Text*/
    TokenScanner scanner(line); //the constructor
    while (scanner.hasMoreTokens()){
        string token=scanner.nextToken();
        myVectorOfTokens.add(token);
    }
}

Answer 1

while(!inFile.eof()

is just wrong (in almost any case)

while(getline(inFile, line2))
      evaluate(line2);

is better