c++: how to break (not parse) a string into command line arguments?

Question

I am using boost::program_options to parse my command line. Now, I am adding support for batch execution, by means of a --script argument denoting a file, containing command line options on every line, for instance:

--src="\"z:\dev\veds\sqlexpress\Run 1.ved\"" --src-kind=bla --yz
--src=z:\dev\veds\sqlexpress\db.ebf
--src=z:\dev\veds\sqlexpress\db2.mdf
--src=db3
--src="\"z:\dev\veds\sqlite\Run 41 (Run 23).ved\""
--src=z:\dev\veds\sqlite\ws_results_db_2012_01_15_18_37_03.db3
--src=z:\dev\veds\mysql\10.ved
--src=z:\dev\veds\mysql\db

Each line in the file denotes a single execution of my tool and lists the command line options for this particular execution.

The problem is that reading the script file yields complete lines, which are not broken into individual command line options. But, one has to have argc and argv in order to use boost::program_options, i.e. it depends on someone to break the command line into different options.

I cannot simply break by spaces, because some values contain spaces and hence they are enclosed with double quotes, even nested double quotes.

On the other hand, I do not want to run the tool from the OS command prompt for each set of command line options, because of the expensive bootstrap - the reason why I am introducing the script feature in the first place.

Is there a simple way to break the lines into the command line arguments in the same way the OS does it?

Thanks.

Answer 1

OK, I have it figured out. Here is my code:

  string script;
  {
    ifstream file(scriptPath.c_str());
    file.seekg(0, ios::end);
    script.resize(file.tellg());
    file.seekg(0, ios::beg);
    file.read(const_cast<char *>(script.c_str()), script.size());
  }
  boost::replace_all(script, "\\", "\\\\");       // Escape the backslashes
  boost::replace_all(script, "\\\\\"", "\\\"");   // Except for those escaping the quotes
  boost::trim_right_if(script, is_space_or_zero); // There are extra '\0' in the string, because the file is read as text, but its length was computed as binary
  vector<string> lines;
  boost::split(lines, script, boost::is_any_of("\n"));  // I prefer getting a string line iterator here, the question is how?
  escaped_list_separator<char> sep('\\', ' ', '"');
  int res = 0;
  BOOST_FOREACH (const string& line, lines) 
  {
    // reset the command line variables here, since this is like a new execution

    // Tokenize the command line, respecting escapes and quotes  
    tokenizer<escaped_list_separator<char>> tok(line, sep);
    vector<string> args(tok.begin(), tok.end());

    po::variables_map vm;
    po::store(po::command_line_parser(args).options(options).run(), vm);

    res += run(vm);
  }

I am using http://www.boost.org/doc/libs/1_48_0/libs/tokenizer/ to break the lines. Works very well.

Answer 2

The Boost documentation covers Response Files, including a simple example using them. That would be close to what you want, except that they also say it "has some limitations," which include the parsing of spaces.

They also have parse_config_file() which will load options from a file. Here you'd be giving up having identical syntax in the file as on the command line, and their included implementation would only (easily) support one "command invocation" per program invocation. But I bet you could look at how they do it and copy some of that code. If I were you I might adjust it to support .ini syntax like this:

[job1]
src=z:\dev\veds\sqlexpress\Run 1.ved
src-kind=bla
y=
z=

[job2]
src=z:\dev\veds\sqlexpress\db.ebf

[another_job]
src=z:\dev\veds\sqlexpress\db2.mdf

I bet this is not a horrible amount of extra work, and it gives one extra benefit of having an explicit name for each job you run. Boost's own parse_config_file() uses the section names (in brackets) as a prefix for the option names, but this is not necessary, so you may as well repurpose them in the interest of keeping the simple .ini syntax.

Edit: You want something simpler? OK. Abandon the idea of having identical syntax in the file as on the command line with respect to quoting and spaces. Decide on a suitable delimiter like ; between options if you must support spaces within your arguments. Turning something like this into argc/argv pairs should be easy enough using std::string::find() or Boost Tokenizer:

--src=z:\dev\veds\sqlexpress\Run 1.ved; --src-kind=bla; --yz
--src=z:\dev\veds\sqlexpress\db.ebf
--src=z:\dev\veds\sqlexpress\db2.mdf

Split on ; to make argv[1,2,3] in the first example, and copy your program's own argv[0] to the "fake" argv[0], then parse the options using Boost.

Answer 3

Check this out:: C++ Cookbook Splitting a string (Recipe 4.6)

Example 4-10. Split a delimited string

#include <string>
#include <vector>
#include <functional>
#include <iostream>

using namespace std;

void split(const string& s, char c,
           vector<string>& v) {
   string::size_type i = 0;
   string::size_type j = s.find(c);

   while (j != string::npos) {
      v.push_back(s.substr(i, j-i));
      i = ++j;
      j = s.find(c, j);

      if (j == string::npos)
         v.push_back(s.substr(i, s.length( )));
   }
}

int main( ) {
   vector<string> v;
   string s = "Account Name|Address 1|Address 2|City";

   split(s, '|', v);

   for (int i = 0; i < v.size( ); ++i) {
      cout << v[i] << '\n';
   }
}

--

template<typename T>
void split(const basic_string<T>& s, T c,
           vector<basic_string<T> >& v) {
   basic_string<T>::size_type i = 0;
   basic_string<T>::size_type j = s.find(c);

   while (j != basic_string<T>::npos) {
      v.push_back(s.substr(i, j-i));
      i = ++j;
      j = s.find(c, j);

      if (j == basic_string<T>::npos)
         v.push_back(s.substr(i, s.length( )));
   }
}

Example 4-11. Splitting a string with Boost

#include <iostream>
#include <string>
#include <list>
#include <boost/algorithm/string.hpp>

using namespace std;
using namespace boost;

int main( ) {

   string s = "one,two,three,four";
   list<string> results;

   split(results, s, is_any_of(","));  // Note this is boost::split

   for (list<string>::const_iterator p = results.begin( );
        p != results.end( ); ++p) {
      cout << *p << endl;
   }
}

--

template<typename Seq,
         typename Coll,
         typename Pred>
Seq& split(Seq& s, Coll& c, Pred p,
        token_compress_mode_type e = token_compress_off);

I'm not at liberty of sharing the text (illegal to copy/paste from a book), but these examples are pretty explainative. If you want to see the text you will need to refer the book.

These two examples were taken from recipe 4.6 of

C++ Cookbook

By Jeff Cogswell, Christopher Diggins, Ryan Stephens, Jonathan Turkanis

Publisher: O'Reilly

Pub Date: November 2005

ISBN: 0-596-00761-2