-2

Say for example I have a string which is the following "3*x+(b[3]+c)+x"

And my delimiters are " \t*+-/()[]"

When I use the string tokenizer to split the string and I get to token "b", how can I check to make sure that the next delimiter is "[".

I need to do this so that token "x" is different from token "b", as token x represents a simple variable and token b represents an array in the program I am trying to construct.

I am trying to write this code in Java.

O. Syed
  • 1
  • 1
  • 3
  • 1
    You can't. If your character is meaningful you shouldn't use it as a delimiter to split. – Aaron Feb 11 '18 at 01:05
  • What language is expression? – Roman C Feb 11 '18 at 01:07
  • I am trying to write this code in java – O. Syed Feb 11 '18 at 01:19
  • why dont you use simple string processing? is it tokenzer or nothing? – gpasch Feb 11 '18 at 01:24
  • @gpasch variable names can be really long thats why I want to use string tokenizer – O. Syed Feb 11 '18 at 01:34
  • None of these characters are delimiters in any programming language. They are all operators. You are using the wrong tool for the job, – user207421 Feb 11 '18 at 01:35
  • @EJP I am defining my delimiters to be these characters. I want a way to separate all the variables from the operators using StringTokenizer. My program requires me to use StringTokenizer. – O. Syed Feb 11 '18 at 01:38
  • 1
    Your program doesn't require you. Maybe you assignment requirements do .... but that is a different matter. (Or to put it another way, if your program "requires" it, then change the program!) – Stephen C Feb 11 '18 at 01:42
  • I know what you're doing, and how you're doing it. I'm telling you it's wrong. This is not how programming languages are analyzed. – user207421 Feb 11 '18 at 02:24

1 Answers1

0

When using a string tokenizer , how do I check the next character after a token if it is a delimiter?

If you use the 3 argument constructor for StringTokenizer, then 3rd argument says whether or not the delimiters are returned as tokens.

However1, this still leaves you with the awkward problem of dealing with whitespace. For example "1+1" gives you 3 tokens, but "1 + 1" gives you 5 tokens. You either need to:

  • pick a better way2 to tokenize the input,
  • filter out any white-space "tokens" before you start analyzing them, or
  • remove any whitespace from the input string before you tokenize it.

1 - I'm inclined to agree with @EJP that StringTokenizer is the wrong tool for this job. You could make it work, but IMO you shouldn't. (Unless you have been explicitly directed to do this.)

2 - Better ways might be StreamTokenizer, String.split (with a cunning regex involving lookaheads or lookbehinds), a tokenizer generated using ANTLR, javacc or some other parser generator, or ... a hand built tokenizer.

Stephen C
  • 698,415
  • 94
  • 811
  • 1,216