2

is there a way to get ocammlex work with more keywords? i've written an interpreter and parser for the german language, which "compiles" german text into latex-pictures for sake of language analysis. it works really fine and is really new in the world of linguistics. thanks to all the developers of ocaml, that you can misuse tools like ocamllex and menhir for such things. but soon i will arrive on the limit of the ocammlex-automaton by the size of the lexicon, that it will say: "automaton to big."

i know the solution with the hashtables. I didnt try already, if that solves the problem coming, but does anybody know, if it is easy to change some type of the limiting integer or the like in the ocamllex-source-code, that I can misuse it without changing my program code? in my opinion this error about the size of the automation is needlessly silly and awkward for further abuse, so I want to ask.

thanks and have a nice day.

nlucaroni
  • 47,556
  • 6
  • 64
  • 86
user3734280
  • 21
  • 1

1 Answers1

3

I applaud your persistence in abusing the tools :-)

I looked through the sources for ocamllex and I see just one place that's checking whether the automaton is getting too big.

lexgen.ml near line 780:

let do_alloc_cell used t =
  let available =
    try Hashtbl.find tag_cells t with Not_found -> Ints.empty in
  try
    Ints.choose (Ints.diff available used)
  with
  | Not_found ->
      temp_pending := false ;
      let n = !next_mem_cell in
      if n >= 255 then raise Memory_overflow ;
      Hashtbl.replace tag_cells t (Ints.add n available) ;
      incr next_mem_cell ;
      n

There's just a mysterious comparison against 255, with no comment explaining any invariants etc. I looked through the code briefly and also looked at the Lexing module. I don't see any non-obvious dependencies on the value 255. So it's possible you could build your own copy of ocamllex with a larger value here. You might try 1023 (one less than power of 2).

You might also just want to break down and use a different tool. I realize this is what others are probably telling you. As I say, I admire your persistence.

Jeffrey Scofield
  • 65,646
  • 2
  • 72
  • 108
  • Oh, thanks, i will try that! What tool would you propose? I've seen some lingustic inviroment, but it was really slow and did not the complex tasks, that i wanted. With these ocaml-tools i can parse the ugliest multi-clause-senteces, that are invented in german. - but sure, i put some additional symbols in the text. and the ocaml-tools do it in seconds with large amounts of text in opposition to that, what i have seen yet. – user3734280 Jul 15 '14 at 20:35
  • If you're just testing for membership in a set of strings, a hash table will work (as you said). The reason ocamllex is faster is (possibly) that the table is essentially pre-loaded for you. So you could figure a way to build a *static* hash table. If you're doing more interesting things with finite state automata, I don't actually know what to suggest. I searched the net for "ocaml fsa" and found some computational linguistics tools in OCaml. But possibly you're the guy writing them :-) – Jeffrey Scofield Jul 15 '14 at 20:43