1

I have a file that has fairly long lines. The longest line has length 4609:

% perl -nle 'print length' ~/very_large_file | sort -nu | tail -1
4609

Now, when I just run cat ~/very_large_file it runs fine. But when I put inside backticks, it gives a 'word too long' error

% foreach line (`cat ~/very_large_file`)
Word too long.

% set x = `cat ~/very_large_file`
Word too long.

Is there an alternative to using backticks in csh to process each line of such a file?

Update My problem was solved by using a different language, but I still couldn't get the reason for the failing csh. Just came across this page that describes the manner of finding ARG_MAX. In particular, the getconf command is useful. Of course, I am still not sure whether this limit is the root cause, and if the limit applies to the languages other than csh.

Unos
  • 1,293
  • 2
  • 14
  • 35
  • 2
    What are you doing with each line? (Inevitably, someone will point you to http://www.faqs.org/faqs/unix-faq/shell/csh-whynot/.) – Keith Thompson Mar 02 '12 at 12:49
  • Are you using csh or tcsh? I tried this using tcsh 6.17.02 on a file with a single 4609-character line and didn't get any "word too long" errors. Slurping the entire file into `$x` might still cause problems. But since you have Perl, why not just use that? It's a *much* better scripting language than csh. – Keith Thompson Mar 02 '12 at 12:56
  • i need to maintain this shell script, and honestly i have a hard time doing it! The foreach loop is doing a lot of (mostly) text operations, and this bug has appeared only recently...yes, i am planning to rewrite the entire thing in perl, but if there is a way of substituting the `cat` operation with something non buggy, it would greatly help me in the short term. – Unos Mar 02 '12 at 14:30
  • Hi again...i overlooked your other query abt the shell. i am using tcsh, on rh4, so probably it is an older version...let me look up and get back... – Unos Mar 02 '12 at 14:39
  • In the old Sun/Solaris CSH man page, there was a section called LIMITS that specified how big a line of data could be processed both as characters and words (if I remember correctly). Maybe your system's man page has a similar section. Good luck. – shellter Mar 02 '12 at 22:19

1 Answers1

1

I don't mean to beat an old horse, but if you're scripting do consider moving to bash, zsh or even Korn. csh has disadvantages.

What you can try without abandoning csh completely:

  • Move to tcsh if you're with regular old (very old) csh.
  • Recompile tcsh with a longer word length (the default is 1000 bytes, I think) or with dynamic allocation.

  • If possible move the line processing to a secondary script or program and write that loop like this:

    cat ~/very_large_file | xargs secondary_script

Eduardo Ivanec
  • 11,668
  • 2
  • 39
  • 42
  • Hi...i am not sure if i can recompile the tcsh as i dont have root priveleges...but i will try your other solutions and get back... – Unos Mar 02 '12 at 14:33
  • 1
    @Unos Rebuilding the shell to have a longer maximum word length is not going to help you, because this is a kernel thing, probably `NCARGS` or `ARG_MAX` in your `` include file. Although a few kernels allow this to grow dynamically, most do not, and you should always be prepared to use an `xargs` approach, or else find some other way not to supply everything on the command line, such as reading the data from standard input instead. – tchrist Mar 03 '12 at 17:08
  • @Unos: You don't meed root privileges to build `tcsh` from source and install it somewhere under your home directory. – Keith Thompson Mar 03 '12 at 19:21
  • @all, I just rewrote the whole script in perl. I personally liked the `xargs` solution, but if I did that, I would've needed to maintain one more csh script (one too many). The links about disadvantages of csh made me thank God that I did not come across its more cryptic problems. Thanks everyone :) – Unos Mar 04 '12 at 15:59