How to automate dos2unix using shell script?

Question

I have a bunch of xml files in a directory that need to have the dos2unix command performed on them and new files will be added every so often. I Instead of manually performing dos2unix command on each files everytime I would like to automate it all with a script. I have never even looked at a shell script in my life but so far I have this from what I have read on a few tutorials:

FILES=/tmp/testFiles/*
for f in $FILES
do
  fname=`basename $f`
  dos2unix *.xml $f $fname
done

However I keep getting the 'usage' output showing up. I think the problem is that I am not assigning the name of the new file correctly (fname).

score 8 · Accepted Answer · answered Jul 02 '12 at 11:31

The reason you're getting a usage message is that dos2unix doesn't take the extra arguments you're supplying. It will, however, accept multiple filenames (also via globs). You don't need a loop unless you're processing more files than can be accepted on the command line.

dos2unix /tmp/testFiles/*.xml

Should be all you need, unless you need recursion:

find /tmp/testFiles -name '*.xml' -exec dos2unix {} +

(for GNU find)

sorpigal · Answer 2 · 2012-07-02T11:29:13.520

If all files are in one directory (no recursion needed) then you're almost there.

for file in /tmp/testFiles/*.xml ; do
    dos2unix "$file"
done

By default dos2unix should convert in place and overwrite the original.

If recursion is needed you'll have to use find as well:

find /tmp/testFiles -name '*.xml' -print0 | while IFS= read -d '' file ; do
    dos2unix "$file"
done

Which will work on all files ending with .xml in /tmp/testFiles/ and all of its sub-directories.

If no other step are required you can skip the shell loop entirely:

Non-recursive:

find /tmp/testFiles -maxdepth 1 -name '*.xml' -exec dos2unix {} +

And for recursive:

find /tmp/testFiles -name '*.xml' -exec dos2unix {} +

In your original command I see you finding the base name of each file name and trying to pass that to dos2unix, but your intent is not clear. Later, in a comment, you say you just want to overwrite the files. My solution performs the conversion, creates no backups and overwrites the original with the converted version. I hope this was your intent.

He's not storing the results of an expanded glob. The glob is not expanded on assignment. — Dennis Williamson, Jul 02 '12 at 11:26
@DennisWilliamson: I see that you are right. I have removed this comment, thank you for the correction. — sorpigal, Jul 02 '12 at 11:31

Igor Chubin · Answer 3 · 2012-07-02T11:30:14.440

1

mkdir /tmp/testFiles/converted/
for f in /tmp/testFiles/*.xml
do
  fname=`basename $f`
  dos2unix $f ${f/testFiles\//testFiles\/converted\/}
  # or for pure sh:
  # dos2unix $f $(echo $f | sed s@testFiles/@testFiles/converted/@)
done

The result will be saved in the converted/ subdirectory.

The construction ${f/testFiles\//testFiles\/converted\/} (thanks to Rush) or sed is used here to add converted/ before the name of the file:

$ echo /tmp/testFiles/1.xml | sed s@testFiles/@testFiles/converted/@
/tmp/testFiles/converted/1.xml

edited Jul 02 '12 at 11:30

answered Jul 02 '12 at 10:43

Igor Chubin

61,765
13
122
144

Thank you but can you explain what this does: "$(echo $f | sed s@testFiles/@testFiles/converted/)" Edit: this command does not work. Error: sed: command garbled: s@testFiles/@testFiles/converted/, removing the sed command gives error: COuldn't open input file FILES – Alan Smith Jul 02 '12 at 10:45
I just add `converted/` before that name of the file. – Igor Chubin Jul 02 '12 at 10:48
2

You don't need `sed` for that. You can do it with `${f/testFiles\//testFiles\/converted\/}` – rush Jul 02 '12 at 10:50
@Rush: that works only in bash not in Bourne shell (`/bin/sh`) – Igor Chubin Jul 02 '12 at 10:54
1

Guys, I want to overwrite the files so moving the new files to another directory isn't an issue. The above command doesn't work. Any other suggestions? Thanks. – Alan Smith Jul 02 '12 at 11:02
What does it mean "doesn't work"? Which error do you get? Can please show the error message? – Igor Chubin Jul 02 '12 at 11:12
1

@AlanSmith: How about `find /tmp/testFiles/ -maxdepth 1 -type f -exec dos2unix {} +` (if you dont want to have a backup)? – another.anon.coward Jul 02 '12 at 11:14
1

@IgorChubin: This question is tagged `bash` and is not tagged `sh` or `shell` or `posix-sh`, so I think bashisms are acceptable. – sorpigal Jul 02 '12 at 11:27

score -1 · Answer 4 · edited Dec 03 '21 at 04:28

-1

It is not clear which implementation of dos2unix you are using. Different implementations require different arguments. There are many different implementations around.

On RedHat/Fedora/Suse Linux you could just type

    dos2unix /tmp/testFiles/*.xml

On SunOS you are required to give an input and output file name, and the above command would destroy several of your files.

edited Dec 03 '21 at 04:28

Nimantha

6,405
6
28
69

answered Dec 23 '12 at 09:31

Erwin Waterlander

1

How to automate dos2unix using shell script?

4 Answers4