16

How do I remove ^H and ^M characters from a file using Linux shell scripting?

^[[0^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H   rcv-packets: 0
^[[0^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H      rcv-errs: 0
     rcv-drop: 0
     rcv-fifo: 0
     rcv-frame: 0
sarnold
  • 102,305
  • 22
  • 181
  • 238
Hasan
  • 169
  • 1
  • 1
  • 3

7 Answers7

25

What you're seeing there are control characters, you simply could delete them with tr

cat your_file |
tr -d '\b\r'

this is better:

tr -d '\b\r' < your_file
Patrick J. S.
  • 2,885
  • 19
  • 26
  • @dtmilano: yes, sorry, edited that - I learned cat in university and still use it even though some shells I work on have the 'useless use of cat detected' warning – Patrick J. S. Jun 30 '11 at 15:30
9

Two methods come to mind immediately:

  • tr -d control+v control+h
  • sed 's/control+v control+h//g'

Here's both in action:

$ od -c test
0000000  \b   h   e   l   l   o  \b   t   h   e   r   e  \b  \n
0000016
$ sed 's/^H//g' < test | od -c
0000000   h   e   l   l   o   t   h   e   r   e  \n
0000013
$ tr -d ^H < test | od -c
0000000   h   e   l   l   o   t   h   e   r   e  \n
0000013
sarnold
  • 102,305
  • 22
  • 181
  • 238
2

For removing ^M characters appearing at the end of every line, I usually do this in vi editor.

:%s/.$//g

It just removes the last character of every line irrespective of what the character is. This solved my provlem.

subbu
  • 524
  • 6
  • 16
  • Nice trick! Right what I was looking for. I found on some site to use :%s/^M//g , but it didn't work. Can you tell me why? – rightaway717 Oct 16 '13 at 11:29
  • '^' indicates start of the string/word/line... So, your expression might mean all the words/lines that starts with 'M' should be replaced or deleted... Alternatively, you can also use this: `:%s/\/r//g` to remove those characters. – subbu Oct 25 '13 at 08:30
  • "irrespective of what the character is" is not usually what you want; the comment "`:%s/^M//g` didn't work" is probably because the "^M" was typed literally; it actually meant "control-M", which can be "typed" by doing `control-V` + `control-M`. Even better (as per others answers) is just `tr -d '\r'`, or `unix2dos {file}`. – michael Sep 19 '14 at 08:23
  • @michael_n: I think you mean `dos2unix` for deleting the carriage returns. `unix2dos` would add them. – ShadowRanger Apr 12 '16 at 00:33
0

You can remove all control characters by using tr, e.g.

tr -d "[:cntrl:]" file.txt

To exclude some of them (like line endings), check: Removing control characters from a file.

Community
  • 1
  • 1
kenorb
  • 155,785
  • 88
  • 678
  • 743
0

if you want to change original file, do this:

sed -i '.bak' 's/^M//g ; s/^H//g' test.md

(^M is control+v control+m)
(^H is control+v control+h)

much file, you can do this:

find source -name '*.md' | xargs sed -i '.bak' 's/^M//g ; s/^H//g'
xiaweiss
  • 1
0

Use sed utility. See below as per examples:

sed 's/%//' file > newfile
echo "82%%%" | sed 's/%*$//'
echo "68%" | sed "s/%$//" #assume % is always at the end.
kenorb
  • 155,785
  • 88
  • 678
  • 743
Sergey
  • 749
  • 6
  • 13
0

I used dos2unix and it worked for me.

If you are using a Debian-based distro, you should be able to do sudo apt-get install dos2unix.

If you are using a RH-like distro, you should be able to do sudo yum install dos2unix.

Once it is installed, you can just give the target file as an argument

dos2unix file.sh
Fariman Kashani
  • 856
  • 1
  • 16
  • 29