352

If the HTTP response body for a curl request doesn't contain a trailing newline, I end up with this really annoying condition where the shell prompt is in the middle of the line, and escaping is messed up enough that when I put the last curl command on the screen, deleting characters from that curl command deletes the wrong characters.

For example:

[root@localhost ~]# curl jsonip.com
{"ip":"10.10.10.10","about":"/about"}[root@localhost ~]#

Is there a trick I can use to automatically add a newline at the end of a curl response, to get the prompt back on the left edge of the screen?

Kevin Burke
  • 61,194
  • 76
  • 188
  • 305
  • 1
    None of these answers work for me with whatever version of curl my cygwin install has; only wrapping the entire curl command in an echo statement does, e.g. `echo "$(curl localhost:8001/api)"`, re this answer: https://unix.stackexchange.com/a/217611/110338 – Alex Hall Dec 07 '17 at 04:59
  • 1
    for json specifically, ```curl -s https://jsonip.com | jq``` – hanshenrik Aug 01 '20 at 08:28

5 Answers5

593

From the man file:

To better allow script programmers to get to know about the progress of curl, the -w/--write-out option was introduced. Using this, you can specify what information from the previous transfer you want to extract.

To display the amount of bytes downloaded together with some text and an ending newline:

curl -w 'We downloaded %{size_download} bytes\n' www.download.com

So try adding the following to your ~/.curlrc file:

-w "\n"
David J.
  • 6,546
  • 1
  • 19
  • 14
  • 81
    You can run the following command that does that for you: `echo '-w "\n"' >> ~/.curlrc` – tbraun Aug 09 '16 at 09:20
  • 15
    I always add `-w "\n"` manually to every curl command and TIL `~/.curlrc` exists! – Zhuoyun Wei Aug 18 '16 at 04:36
  • 2
    this may not work under git-bash on Windows; I added `-w "\nKLJLJKLJ LJKLLKJ"` to my `~/.curlrc` and ran a curl and got neither a new line or the weird characters. – jcollum Aug 10 '17 at 20:57
  • @jcollum Some Windows ports prefer `_` over `.`. Chances are your curl is looking for `~/_curlrc` and consequently `echo '-w "\n"' >> ~/_curlrc` will probably work. It does for me. btw: curl will complain over unescaped whitespace – user1129682 Nov 30 '18 at 20:42
  • 1
    There's value in using the command line with `-w "\n"` or appending `; echo` over modifying your `.curlrc`. If you modify your curlrc, some scripts may break if they do not handle a newline at the end of the response data. I personally would stick with the additions on the command line over modifying the rc. – Jason Jan 28 '20 at 21:52
  • 2
    I would use a shell alias rather then modifying the `~/.curlrc` to prevent scripts to break – Bastien Durel Dec 03 '20 at 09:05
  • 1
    Doing this will break hash checks on files downloaded with curl since it adds the extra newline, use at your own risk! – Jonno_FTW Feb 22 '22 at 00:12
  • Not working well while `pip_configs | curl --config - --parallel --parallel-immediate --parallel-max 10` as I found some respnses are merged like `{a:1}{a:1}{a:1}{a:1}\n\n\n\n`.Is there a fix for that? – Mamdouh Saeed May 14 '23 at 14:11
140

Use this:

curl jsonip.com; echo

If you need grouping to feed a pipe :

{ curl jsonip.com; echo; } | tee new_file_with_newline

OUTPUT

{"ip":"x.x.x.x","about":"/about"}

This is that simple ;)

(and not limited to curl command but all commands that not finish with a newline)

Gilles Quénot
  • 173,512
  • 41
  • 224
  • 223
  • 3
    Is there any way to wrap this in a script or bash alias or anything similar? Typing "; echo" at the end of every line can get tedious, and I feel like this should be able to be automated somehow. – Matthew G Jan 15 '13 at 00:05
  • On SO, it's better to provide some search effort. Finding how to put this in an alias or script is obvious. – Gilles Quénot Jan 15 '13 at 06:44
  • 3
    Notice that adding the echo command clears the exit code from curl command. The -w "\n" option to curl suggested in the most popular answer keeps the curl exit code available for inspection. – bradoaks Oct 04 '19 at 17:53
  • How to get `-w "\n"` works when piping to curl like `pip_as_config | curl --config - --parallel --parallel-immediate --parallel-max 10` as some respnses are merged like `{a:1}{a:1}{a:1}{a:1}\n\n\n\n` due to parallelized requests? – Mamdouh Saeed May 14 '23 at 14:18
22

For more info as well as a clean new line after curl

~/.curlrc

-w "\nstatus=%{http_code} %{redirect_url} size=%{size_download} time=%{time_total} content-type=\"%{content_type}\"\n"

(More options are available here)

redirect_url will be blank if the request doesn't get redirected or you use -L to follow the redirect.

Example output:

~ ➤  curl https://www.google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="https://www.google.co.uk/?gfe_rd=cr&amp;ei=FW">here</A>.
</BODY></HTML>

status=302 https://www.google.co.uk/?gfe_rd=cr&ei=FW size=262 time=0.044209 content-type="text/html; charset=UTF-8"
~ ➤

Edit, to make things more readable you can add ANSI colours to the -w line, it's not that easy to write directly, but this script can generate a ~/.curlrc file with colours.

#!/usr/bin/env python3
from pathlib import Path
import click
chunks = [
    ('status=', 'blue'),
    ('%{http_code} ', 'green'),
    ('%{redirect_url} ', 'green'),
    ('size=', 'blue'),
    ('%{size_download} ', 'green'),
    ('time=', 'blue'),
    ('%{time_total} ', 'green'),
    ('content-type=', 'blue'),
    ('\\"%{content_type}\\"', 'green'),
]
content = '-w "\\n'
for chunk, colour in chunks:
    content += click.style(chunk, fg=colour)
content += '\\n"\n'

path = (Path.home() / '.curlrc').resolve()
print('writing:\n{}to: {}'.format(content, path))
path.write_text(content)
SColvin
  • 11,584
  • 6
  • 57
  • 71
5

The general solution for bash is to add a newline symbol into the command prompt:

See related question (How to have a newline before bash prompt? ) and corresponding answer

This solution covers each command, not only curl.

echo $PS1 # To get your current PS1 env variable's value aka '_current_PS1_'
PS1='\n_current_PS1_'

The only side-effect is that you get command prompt after each 2nd line.

Marinos An
  • 9,481
  • 6
  • 63
  • 96
Dionio
  • 51
  • 1
  • 2
2

I managed to get a new line added dynamically to prompt when command output didn't have a new line at end. So it works not only with curl but also any other command.

# https://github.com/dylanaraps/pure-bash-bible#get-the-current-cursor-position
new_line_ps1() {
  local _ y x _
  local LIGHT_YELLOW="\001\033[1;93m\002"
  local     RESET="\001\e[0m\002"

  IFS='[;' read -p $'\e[6n' -d R -rs _ y x _
  if [[ "$x" != 1 ]]; then
    printf "\n${LIGHT_YELLOW}^^ no newline at end of output ^^\n${RESET}"
  fi
}

PS1="\$(new_line_ps1)$PS1"

my answer in UL site: https://unix.stackexchange.com/a/647881/14907

akostadinov
  • 17,364
  • 6
  • 77
  • 85