0

When I hit an endpoint of the Docker remote API, for example with cUrl in Bash, I get a response streamed to the console which might look like 

[...]
{"stream":"\u001b[91m.\u001b[0m"}
{"stream":"\u001b[91m.. .....\u001b[0m"}
{"stream":"\u001b[91m.\u001b[0m"}
{"stream":"\u001b[91m.... ...\u001b[0m"}
{"stream":"\u001b[91m.....\u001b[0m"}
{"stream":"\u001b[91m.. ....       14.2M=0.5s\u001b[0m"}
{"stream":"\u001b[91m\n\n\u001b[0m"}
{"stream":"\u001b[91m2015-08-06 09:41:20 (10.1 MB/s) - ‘workspace.zip’ saved [5063084]\n\n\u001b[0m"}
{"stream":" ---\u003e aa6d979beeec\n"}
{"stream":"Removing intermediate container fa73eeb4531d\n"}
{"stream":"Step 3 : WORKDIR ./workspace\n"}
{"stream":" ---\u003e Running in 1dc8301bfd34\n"}
{"stream":" ---\u003e 4bddbc0282c9\n"}
{"stream":"Removing intermediate container 1dc8301bfd34\n"}
{"stream":"Step 4 : EXPOSE 8080\n"}
{"stream":" ---\u003e Running in 187a95569e84\n"}
{"stream":" ---\u003e b26c7b990996\n"}
{"stream":"Removing intermediate container 187a95569e84\n"}
{"stream":"Step 5 : CMD /bin/bash some_script.sh\n"}
{"stream":" ---\u003e Running in a5027b1082c3\n"}
{"stream":" ---\u003e 276ee1506ea0\n"}
{"stream":"Removing intermediate container a5027b1082c3\n"}
{"stream":"Successfully built 276ee1506ea0\n"}
[...]

This is really annoying to read with all the escape and unicode characters. How can I print the cUrl response on the console in an easier readable form without escaping all the special characters?

This answer suggests to pipe the response to Python and use its json module, dumping it again in UTF-8. However, when using it as in the following example which is the remote API way to build a Docker image from a local Dockerfile:

tar -cvf - Dockerfile | \
curl --silent --show-error -X POST -H "Content-Type:application/tar" --data-binary @- \
"http://myDockerHost:4243/build?t=myRepo/myImage" | \
python -c 'import json, sys; sys.stdout.write(json.load(sys.stdin)[0].encode("utf-8"))'

then I get an error like

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/json/__init__.py", line 290, in load
    **kw)
  File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 369, in decode
    raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 2 column 1 - line 10 column 1 (char 48 - 764)

Looking it up told me that this occurs because Python's json module can only read a single json string but not a streamed multiline response from cUrl.

What else could be done to solve this?

Community
  • 1
  • 1
Dirk
  • 9,381
  • 17
  • 70
  • 98

2 Answers2

1

Your input is multiple json documents -- one per line. Feed each line to json.loads() separately:

>>> print(json.loads(r'{"stream":"\u001b[91m.\u001b[0m"}')['stream'])
.

It is displayed as a red dot on my screen (due to ANSI escape sequences):

>>> json.loads(r'{"stream":"\u001b[91m.\u001b[0m"}')['stream']
u'\x1b[91m.\x1b[0m'

You could use jq, to work with json on the command line:

$ echo '{"stream":"\u001b[91m.\u001b[0m"}' | jq -r .stream
.

Unrelated: Don't encode to utf-8, print Unicode directly instead. Don't hardcode the encoding of your environment inside your script. If you want to change the output encoding, set PYTHONIOENCODING envvar instead.

jfs
  • 399,953
  • 195
  • 994
  • 1,670
  • How could I feed each line separately when cUrl receives them as a stream without having saving the whole stream into a file or string and finally parse it after the connection has been closed (which would dismiss the "live" effect of the stream of course)? Parsing each line while the stream is still active does not work as stated in the question because `json.loads()` complains about further data arriving while already parsing. – Dirk Aug 20 '15 at 15:21
  • @Dirk: If you can't read standard input line-by-line in Python using `for line in sys.stdin: use(line)`; ask a separate question – jfs Aug 20 '15 at 15:25
1

Solution, based on J.F. Sebastian's answer

Both suggested approaches work well. After installing jq, it can be used like this:

tar -C ./ -cvf - Dockerfile | curl --silent --show-error -X POST -H "Content-Type:application/tar" --data-binary @- "$DOCKERHOST/build?t=repo/imageName" | jq -r .stream

Using Python instead it, handling each line of the stream separately, it looks like this:

tar -C ./ -cvf - Dockerfile | curl --silent --show-error -X POST -H "Content-Type:application/tar" --data-binary @- "$DOCKERHOST/build?t=repo/imageName" | python -c 'import json, sys; [sys.stdout.write(json.loads(line)["stream"]) for line in sys.stdin]'

Both solutions give the desired "readable" output on stdout, e.g.

Dockerfile
Step 0 : FROM ubuntu
 ---> 91e54dfb1179
Step 1 : RUN apt-get update -y
 ---> Using cache
 ---> 211dc37ab584
Step 2 : RUN apt-get install -y default-jre
 ---> Using cache
 ---> 0045a653edb9
Successfully built 0045a653edb9
Dirk
  • 9,381
  • 17
  • 70
  • 98