Passing key and values to external command from jq

Question

What I'm trying to do:

Use jq to pass along parameters to ffmpeg in a bash script. Have a JSON in this external file that I generate regularly.

{
   "streams":[
      {
         "track":"/var/www/html/stream1.m3u8",
         "link":"http://playertest.longtailvideo.com/adaptive/bipbop/bipbop.m3u8"
      },
      {
         "track":"/var/www/html/stream2.m3u8",
         "link":"https://mnmedias.api.telequebec.tv/m3u8/29880.m3u8"
      },
      {
         "track":"/var/www/html/stream3.m3u8",
         "link":"http://www.streambox.fr/playlists/test_001/stream.m3u8"
      }
   ]
}

This is the command I've tried based on the response found here https://github.com/stedolan/jq/issues/503

jq -r '.streams[] | ffmpeg -v verbose -i  \(.link | @sh) -c copy -flags -global_header -hls_time 10 -hls_list_size 6 -hls_wrap 10 -start_number 1 \(.Track | @sh)"' streams.json | sh

However I get this error message:

jq: error: syntax error, unexpected IDENT, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
.streams[] | ffmpeg -v verbose -i  \(.link | @sh) -c copy -flags -global_header -hls_time 10 -hls_list_size 6 -hls_wrap 10 -start_number 1 \(.Track | @sh)"                       
jq: 1 compile error

Sure I guess . Have nodejs also installed on the box. – C.Astraea Feb 28 '18 at 16:53 — C.Astraea, Feb 28 '18 at 16:53

Charles Duffy · Accepted Answer · 2018-02-28T20:48:58.527

The shortest possible change to your original code is just to add the quotes that were missing:

jq -r '.streams[] | "ffmpeg -v verbose -i  \(.link | @sh) -c copy -flags -global_header -hls_time 10 -hls_list_size 6 -hls_wrap 10 -start_number 1 \(.Track | @sh)"' streams.json | sh
#                   ^-- this was missing

Note that "ffmpeg ..." is a string, and is contained in quotes. That said, you're relying on jq to generate safe code for your shell -- since it has features explicitly built for the purpose, this isn't as bad as an idea as it might be otherwise; but it's still better practice to avoid code generation wherever possible.

As an alternate approach that avoids code generation and is safe with all possible filenames, use jq to generate a NUL-delimited stream of track / link pairs, and a BashFAQ #1 loop to iterate over them:

#!/usr/bin/env bash

while IFS= read -r -d '' track && IFS= read -r -d '' link; do
  ffmpeg -v verbose -i "$link" -c copy -flags -global_header -hls_time 10 \
         -hls_list_size 6 -hls_wrap 10 -start_number 1 "$track" 
done < <(jq -j '.streams[] | ( .track + "\u0000" + .link + "\u0000" )' streams.json)

Thanks everyone! Yea ended up using this solution. Though I have full control of the input ( the urls in the json file) still it's good to be on the safe side and be careful of inputs. — C.Astraea, Feb 28 '18 at 20:52

Gilles Quénot · Answer 2 · 2018-02-28T17:46:28.297

1

Using bash and jq :

#!/bin/bash

file="$1"
c=0
while true; do
    track=$(jq -r ".streams[$c].track" "$file" 2>/dev/null)
    link=$(jq -r ".streams[$c].link" "$file" 2>/dev/null)
    [[ ! $stream || ! $link ]] && break
    ffmpeg -v verbose -i "$link" -c copy -flags -global_header -hls_time 10 \
        -hls_list_size 6 -hls_wrap 10 -start_number 1 "$track" 
    ((c++))
done

Usage :

./script.bash file.json

edited Feb 28 '18 at 17:46

answered Feb 28 '18 at 16:56

Gilles Quénot

173,512
41
224
223

1

I don't know why anyone would have downvoted this -- it's correct and safe code. – Charles Duffy Feb 28 '18 at 20:47

Gilles Quénot · Answer 3 · 2018-02-28T18:05:59.980

0

Using nodejs to generate the shell commands :

(replace file.json with your own path/file)

#!/bin/bash

node<<EOF
var j=$(<file.json);
for (var i = 0; i<j.streams.length; i++) {
   console.log("ffmpeg -v verbose -i '" + j.streams[i].link + "' -c copy -flags -global_header -hls_time 10 -hls_list_size 6 -hls_wrap 10 -start_number 1 '" + j.streams[i].track + "'");
}
EOF

Output :

ffmpeg -v verbose -i 'http://playertest.longtailvideo.com/adaptive/bipbop/bipbop.m3u8' -c copy -flags -global_header -hls_time 10 -hls_list_size 6 -hls_wrap 10 -start_number 1 '/var/www/html/stream1.m3u8'
ffmpeg -v verbose -i 'https://mnmedias.api.telequebec.tv/m3u8/29880.m3u8' -c copy -flags -global_header -hls_time 10 -hls_list_size 6 -hls_wrap 10 -start_number 1 '/var/www/html/stream2.m3u8'
ffmpeg -v verbose -i 'http://www.streambox.fr/playlists/test_001/stream.m3u8' -c copy -flags -global_header -hls_time 10 -hls_list_size 6 -hls_wrap 10 -start_number 1 '/var/www/html/stream3.m3u8'

edited Feb 28 '18 at 18:05

answered Feb 28 '18 at 17:44

Gilles Quénot

173,512
41
224
223

Dangerous if you don't control the filenames. If someone creates `http://example.com/bobby/$(rm -rf ~)/tables.m3u8`... – Charles Duffy Feb 28 '18 at 17:49
Do you have a precise idea/example of something really dangerous using this, please ? – Gilles Quénot Feb 28 '18 at 17:53
...I'm assuming here that the OP is getting this JSON from a website or API they don't control. Trusting content from a source you don't control to run arbitrary shell commands is a Bad Idea. – Charles Duffy Feb 28 '18 at 17:55
(This is also a bit risky insofar as it executes `file.json`'s contents as code in node.js; if someone knew you were going to do that with a file they were giving you to download -- or was doing a man-in-the-middle attack and wanted to try replacing the contents to see what they could get away with -- they could put more malicious contents in place rather than a well-formed JSON document). – Charles Duffy Feb 28 '18 at 20:39
1

As edited, a URL used to attack this would need to be more like `http://example.com/bobby/$(rm -rf ~)'$(rm -rf ~)'/tables.m3u3` to cover both the case with and the case without surrounding literal single-quotes. – Charles Duffy Feb 28 '18 at 20:51

Passing key and values to external command from jq

3 Answers3

Usage :

Output :