6

I'm trying to run a series of shell commands with Perl6 to the variable $cmd, which look like

databricks jobs run-now --job-id 35 --notebook-params '{"directory": "s3://bucket", "output": "s3://bucket/extension", "sampleID_to_canonical_id_map": "s3://somefile.csv"}'

  1. Splitting the command by everything after notebook-params

my $cmd0 = 'databricks jobs run-now --job-id 35 --notebook-params '; my $args = "'{\"directory\": \"$in-dir\", \"output\": \"$out-dir\", \"sampleID_to_canonical_id_map\": \"$map\"}'"; my $run = run $cmd0, $args, :err, :out;

Fails. No answer given either by Databricks or the shell. Stdout and stderr are empty.

  1. Splitting the entire command by white space

    my @cmd = $cmd.split(/\s+/); my $run = run $cmd, :err, :out

Error: Got unexpected extra arguments ("s3://bucket", "output": "s3://bucket/extension", "sampleID_to_canonical_id_map": "s3://somefile.csv"}'

  1. Submitting the command as a string my $cmd = "$cmd0\"$in-dir\", \"output\": \"$out-dir\", \"sampleID_to_canonical_id_map\": \"$map\"}'";

again, stdout and stderr are empty. Exit code 1.

this is something about how run can only accept arrays, and not strings (I'm curious why)

If I copy and paste the command that was given to Perl6's run, it works when given from the shell. It doesn't work when given through perl6. This isn't good, because I have to execute this command hundreds of times.

Perhaps Perl6's shell https://docs.perl6.org/routine/shell would be better? I didn't use that, because the manual suggests that run is safer. I want to capture both stdout and stderr inside a Proc class

EDIT: I've gotten this running with shell but have encountered other problems not related to what I originally posted. I'm not sure if this qualifies as being answered then. I just decided to use backticks with perl5. Yes, backticks are deprecated, but they get the job done.

con
  • 5,767
  • 8
  • 33
  • 62
  • 2
    Why do you split on `/\s+/` ? The `--notebook-params` expects to be followed by one string, which is `''{"directory": "s3://bucket", "output": "s3://bucket/ext...'` - but that contains spaces. So your splitting on spaces is a bad/weird idea. Most likely you will want to involve a shell by using `shell $cmd` instead. – Corion Jan 24 '19 at 16:02
  • I don't understand the logic about why an array should be submitted to `run` and not a string. This makes 0 sense to me. I've tried what you said, and that doesn't work either. – con Jan 24 '19 at 16:14
  • 2
    Your simplistic `split` destroys the quoted parameter `'{"directory": ... .csv"}'`. That needs to appear as _one_ parameter on the command line, not as multiple. – Stefan Becker Jan 24 '19 at 16:21
  • @StefanBecker I've tried that too, and that fails – con Jan 24 '19 at 16:22
  • So you tell me you have tried `my @cmd = ( 'databricks', 'jobs', 'run-now', '--job-id', '35', '--notebook-params', '{"dire... file.csv"}' );` (i.e. by-pass the split for a test), and it still fails? – Stefan Becker Jan 24 '19 at 16:25
  • @StefanBecker yes – con Jan 24 '19 at 16:26
  • Then you should read the documentation for `run`. The input seems to be passed through a shell and hence you'll need additional quoting. The other choice is to find the option to disable shell usage in `run`, if that is possible. But then you'll need to supply absolute path to the command as you will have no `$PATH` processing. – Stefan Becker Jan 24 '19 at 16:27
  • The [perl6 documentation for "run"](https://docs.perl6.org/routine/run) states that no shell is involved, hence a single array entry should appear as single argument to the called program. Please copy & paste the by-pass approach to your code example. – Stefan Becker Jan 24 '19 at 16:34
  • @StefanBecker I've given my 3 different approaches – con Jan 24 '19 at 16:54
  • Please use code blocks for code, not quote blocks. – Stefan Becker Jan 24 '19 at 17:01
  • Variants 1 & 3 are wrong, because the first argument to `run` must be the command name (i.e. `databricks`) nothing else. Variant 2 passes `$cmd` to run, not `@cmd` – Stefan Becker Jan 24 '19 at 17:07

2 Answers2

6

I'm trying to run a series of shell commands

To run shell commands, call the shell routine. It passes the positional argument you provide it, coerced to a single string, to the shell of the system you're running the P6 program on.

For running commands without involving a shell, call the run routine. The first positional argument is coerced to a string and passed to the operating system as the filename of the program you want run. The remaining arguments are concatenated together with a space in between each argument to form a single string that is passed as a command line to the program being run.

my $cmd0 = 'databricks jobs run-now --job-id 35 --notebook-params ';

That's wrong for both shell and run:

  • shell only accepts one argument and $cmd0 is incomplete.

  • The first argument for run is a string interpreted by the OS as the filename of a program to be run and $cmd0 isn't a filename.

So in both cases you'll get either no result or nonsense results.

Your other two experiments are also invalid in their own ways as you discovered.

this is something about how run can only accept arrays, and not strings (I'm curious why)

run can accept a single argument. It would be passed to the OS as the name of the program to be run.

It can accept two arguments. The first would be the program name, the second the command line passed to the program.

It can accept three or more arguments. The first would be the program name, the rest would be concatenated to form the command line passed to the program. (There are cases where this is more convenient coding wise than the two argument form.)

run can also accept a single array. The first element would the program name and the rest the command line passed to it. (There are cases where this is more convenient.)

I just decided to use backticks with perl5. Yes, backticks are deprecated, but they get the job done.

Backticks are subject to code injection and shell interpolation attacks and errors. But yes, if they work, they work.

P6 has direct equivalents of most P5 features. This includes backticks. P6 has two variants:

  • The safer P6 alternative to backticks is qx. The qx quoting construct calls the shell but does not interpolate P6 variables so it has the same sort of level of danger as using shell with a single quoted string.

  • The qqx variant is the direct equivalent of P5 backticks or using shell with a double quoted string so it suffers from the same security dangers.

raiph
  • 31,607
  • 3
  • 62
  • 111
  • "The first argument for run is a string interpreted by the OS as the filename of a program to be run and $cmd0 isn't a filename." I didn't know this, this was the key missing information – con Jan 24 '19 at 20:28
  • 2
    OSes all have a function call where the calling code gives a name, and the OS goes off and finds and runs a program of that name. You can also pass a string that's passed on to the program so it has input arguments. The P6 `run` function corresponds to this raw OS level function so has the program name, plus zero or more concatenated arguments. – raiph Jan 24 '19 at 20:38
  • 2
    Shells sit on top of OSes. Using a shell involves writing a line of text and hitting the Enter key. It's just one string. It might start with a program name or it might just be some shell code. So the P6 `shell` routine just takes a single string and passes that to the OS's default shell. (I believe this is configurable.) – raiph Jan 24 '19 at 20:39
4

Two mistakes:

  • the simplistic split cuts up the last, single parameter into multiple arguments
  • you are passing $cmd to run, not @cmd
use strict;

my @cmd = ('/tmp/dummy.sh', '--param1', 'param2 with spaces');
my $run = run @cmd, :err, :out;

print(@cmd ~ "\n");
print("EXIT_CODE:\t" ~ $run.exitcode  ~ "\n");
print("STDOUT:\t"    ~ $run.out.slurp ~ "\n");
print("STDERR:\t"    ~ $run.err.slurp ~ "\n");

output:

$ cat /tmp/dummy.sh
#!/bin/bash
echo "prog: '$0'"
echo "arg1: '$1'"
echo "arg2: '$2'"
exit 0

$ perl6 dummy.pl
/tmp/dummy.sh --param1 param2 with spaces
EXIT_CODE:      0
STDOUT: prog: '/tmp/dummy.sh'
arg1: '--param1'
arg2: 'param2 with spaces'

STDERR:

If you can avoid generating $cmd as single string, I would generate it into @cmd directly. Otherwise you'll have to implement complex split operation that handles quoting.

Stefan Becker
  • 5,695
  • 9
  • 20
  • 30