Why does my program return an error stating the file doesn't exist when it does?

Question

I'm writing the basics of a custom scheduling tool, which will read a config file for "jobs" and add them to the schedule to run the periodically. It's very basic for now, like a proof of concept before refactoring and some additional more advanced features.

When I try to run this program, it reports that the script is not found. The script "test1.sh" does exist in the path where I'm trying to run it from, and it has execute permissions.

I'm getting the following error, but can't explain it or work it out as the script does exist at the path I'm running it:

-> ./scheduler-example
2021/08/16 12:48:54 fork/exec /Users/me/scheduler-example/scripts/test1.sh: no such file or directory

The scheduler code:

package main

import (
    "io"
    "log"
    "os"
    "os/exec"
    "path/filepath"
    "time"

    "github.com/go-co-op/gocron"
    "gopkg.in/yaml.v3"
)

type Config struct {
    GlobalLog string `yaml:"GlobalLog"`
    Jobs      []Job  `yaml:"Jobs"`
}

type Job struct {
    Name      string `yaml:"Name"`
    Command   string `yaml:"Command"`
    Frequency string `yaml:"Frequency"`
    Tag       string `yaml:"Tag,omitempty"`
    Log       string `yaml:"Log,omitempty"`
}

const ROOT string = "/Users/me/scheduler-example"

func main() {
    // STEP1: Parse the config file
    configFile := filepath.Join(ROOT, "config", "scheduler-example.yaml")
    f, err := os.Open(configFile)
    if err != nil {
        log.Fatalln(err)
    }

    configData, err := io.ReadAll(f)
    if err != nil {
        log.Fatalln(err)
    }

    c := Config{}
    err = yaml.Unmarshal(configData, &c)
    if err != nil {
        log.Fatalln(err)
    }

    // STEP2: Validate the config
    if c.GlobalLog == "" {
        log.Fatalln("Global log not defined")
    }
    if c.Jobs == nil {
        log.Fatalln("No jobs defined")
    }
    for _, j := range c.Jobs {
        if j.Name == "" {
            log.Fatalln("Job name not defined")
        }
        if j.Command == "" {
            log.Fatalln("Job command not defined")
        }
        if j.Frequency == "" {
            log.Fatalln("Job frequency not defined")
        }
    }

    // STEP3: Create the scheduler and add jobs
    s := gocron.NewScheduler(time.UTC)

    for _, j := range c.Jobs {
        script := filepath.Join(ROOT, "scripts", j.Command)
        cmd := exec.Command(script)
        cmdLog := filepath.Join(ROOT, "logs", j.Log)
        l, err := os.OpenFile(cmdLog, os.O_WRONLY|os.O_APPEND|os.O_CREATE, 0666)
        if err != nil {
            log.Fatalln(err)
        }
        cmd.Stdout = l
        cmd.Stderr = l
        freq := j.Frequency
        tag := j.Tag
        s.Every(freq).Tag(tag).Do(func() {
            err = cmd.Run()
            if err != nil {
                log.Fatalln(err)
            }
        })
    }

    // STEP4: Run the scheduler
    s.StartBlocking()
}

Config file:

GlobalLog: /tmp/scheduler-example.log
Jobs:
  - Name: test1
    Command: test1.sh
    Frequency: 5s
    Tag: test1
    Log: test1.log
  - Name: test2
    Command: test2.sh
    Frequency: 5s
    Tag: test2
    Log: test2.log

Directory structure:

-> tree .
.
├── config
│   └── scheduler-example.yaml
├── go.mod
├── go.sum
├── logs
│   ├── test1.log
│   └── test2.log
├── scheduler-example.go
└── scripts
    ├── test1.sh
    └── test2.sh

3 directories, 8 files

The test1.sh script:

#!/bin/env bash

for i in {1..100}; do
    echo i
    sleep 10
done

Thanks for any and all help!

Answer 1

according to Go documents

Package exec runs external commands. It wraps os.StartProcess to make it easier to remap stdin and stdout, connect I/O with pipes, and do other adjustments.

It starts an isolated OS process. what you need is a terminal session or a direct bash call.

In Linux when you call ./script.sh it actually means /bin/sh script.sh. ./script.sh works because the terminal (or CMD for .bat files on windows) treats the file as pure bash commands, line by line. so to run user-defined bash files we may use

cmd := exec.Command("/bin/sh",script)

or

cmd := exec.Command("bash",script)

UPDATE: if this approach worked for you, read torek's answare to learn why exactly this works.

Actually what I said about "./script.py beeing bash ./script.sh is" is not quiet right

Answer 2

Here's a minor technical correction to OZahed's answer, not really relevant to the Go language, but useful to know if you are writing code on Linux and Unix systems.

In Linux when you call ./script.sh it actually means /bin/sh script.sh.

This is not quite right.

On a Unix-like system, when you're sitting at a terminal prompt ($ or > or % or whatever you have your prompt set to—lots of people use magic prompt-setters so that they get their current working directory plus perhaps some Git repository information or other useful items), you'll enter a command as:

$ cmd arg1 arg2

for instance. The shell you are using—/bin/sh, /bin/bash, /usr/local/bin/bash, dash, tcsh, fish, etc.—is responsible for breaking up the entered line(s) and attempting to run one or more processes. In general, they'll break this into cmd, arg1, and so on. If some of these words contain shell metacharacters (*, $, and so on), they will do their own special actions with those metacharacters. These can get quite complicated, but they are all up to that particular shell: the syntax used in bash differs from that in tcsh, for instance, in a number of important ways.

In any case, once this shell has gotten past the point of splitting up arguments and getting things ready to invoke the same execve system call that exec.Cmd will use in Go, the shell generally does invoke execve, often after using a $path or $PATH variable to search for the first or best executable to execve (again in a shell-dependent manner). This system call:

• requires that the supplied path name a file;
• requires that the named file be marked with execute permissions; and
• requires that the named file contain data that the OS itself considers executable.

If any of these three tests fail, the execve system call itself will fail.

It's at this point that Go's exec.Cmd and the shells tend to diverge rather sharply. If the file exists and is marked executable, but execve fails, a shell will usually do one or both of the following:

• open and read (perhaps just part of) of the file to try to guess which shell, if any, might run this file, then
• run that shell, or some default shell, on that file.

It's that last step that results in /bin/sh ./script.sh, for instance. If the script appears to be a /bin/sh script, your shell—even if it's tcsh—should use /bin/sh ./script. If the script appears to be a bash script, your shell should locate the bash program and run that program with ./script.sh as an argument.

The key trick to know about executable scripts: #!

Note that the above requires that we get an error from the OS-level execve system call. We can avoid this error, on any modern Unix system, by starting our script with a special line. This special line takes the form of two characters, # and !, followed by optional white space, followed by the path name of an interpreter program, followed by more optional white space and—depending on the particular OS—one or more arguments.

That is, if we write our shell script as:

#! /bin/sh
echo this was run by /bin/sh

and make it executable, the execve system call, applied to this script, acts as if it were an execve system call invoking /bin/sh followed by the path name of this script. So if the path name we used was ./script, we get the effect of running /bin/sh ./script.

The /bin/sh part came out of the script, so we get to control the exact path of the interpreter. If we write an awk script, for instance, and if the awk interpreter is in /usr/bin/awk, then:

#! /usr/bin/awk

is the correct first line.

Amusingly, this gives us the ability to write a self-removing script:

#! /bin/rm

When run, this script removes itself. (The remainder of the file, if any, is irrelevant: the /bin/rm command simply removes the named file.) Using /bin/mv as the interpreter produces a self-renaming script.

A common trick these days, due to programs like python being in /usr/bin on some systems and /usr/local/bin on others, is to use the POSIX env command to locate the binary for the interpreter:

#! /usr/bin/env python

invokes /usr/bin/env python ./script.py. The env command uses $PATH to locate the python command and then calls execl (a C library wrapper for the execve system call) with /usr/local/bin/python ./script.py or whatever is appropriate. If a system has both python2 and python3 installed, with python2 and python3 invoking the specific variant, #! /usr/bin/env python3 ensures that we find a python3 interpreter, rather than a python2 interpreter.

Had you written your script with the appropriate #! line, you would never have asked the question that led to these answers. :-)