0

Original Title: Indirect parameter substitution breaks when the script is sourced (zsh)

zsh 5.7.1 (x86_64-apple-darwin19.0)

GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu)

I’m developing a shell script on a Mac and I’m trying to keep it portable between bash & zsh, so array indexing is a consideration. I know that I can set KSH_ARRAYS to get indexing to start at 0, but I decided to query the OS for the shell that’s in use and set the start index accordingly, which led to the issue described below.

It made sense (to me anyway!) to use indirect expansion, which is what led to the problem. Consider the script indirect.sh:

#! /bin/bash

declare -r ARRAY_START_BASH=0
declare -r ARRAY_START_ZSH=1

declare -r SHELL_BASH=0
declare -r SHELL_ZSH=1

# Indirect expansion is used to reference the values of the variables declared
# in this case statement e.g. ${!ARRAY_START}
case $(basename $SHELL) in
  "bash" )
    declare -r SHELL_ID=SHELL_BASH
    declare -r ARRAY_START=ARRAY_START_BASH
    ;;

  "zsh" )
    declare -r SHELL_ID=SHELL_ZSH
    declare -r ARRAY_START=ARRAY_START_ZSH
    ;;

  * )
    return 1
    ;;

esac

echo "Shell ID: ${!SHELL_ID} Index arrays from: ${!ARRAY_START}"

It works fine when run from the command line while in the same directory:

<my home> ~ % echo "$(./indirect.sh)"
Shell ID: 1 Index arrays from: 1

Problems arise when I source the script:

<my home> ~ % echo "$(. ~/indirect.sh)"
/Users/<me>/indirect.sh:28: bad substitution

I don’t understand why sourcing the script changes the behavior of the parameter expansion.

Is this expected behavior? If so, I’d be grateful if someone could explain it and hopefully, offer a work around.

latentcode
  • 43
  • 6
  • If you run it from the command line by just entering the script name, it is executed by bash, but when you source it from within your zsh, it is executed by zsh. In general, you don't expect programs to produce same results when interpreted as different languages. Note also that querying the value of `SHELL` does not reliably tell you what shell you are running. For instance, when I run zsh on my Mac, SHELL is set to bash. In any case, put a `set -x` in your script, so that you can see where and why things go wrong. – user1934428 Jun 19 '20 at 08:08
  • `zsh` uses different syntax for indirect parameter expansion than `bash`.: `${(P)SHELL_ID}`. – chepner Jun 19 '20 at 10:57
  • When you execute the script, the shebang is used. When you use `.`, the current shell (`zsh`) is used, ignoring the shebang. – chepner Jun 19 '20 at 10:58
  • You have two problems: 1) In general, you cannot write a script that will work identically in `bash` and `zsh`, unless you stick to POSIX-compliant features. 2) In general, you shouldn't try to write a script that can be executed or sourced. You generally source a file because it needs to affect the caller's environment, which executing the script *cannot* do. – chepner Jun 19 '20 at 11:00
  • @chepner: Yes of course, the shebang. If it weren’t for all the time wasted, it would have been funny! The zsh syntax for indirect expansion is indicated in the docs. Thought I’d looked there, but clearly I didn’t. Of course the problem was more fundamental. I’m not quite sure the protocol for posting an answer, so I answered myself. Please do let me know if there’s an error and I will edit. – latentcode Jun 20 '20 at 17:52
  • @user1934428: I appreciate your response as well. It helped me glean a better understanding – latentcode Jun 20 '20 at 17:53
  • @user10248678 I re-added the bash tag given that this applies there as much as zsh. – latentcode Jun 20 '20 at 17:53

2 Answers2

1

The problem described in the original post has nothing to do with indirect expansion. The difference in behavior is a result of different shells being invoked depending on whether the script is “executed” or “sourced”. These differences reveal the basic flaw in deriving the shell from the $SHELL variable that underpins the script's design. If the shell defined in $SHELL does not match the shebang, the script will fail either when sourced or executed. An explanation follows.

Indirect expansion doesn’t offer value in the given scenario because values could just as easily be assigned directly. They’ll have to be assigned that way regardless given the different syntax used for indirect expansion between shells. In fact, other syntax differences between shells makes the entire premise for detecting the shell moot! However, putting that aside, the difference in behavior is a result of different shells being invoked based on whether the script is “executed” or “sourced”. The behavior of sourcing is well documented with numerous explanations on the web, but for context here’s how it works:

Executing a Script Use the “./“ syntax to execute a script. When run this way, the script executes in a sub-shell. Any changes the script makes to it’s shell are applied to the sub-shell, not the shell in which the script was launched, so those changes are lost when the shell exits because the sub-shell in which it executed is destroyed as well. For example, if the script changes the working directory, it does so in the sub-shell. The working directory of the main shell that launched the script is unchanged when the script terminates. If you want to make changes to the shell in which the script was launched, it must be sourced.

Sourcing a Script Use the “source “ syntax to source a script. When run this way, the script essentially becomes an argument for the source command, which handles invoking the appropriate execution. Some shells (e.g. ksh) use a single period “.” instead of “source”.

When a script is executed with the “./“ syntax, the shebang at the top of the file is used to determine which shell to use. When a script is sourced, the shebang is ignored and the shell in which the script is launched is used instead. Also note that the period that appears in the “./“ command syntax used to execute a script, is not related to the period that’s occasionally used as an alias for the source command.

The script in the post uses bash in the shebang statement, so it works when executed because it’s run using bash. When it’s sourced from zsh, it encounters the incorrect indirect expansion syntax:

“${!A_VAR}"

The correct syntax is:

"${(P)A_VAR}"

However, correcting the syntax won’t help because it will then fail when executed. The shebang will invoke bash and the syntax will be wrong again. That renders indirection useless for accessing a variable designed to indicate the shell in use. More importantly, a design based on querying an environment variable for the shell is flawed due to differences in the shell that’s ultimately used depending on whether the script is executed or sourced.

latentcode
  • 43
  • 6
1

To add to your answer (what I'm going to say is too long for a comment), I can not think of any application, why your script could be useful if not sourced. Actually, I came accross the need of such a script by myself in exactly one occasion:

Since I use as interactive shell not only zsh, but also sometimes bash, so I have written my .zshrc and .bashrc to set up everything (including defining variables and shell functions for interactive use). In order to safe work, I try to put code which works under both bash and zsh into a single file (say: .commonrc), and my .zshrc and .bashrc have inside them a

source .commonrc

While many things are so different in bash and zsh, that I can't put them into .commonrc, some can, provided I do some tweaking. One reason for headache is obviously the different indexing of arrays, which you seemingly try to solve. So I have also a similar feature. However, I don't nee ca case construct for this. Instead, my .bashrc looks like this (using your naming of the variables):

...
declare -r ARRAY_START=0
source .commonrc
...

and my .zshrc looks like this:

...
declare -r ARRAY_START=1
source .commonrc
...

Since it does not happen that the .bashrc is run from a zsh and vice versa, I don't need to query what kind of shell I have.

user1934428
  • 19,864
  • 7
  • 42
  • 87