4

I have a bash script that makes a cURL request and writes the output to a file called resp.txt. I need to create an exclusive lock so that I can write to that file and not worry about multiple users running the script and editing the text file at the same time.

Here is the code that I expect to lock the file, perform the request, and write to the text file:

(
flock -e 200
curl 'someurl' -H 'someHeader' > resp.txt
) 200>/home/user/ITS/resp.txt

Is this the correct way to go about this? My actual script is a bit longer than this, but it seems to break when I add the flock syntax to the bash script. My understanding is that this creates a subshell that can write to the locked file. I'm just not sure if I need to add the file descriptor to the curl request like this:

curl 'someurl' -H 'someHeader' 200>resp.txt

If someone could explain how these file descriptors work and let me know if I am locking the file correctly that would be awesome!

Jonathan Hall
  • 75,165
  • 16
  • 143
  • 189
RattleyCooper
  • 4,997
  • 5
  • 27
  • 43
  • Why do you pipe the curl response into a file? – hek2mgl Feb 11 '15 at 15:53
  • How does your script "break" when you add this? – Etan Reisner Feb 11 '15 at 15:55
  • Because I need to extract an ID from the response to use in the next request. Is there a reason I shouldn't do that, or a better way to accomplish the same thing? Basically I have a few requests that I need to get info from. I guess I could rewrite some stuff, it's just getting to be a really long line with the cURL request and whatnot. Can i use a `|` and continue the script on the next line? I though it had to be on one line. – RattleyCooper Feb 11 '15 at 15:56
  • If you want curl to write into the lock file then you want `>&200` on the curl command line. – Etan Reisner Feb 11 '15 at 15:58
  • @EtanReisner, well I'm guessing that because it creates a subshell that some variables that I set within the lock are not accessible to the bash script, but like I said, I'm new to `flock`, and to be honest, I'm new to bash scripting as well. – RattleyCooper Feb 11 '15 at 15:58
  • 1
    That still doesn't tell me *what* broke. Do you get an error? Do kittens eat your face? – Etan Reisner Feb 11 '15 at 15:59
  • @EtanReisner, well it kind of does. If there was an error, I would tell you what that is, but there are not errors, so that is that. One of the requests that I make requires an ID that is returned by the previous request. That is what is not working. I'm thinking that if the variable is set in the subshell and the bash script cannot access it, then the second request will not go through as expected due to the fact that it is missing the ID required.. – RattleyCooper Feb 11 '15 at 16:01
  • 1
    So the issue is that you cannot read the request out of `resp.txt` when you add the flock? I'm betting `curl` can't write to the file then. Try `>&200` like I suggested in a previous comment to get it to write to the `flock`ed fd. Also, you would not believe how often people leave out the text of error messages they get. My intention was not to insult you there. – Etan Reisner Feb 11 '15 at 16:04
  • @EtanReisner, gotcha. No worries, I appreciate the help. I will see if that works! – RattleyCooper Feb 11 '15 at 16:06

1 Answers1

17

flock supports a command argument (-c). You can use the following one liner in your case:

flock resp.txt -c "curl 'someurl' -H 'someHeader' > resp.txt"

The above command tries to obtain an exclusive (write) lock for resp.txt. It waits unless the lock could be obtained if another script has locked the file.

As soon as the lock can be obtained it executes the curl command and releases the lock again.

If you want to perform multiple commands while obtaining the lock, I suggest to use a lockfile different from the file you are writing to.

Like this:

( 
    flock -n 200 || exit 1
    # commands executed under lock 
    sleep 3
    cat <<< 'hello' > test.txt
) 200> /path/to/lockfile

The above creates a sub shell () and redirects the output of that sub shell to /path/to/lockfile using the file descriptor number 200. The sub shell will inherit the file descriptor 200 from the base shell.

Inside the sub shell flock tries to obtain a lock for file descriptor 200. after obtaining the lock the following commands get executed.

hek2mgl
  • 152,036
  • 28
  • 249
  • 266
  • This doesn't help if you want the lock held for the subsequent commands though. – Etan Reisner Feb 11 '15 at 16:05
  • @EtanReisner Yes, in the question was just a single command was executed. In this case you can use the `-c` command. I can give an example how to use it with multiple commands... – hek2mgl Feb 11 '15 at 16:07
  • @hek2mgl, I would appreciate that! The documentation gives an example similar to what I have in my question, but for a beginner to bash scripting it's difficult to understand because the man page doesn't go very in depth. – RattleyCooper Feb 11 '15 at 16:23
  • Sorry, I almost forgot that :) ... I'm AFK for a while. Can explain more later ... – hek2mgl Feb 11 '15 at 16:54