Something blocks threads while accessing fifo from different processes

Question

I have a Perl script and Python script. Perl script feeds Python script by means of FIFO. Python script executes the Perl script and also runs some threads to process the data output from Perl. I read the FIFO directly into pandas DataFrame.

Sometimes it runs, sometimes it is stuck. Perhaps dead lock. What am I doing wrong?

Python code:

from threading import Thread
import os
import subprocess
import numpy as np
import pandas as pd

class MyThread(Thread):
    def __init__(self, fifo_name):
        self.fifo_name = fifo_name
        self.results = None
        super(MyThread, self).__init__()

    def run(self):
        try:
            os.mkfifo(self.fifo_name)
        except FileExistsError:
            pass
        self.results = pd.read_csv(self.fifo_name)

def Main():
    t11= MyThread("fifo1_msg1")
    t11.start()

    t12= MyThread("fifo1_msg2")
    t12.start()

    t21= MyThread("fifo2_msg1")
    t21.start()

    t22= MyThread("fifo2_msg2")
    t22.start()

    # run perl scripts that writes to fifos
    subprocess.Popen(["perl", "dumpData.pl", file1, fifo1_msg1, fifo1_msg2, bufsize=1, stderr=subprocess.STDOUT)
    subprocess.Popen(["perl", "dumpData.pl", file2, fifo2_msg1, fifo2_msg2], bufsize=1, stderr=subprocess.STDOUT)

    t11.join()
    t12.join()
    t21.join()
    t22.join()

    t_plot = Thread(target=make_plots, args=(t11.results, t12.results, t21.results, t22.results))
    t_plot.start()
    t_plot.join()

if __name__ == "__main__":
    Main()

Perl Code:

use warnings;
use strict;
$|++;  # turn on autoflush
sub dump_file
{
    my $filename = shift;
    my $fifo_msg1 = shift;
    my $fifo_msg2 = shift;

    open(my $fifo1, ">", $fifo_msg1 ) || die();
    open(my $fifo2, ">", $fifo_msg2) || die();

    print $fifo1 "date,t,p\n";
    print $fifo2 "time,x,y,z\n";
    # some pseudo code as I don't want to put whole code
    # while not eof, read line..
       # read variables from line 
       if(case_bla) {
           printf $fifo1 "%d,%d,%d\n", date,t,p;
       }
       else {
           printf $fifo2 "%d,%d,%d,%d\n", time,x,y,z;
       }
    # end while
    close($fifo1);
    close($fifo2);
}
dump_file($inputfile, $fifo_msg1, $fifo_msg2);

For one, you're not autoflushing the Perl output streams, so the consumer (Python script) will not receive anything until there are enough (8K? 4K? 16K?) bytes to fill the output buffer. — mob, Jun 12 '18 at 13:12
@mob, Before 5.14, Perl read in 4 KiB blocks. Since 5.14, the size of the blocks is configurable when you build perl, with a default of 8 KiB. — ikegami, Jun 12 '18 at 17:26
`$_->autoflush(1) for $fifo1, $fifo2;` will disable buffering on the handles. — ikegami, Jun 12 '18 at 17:27
When opening $fifo2, you are opening `$fifo_msg`, which should be undefined. You probably meant `$fifo_msg2`. Always put `use strict; use warnings;` at the beginning of your perl scripts. — bytepusher, Jun 12 '18 at 18:39
what I forgot to paste here was, this line at top of my perl script: `$|++; # turn on autoflush` does it do the same job? — nurp, Jun 13 '18 at 13:18
No it doesn't. It enables `autoflush` on the *currently selected output handle*, which is `STDOUT` unless you have changed it. You need to add `$fifo1->autoflush` and `$fifo2->autoflush` after they are opened. And please *copy and paste* your code into questions: typing stuff in and hoping it's close enough will lead to incorrect and misleading answers. — Borodin, Jun 13 '18 at 13:43
What's more, the code you have shown won't even compile because you have `use strict` in place but haven't declared `$fifo_msg`. Please be much more careful. — Borodin, Jun 13 '18 at 13:46
Thanks, it doesn't block anymore after @ikegami's suggestion. Thanks all! — nurp, Jun 13 '18 at 13:55
@Borodin, thanks for warning. I will put a compiling code next time. — nurp, Jun 13 '18 at 14:00
Now I call that Main from within another Python script multiple times and it blocks in the second call. It always runs first one, and always blocks second one. i put prints everywhere but is always different where it blocks. It is pandas causing it. I now changed it to basic file read and it works fine. Just in case anyone finds this thread and tries themselves, I wanted to write. — nurp, Jun 15 '18 at 20:15

Something blocks threads while accessing fifo from different processes

0 Answers0