Python - how to share real-time data with another script

Question

I am trying to find easiest way to share real time variable from one script to another. First script will read seonsor data and other will make calculations based on real-time data from the first script. I want to run them separetly. I want to be able to kill second script and run it again without any problems.

I would like to have second script to print real-time data whenever it is started.

---

Update:

I have finaly got some time to play with os.pipe(). I have managed to run some scripts that use os.fork() but when I tried to split one script into two separate programs I start haveing some issues.

Program I have initated and was working:

    #!/usr/bin/python
     import os, sys
     r, w = os.pipe() 
     processid = os.fork()
     if processid:
           os.close(w)
           r = os.fdopen(r)
           print("Parent reading")
           str = r.read()
           print("text =", str)   
           sys.exit(0)
     else:
           os.close(r)
           w = os.fdopen(w, 'w')
           print("Child writing")
           w.write("Text written by child...")
           w.close()
           print("Child closing")
           sys.exit(0)

Based on that script I tried to write my own separate scripts.

First script that prints time to pipe:

   #!/usr/bin/python
   import os, sys, time
   stdout = sys.stdout.fileno()
   r, w  = os.pipe()
   #os.close(r)
   w = os.fdopen(w, 'w')
   i = 0
   while i < 1000:
         i = i + 1
         w.write('i' + " ")
         time.sleep(1)

Second script that reads time from pipe:

   #!/usr/bin/python

   import os, sys, time
   r, w = os.pipe()
   r = os.fdopen(r)
   str = r.read()
   print(str)

When I try to run my scripts nothing happens. Any suggestions what am I doing wrong? Maybe I missed some details about standard input and output and os.pipe()?

Answer 1

Edit: Just to illustrate a crude example of a possible poison pill approach. Unfortunately, python's (multiprocessing) Queues do not implement a peek call so I used a second queue:

Writer:

import time
import sys

def write(queue_msg, queue_term):
     i = 0
     sys.stdout = open("writer" + ".out", "w")
     sys.stderr = open("writer_err" + ".out", "w")
     stop_writing = True
     while i<1000:
        if not(stop_writing):
            print("Writer at iteration ", i)
            queue_msg.put("Iteration " + str(i)) # put a value inside the queue
            i += 1
        time.sleep(1)
        q_term_val = queue_term.get() if not(queue_term.empty()) else "CONTINUE" #Check that the queue is not empty otherwise put a non important value
        print(q_term_val)
        if q_term_val == "START":
            print("Start working")
            stop_writing = False 
        elif q_term_val == "STOP":
            print("Stop working")
            stop_writing = True 
        sys.stderr.flush() #flush stdout and stderr to disk 
        sys.stdout.flush()

Reader:

import time
import sys

def read(queue_msg, queue_term):
    sys.stdout = open("reader" + ".out", "w") #redirect process stdout to file "reader.out"
    sys.stderr = open("reader_err" + ".out", "w")
    queue_term.put("START")
    while True:
        value = queue_msg.get()
        print(value) # write value to reader.out filer as mentioned above
        if int(value.split(" ")[1]) > 10:
            print("Stop working, I am tired...")
            queue_term.put("STOP")
            time.sleep(5) #wait 5 seconds before sending start to the worker
            queue_term.put("START")
        time.sleep(1)
        sys.stderr.flush()
        sys.stdout.flush()

Main:

import reader
import writer
import multiprocessing as mp

def main():

    q_msg = mp.Queue()
    q_term = mp.Queue()
    r = mp.Process(target=reader.read, args=(q_msg, q_term,))
    w = mp.Process(target=writer.write, args=(q_msg, q_term,))
    r.start()
    w.start()
    print("Processes started, check that in your folder there are a reader.out and a writer.out files")

if __name__ == "__main__": # "workaround" needed on Windows
    main()

And you should have the reader.out file (also writer.out) in your folder showing that the reader indeed read values written to the queue by the writer and in "writer.out", you can see that the writer stops writing to the queue when he received the poison pill. And the reader after the time.sleep(5) call starts again from the right position.