1

So, I have a socket.io connection with a React.js frontend that sends a blob of wav audio data every 100 ms to my Flask backend. On my backend, I am maintaining a thread safe queue

audio_buffer = queue.Queue()

@socketio.on('stream')
def stream(data):
    #Data is a properly formatted wav file in bytes
    audio_buffer.put(data)

Now, my goal is that while my backend is taking the streamed data from the frontend, I also want to send the data to google ASR from my backend at the same time. I am basing my code here on the documentation here: https://cloud.google.com/speech-to-text/docs/streaming-recognize

Basically, I am going to create a generator with a reference to the queue and give that to the google api.

audio_generator = stream.generator() #Here, I wrote a new method for creating a generator from audio_buffer
requests = (
        speech.StreamingRecognizeRequest(audio_content=content)
        for content in audio_generator
    )

responses = client.streaming_recognize(streaming_config, requests)

# Now, put the transcription responses to use.
listen_print_loop(responses)

Now the issue that I have is that essentially, if I try to send things to the google api, my backend stops listening to the frontend and just waits for the queue to be populated, ignoring socket.io requests from the client. For example, I tried to have the client to send a start command for the backend to create the generator and start streaming data to google while the client also sends data to populate the queue through the stream method. What happened was that it got stuck on trying to stream data to google and never got around to actually listening to the data blobs.

My question is that should I attempt to use threading or celery to try and run the google api code simultaneously? Basically, I am thinking of instantiating a thread or worker to do this google api streaming recognize, and the main flask thread would keep listening to the client and populate the queue. Is this a sound approach? Is there a library that implements this for me?

Update 5/16: So I am attempting to run the google api on a standard thread. My code is pretty simple. On start, I start a thread with

yourThread = threading.Thread(target=recognize, args=[stream])

and my recognize code passes in the thread safe queue and actually sends the code to the google api properly. However, my issue now is that emits that I put in the code don't actually make it to the client. My code is based on the google api code:

for response in responses:
    #Some other code from the tutorial
    socketio.emit("transcription", transcript)

that last emit call never actually gets to the frontend at all. Now, looking around, I found the same issue here: https://github.com/miguelgrinberg/Flask-SocketIO/issues/1239

import httplib2shim
httplib2shim.patch()

but when I tried the fix over there, and switched async_mode to eventlet, the google api calls still worked, but none of the emits still get to the client

dragon18456
  • 41
  • 7
  • Are you using eventlet or gevent with this project? These are likely incompatible with Google API calls, which block the async loop. Try running your application in threading mode, without eventlet/gevent. – Miguel Grinberg May 16 '21 at 10:46
  • Yeah, so when I try to run it with a regular thread, the google api calls work, but any emits that I call from the thread just fail. However, will websockets from my client still work without eventlet or gevent? Basically, right now, I instantiate a regular thread and the google api call is perfectly fine, but I wanted to add some emits inside of the listen_print_loop to send the response to react, and they just don't go to the frontend. I then tried to run it with socketio.start_background_task, and it just hangs for a really long time and then the worker times out – dragon18456 May 16 '21 at 21:04
  • I saw your response on https://stackoverflow.com/questions/56157266/how-can-i-use-websocket-transport-and-async-mode-threading-in-flask-socketio-at. Is there a way for my frontend and backend to work with socket.io (with websockets) and to use threading at the same time? – dragon18456 May 16 '21 at 21:10
  • As a reference, my frontend code is using const socketio = io('http://127.0.0.1:5000', { transports: ['websocket'], upgrade: false }); and my backend code is socketio = SocketIO(app, cors_allowed_origins="*", async_handlers=True). If I blindly remove transports: [websocket] from the frontend and enable threading mode, the requests from the frontend to the backend get really slow and laggy. On the other hand, right now, without threading mode on, none of the emits that happen in my thread get to my frontend client at all. – dragon18456 May 16 '21 at 21:17

1 Answers1

1

Okay, so I solved my own issue. The problem is just that flask_socketio + python google speech api just doesn't work due to incompatibility between eventlet/gevent and grpc.

My solution was to switch to using regular websockets. By using a regular websocket on the frontend and flask_sockets on the backend, I managed to get the google cloud speech python api to work without issue. I've placed some sample code for this on my github: https://github.com/dragon18456/flask-sockets-google-speech-api

dragon18456
  • 41
  • 7