I'm trying to adapt this Java codeLabs to Python / SocketIO (Flask Socket IO on server side). My socket is working, I'm able to pass both the sample rate (and to retrieve on server side), and the audio data.
The problem is that the format of my audio data is not correct. I send it this way:
// Create a node that sends raw bytes across the websocket
var scriptNode = context.createScriptProcessor(4096, 1, 1);
// Need the maximum value for 16-bit signed samples, to convert from float.
const MAX_INT = Math.pow(2, 16 - 1) - 1;
scriptNode.addEventListener('audioprocess', function(e) {
var floatSamples = e.inputBuffer.getChannelData(0);
// The samples are floats in range [-1, 1]. Convert to 16-bit signed
// integer.
socket.emit('audiodata', Int16Array.from(floatSamples.map(function(n) {
return n * MAX_INT;
})));
This is the code from the codeLabs but I use socket.emit
instead of socket.send
.
The python code to handle and send it to Speech API is this one:
credentials = service_account.Credentials.from_service_account_file(CONFIG[u"service_account"])
client = speech.SpeechClient(credentials=credentials)
config = types.RecognitionConfig(
encoding=enums.RecognitionConfig.AudioEncoding.LINEAR16,
sample_rate_hertz=44100,
language_code=u"fr"
)
streaming_config = types.StreamingRecognitionConfig(
config=config,
interim_results=True,
single_utterance=False)
@socketio.on(u'audiodata')
def handle_audio_data_msg(audio_data_msg):
print u'Received raw audio.'
print audio_data_msg
for response in client.streaming_recognize(streaming_config, [audio_data_msg]):
print response
emit(u'audiodatareceived', {u"AudioData": u"Acquired"})
I got TypeError: descriptor 'SerializeToString' requires a 'google.protobuf.pyext._message.CMessage' object but received a 'dict'
so I tried, on front side, to pass Int16Array.from(...).buffer
but I get TypeError: descriptor 'SerializeToString' requires a 'google.protobuf.pyext._message.CMessage' object but received a 'str'
So I'm not sure how I should pass the data... Any help appreciated!
[edit] I re-wrote my handler cause I think I was not using the right type, it is now like that:
@socketio.on(u'audiodata')
def handle_audio_data_msg(audio_data_msg):
print u'Received raw audio.'
request = types.StreamingRecognizeRequest(audio_content=audio_data_msg)
response = client.streaming_recognize(streaming_config, [request])
for item in response:
print u"error", item.get(u"error")
print u"results", item.get(u"results")
print u"resultIndex", item.get(u"resultIndex")
print u"endpointerType", item.get(u"endpointerType")
emit(u'audiodatareceived', {u"AudioData": u"Acquired"})
I don't get any error but I get no response (empty list) and so nothing printed...
[edit2]: I realized that the type of my input was never binary data but str. Could the problem come from here ? I try to instantiate my Flask Socket IO app as follow, but still the type is str.
socketio = SocketIO(app, binary=True)