2

I'm writing an app that requires an audio stream to be recording while a backing track is played. I have this working, but there is an inconsistent gap in between playback and record starting.

I don't know if I can do anything to make the sync perfect every time, so I've been trying to track what time each stream starts so I can calculate the delay and trim it server-side. This also has proved to be a challenge as no events seem to be sent when a connection starts (as far as I know). I've tried using various properties like the streams' buffer sizes, etc.

I'm thinking now that as my recorded audio is only mono, I may be able to put some kind of 'control signal' on the second stereo track which I could use to determine exactly when a sound starts recording (or stick the whole backing track in that channel so I can sync them that way). This leaves me with the new problem of properly injecting this sound into the NetStream.

If anyone has any idea whether or not any of these ideas will work, how to execute them, or some alternatives, that would be extremely helpful! Been working on this issue for awhile

Lowgain
  • 3,254
  • 5
  • 27
  • 30

4 Answers4

0

The only thing that comes to mind is to try and use metadata, flash media streams support metadata and the onMetaData callback. I assume you're using flash media server for the audio coming in and to record the audio going out. If you use the send method while your streaming the audio back to the server, you can put the listening audio track's playhead timestamp in it, so when you get the 2 streams back to the server you can mux them together properly. You can also try encoding the audio that is streamed to the client with metadata and try and use onMetaData to sync them up. I'm not sure how to do this, but a second approach is to try and combine the 2 streams together as the audio goes back so that you don't need to mux them later, or attach it to a blank video stream with 2 audio tracks...

superfro
  • 3,327
  • 1
  • 18
  • 14
0

If you're to inject something into the NetStream... As complex as SOUND... I guess here it would be better to go with Socket instead. You'll be directly reading bytes. It's possible there's a compression on the NetStream, so the data sent is not raw sound data - some class for decompressing the codec there would be needed. When you finally get the raw sound data, add the input in there, using Socket.readUnsignedByte() or Socket.readFloat(), and write back the modified data using Socket.writeByte(), or Socket.writeFloat().

This is the alternative with injecting the back into the audio.

For syncing, it is actually quite simple. Even though the data might not be sent instantly, one thing still stays the same - time. So, when user's audio is finished, just mix it without anything else to the back track - the time should stay the same.

IF the user has slow internet DOWNLOAD, so that his backtrack has unwanted breaks - check in the SWF if the data is buffered enough to add the next sound buffer (usually 4096 bytes if I remember correctly). If yes, continue streaming user's audio.

If not, do NOT stream, and start as soon as the data catches back on.

Aurel Bílý
  • 7,068
  • 1
  • 21
  • 34
  • Would I be able to use a Socket to connect with the media server the same way, or at that point am I going to need to handle all the protocols, etc myself? – Lowgain Nov 10 '10 at 01:53
  • The problem with the sync is that I can't mixdown both tracks into one file because of the nature of the application. When I start recording, there is a varied amount of time between when the backing track starts and when the recording starts. I looped an audio cable from the headphone jack to the microphone jack and recording the backing track that way multiple times, and compared each file. Each one was offset a different amount of time from the original backing track if that makes sense – Lowgain Nov 10 '10 at 01:56
  • It is really annoying, as Cay said - NetStream is quite inaccurate. I've checked OSMF source, but that all just EXTENDS NetStream... I guess you need a custom code of some sort... Unfortunately searching for an RTMP AS3.0 framework shows just NO results - I guess everybody is using the AS3.0 original libraries. As for mixing the file - you don't really need to mix it... Just stream the audio from user. Mic delay is a different problem though... But I think that's not easily solvable as it heavily depends on user's hardware. – Aurel Bílý Nov 10 '10 at 14:14
0

In my experience NetStream is one of the most inaccurate and dirty features of Flash (NetStream:play2 ?!!), which btw is quite ironic seeing how Flash's primary use is probably video playback. Trying to sync it with anything else in a reliable way is very hard... events and statuses are not very straight forward, and there are multiple issues that can spoil your syncing.

Luckily however, netStream.time will tell you quite accurately the current playhead position, so you can eventually use that to determine starting time, delays, dropped frames, etc... Notice that determining the actual starting time is a bit tricky though. When you start loading a netStream, the time value is zero, but when it shows the first frame and is waiting for the buffer to fill (not playing yet) the time value is something like 0.027 (depends on the video), so you need to very carefully monitor this value to accurately determine events.

An alternative to using NetStream is embedding the video in a SWF file, which should make synchronization much easier (specially if you use frequent keyframes on encoding). But you will lose quality/filesize ratio (If I remember correctly you can only use FLV, not h264).

Cay
  • 3,804
  • 2
  • 20
  • 27
  • ahah yes I've noticed the inaccuracies here, I'll see if I can do anything with the time values, thanks. given the nature of the app I dunno if I can easily wrap everything in SWFs (the backing track is user generated content) but perhaps I can find a way to automate that – Lowgain Nov 12 '10 at 19:31
0

no events seem to be sent when a connection starts

sure there does.. NetStatusEvent.NET_STATUS fires for a multitude of reasons for NetConnections and Netstreams, you just have to add a listener and process the contents of NET_STATUS.info

the as3 reference docs here and you're looking for NET_STATUS.info

chrixian
  • 66
  • 2