26

My research has led me to learn that Apple's media element handler is a singleton, meaning I can't have a video playing while an audio is playing in the background. I'm tasked to build a slideshow presentation framework and the client wants a background audio track, timed audio voice-overs which match bullet points, and variable media which can either be an image or video - or a timed cycle of multiple media elements.

Of course, none of the media works on iOS. Each media element cancels out the previous.

My initial thought is to embed the voice-over audio into the video when there's a video present, but there's an existing Flash version of this setup which depends on existing assets so I pretty much have to use what's delivered.

Is there ANY work-around for this? I'm testing on iOS 4.3.5. The smartest devs in the world are on this site - we've got to be able to come up with something.

EDIT: Updated my iPad to iOS 5.0.1 and the issue remains.

AlienWebguy
  • 76,997
  • 17
  • 122
  • 145
  • 5
    +1 i would also like to know how to do this. – Maverick Feb 22 '12 at 01:15
  • Good luck finding a work-around! - Only option i can suggest is a full-blown iOS app instead of a webpage. If it just won't play more than one element and you can't put all the video/sound into a single element with a simple play/pause of the video you might be out of luck. iOS won't run a Java Applet (AFAIK) nor flash, nor Silver light either. You could *try* multiple IFRAMES but i doubt that will work if their internal multimedia player object is indeed a singleton, might be worth a shot however. – White Dragon Feb 22 '12 at 01:35
  • @PHPBree - sorry, not possible in a web page *or* native app.. see answer below.. – Lloyd Feb 22 '12 at 22:14
  • @PHPBree As stated in Apple's [documentation](https://developer.apple.com/library/safari/#documentation/AudioVideo/Conceptual/Using_HTML5_Audio_Video/Device-SpecificConsiderations/Device-SpecificConsiderations.html), "Currently, all devices running iOS are limited to playback of a single audio or video stream at any time." The only way to shim this would be to compile it all before. You can hold two copies of the files. A copy which is the compiled audio/video, and a file with a separate stream for both. Other than that you're out of luck. – LoveAndCoding Feb 25 '12 at 19:50
  • Just for giggles can you try the audio in a frame to see if it would try and create another instance? Probably won't, but worth a try. – Bot Feb 28 '12 at 17:07

5 Answers5

4


How about do it with CSS to do the trick.
Maybe you know about a company called vdopia that distribute video ad on mobile. http://mobile.vdopia.com/index.php?page=mobilewebsolutions
They claim to developed what so called vdo video format, that actually just to do a css sprite running on that :D
I mean you could have your "video" as a framed image, then attach html5 audio tag on there.

I would like to know your response

Alfian Busyro
  • 2,295
  • 2
  • 17
  • 32
  • This looks like it may work. Use the above for video then use the html5 tag for audio. It really all depends on how you want your video to show. This looks like it only shows in a popover ad format but it may be the best solution. – Bot Feb 29 '12 at 16:24
  • So it's like a glorified animated gif? – AlienWebguy Mar 01 '12 at 04:27
  • I created a sample for you, take a look : http://dl.dropbox.com/u/16001200/html5/vdo.html – Alfian Busyro Mar 01 '12 at 07:12
  • I like this one. Sounds like animated GIF is played by the browser itself, leaving the single quicktime instance free for the audio. Given the small screen size and your description you're probably not requiring HD so GIFs poor compression and quality relative to h264/ogg/webm maybe isn't such a big deal. – SpliFF Mar 01 '12 at 14:58
  • Ya it couldn't be an animated gif because the presentation has a pause button. The `setInterval` or recursive `setTimeout` with a background sprite is nothing new, though I hadn't really considered it because of the lack of streaming, the complexity integrating it with the pause button, etc. But so far this is the first legitimate shim I've seen. Kudos. – AlienWebguy Mar 01 '12 at 16:52
  • you could replace the sprite image with ordinary image or a html file within an iframe, and not doing a recursive rendering like this. But you could render it regularly, and with a Controller function you control the render timing (pausing, playing, stoping, etc). – Alfian Busyro Mar 02 '12 at 09:32
  • I got a proof working and learned a few critical things. Photoshop will not save huge images to anything other than PSB, RAW, or TIFF. Had to export my JPG sequence to TIFF and then with IrfanView, save as JPG. JPG max dimensions are 65535×65535 so a really wide stitched jpg sequence won't work if it exceeds that. My original output was 144000x240. Saving that image to PNG will be viewable in Windows, but NOT in a browser. Saving that image to GIF will crop the image to 12928px wide. Final working solution was to export a 24000x1440 (75 frames wide X 6 rows) and it works, though a bit choppy. – AlienWebguy Mar 02 '12 at 18:49
  • Scratch that - when tested on the iPad, it scales the background image way down : http://www.teknocat.org/blog/web-development/show/6/mobile-safari-background-image-scaling-quirk – AlienWebguy Mar 02 '12 at 19:50
  • so we learned for small videos which can fit all its frames in a 1600x1200 (2 megapixel) space, this solution will work. This makes sense why it works for vdopia. I am awarding the bounty for this idea because I think its the only feasible shim provided so far. @AlienWebguy I dont think you should accept this answer though. I would like to put another bounty on this question in a few weeks when I can get some rep back. – Maverick Mar 03 '12 at 19:07
1

Are you working on a Web App or on a Native Application?

If you are working on a Web App you're in a world of hurt. This is because you simply do not have much control over things that Mobile Safari doesn't provide right away. If this is the case I would come forth and be honest with the stakeholders.

If you are working on a Native Application you can resort to a mechanism that involves some back and forth communication between UIWebView and ObjC. It's actually doable.

The idea is the following:

Insert special <object> elements in your HTML5 documents, that you handcraft yourself according to your needs, taking special care to maintain the attr-* naming convention for non-standard attributes. Here you could insert IDs, paths and other control variables in the multimedia artifacts that you want to play.

Then you could actually build some javascript (on top of jQuery,p.e.) that communicates with ObjC through the delegation mechanism on the UIWebView or through HTTP. I'll go over this choice down below.

Say that on $(document).ready() you go through all the objects that have a special class. A class that you carefully choose to identify all the special <object>. You build a list of such objects and pass them on to the ObjC part of your application. You could easily serialize such list using JSON. Then in ObjC you can do what you want with them. Play them through AVPlayer or some other framework whenever you want them played (again you would resort to a JS - ObjC bridge to actually signal the native part to play a particular element).

You can "communicate" with ObjC through the delegation pattern in UIWebView or through HTTP. You would then have a JS - ObjC bridge in place.

The HTTP approach makes sense in some cases but it involves a lot of extra code and is resource hungry.

If you are building an ObjC application and want further details on how to actually build an ObjC - JS bridge that fits these needs get back to us :)

I'm halting this post as of now because it would be nice to know if it is in fact a Native App.

Cheers.

Bot
  • 11,868
  • 11
  • 75
  • 131
janela
  • 31
  • 1
  • 4
  • It is not a native app, but I appreciate your insight if that is the route we choose to take for a workaround. – AlienWebguy Feb 28 '12 at 02:03
  • No problem. The guys that made the HTML5 version of Angry Birds probably had the same problem (Desktop Browsers included). They had to resort to flash for audio playback probably because of that implementation idiosyncrasy that you pointed out. @Computer - Thanks for the revision. – janela Feb 29 '12 at 15:55
0

This is currently not possible. As you notice when a video plays it takes up the full screen with quicktime and moves the browser to the background. The only solution at this time is to merge the audio and video together into an mp4 format and play that single item.

If I understand you correctly, you are not able to merge the audio and video together because it relies on flash? Since iOS can't play flash you should merge the audio and video together and use flash as a backup. There are numerous html5 players which use javascript to try and play the html5 video first then fallback to flash for backup.

Bot
  • 11,868
  • 11
  • 75
  • 131
  • Full screen video is only iPhone. iPad embeds the video just fine. I don't want to merge the audio and video because Flash relies on the assets. Both the Flash and the HTML5 versions will use an XML doc which point to asset locations. – AlienWebguy Feb 29 '12 at 16:13
  • @AlienWebguy When you say assets, do you mean that the music can change? – Bot Feb 29 '12 at 16:18
  • It's not music, it's a voice-over track. Each slide has audio of a lady explaining stuff. Bullets fade in when she says certain things, and either an image or a video will be playing next to the bullets, usually a demonstration animation of some kind. There is also a background music track, but I've already convinced the clients to let that go for the iPad version. – AlienWebguy Feb 29 '12 at 16:27
0

You mention there is an existing Flash setup of the video - is it a an swf file, could you import it into a video/audio editing software and add an audio track on top?

Something like this: http://www.youtube.com/watch?v=J2vvH7oi8m8&feature=youtube_gdata_player

Also, if it is a Flash file, will you be converting it to an avi or like for iOS? If you'd have to do it anyway, there is your chance for adding an audio track.

electronicKT
  • 68
  • 6
  • No, the presentation is interactive with chapter and slide navigation. Each slide can have 0 or 1 audio AND (0 or more images, and 0 or more video). At the very least a slide will have a single video taking up the whole container of someone demonstrating something. At the most (currently), the slide will have a voice-over track with 2-3 images and a video which fade in at specific time intervals and sync up with the voice-over audio track. – AlienWebguy Feb 29 '12 at 16:18
0

Could you use a webservice to merge the streams in real time with FFMpeg and then stream one output to quicktime?

To elaborate maybe a library like http://directshownet.sourceforge.net/about.html could also work. It looks like they have method

DESCombine – A class library that uses DirectShow Editing Services to combine video and audio files (or pieces of files) into a single output file. A help file (DESCombine.chm) is provided for using the class.

This could then be used to return the resulting data as the response to the call and loaded via the HTML5 player.

j_mcnally
  • 6,928
  • 2
  • 31
  • 46