HTML5 audio/video support

[ttcoder]

Following up from my offshoot comment in #12423:

When using Web+ on youtube (any channel I tried, including -- what else -- the TTSystems channel 8-), I have...

• video problems (reproducible only for me?)
• garbled audio

It remains to be seen if the regression is general or not

[ttcoder]

Replying to pulkomandy:

Sounds like some of your problems deserve separate tickets :)

• Regarding the "leak": I have noticed high memory use in app_server, I don't know if there is an actual leak or just more memory used (we do a lot of things on app_server side that used to be on the application side before)

With a few more hours usage, I can be more specific now: 1) when attempting HTML5 playback, the memory usage shoots up explosively (the deskbar replicant goes from blue to red in 3 or 4 seconds) and most of that memory usage 'explosion' is indeed not freed, even after quitting W+ the app_server still consumes 600 MB instead of 15 MB 2) I've never seen that before: whatever the HTML5 memory usage was before (hrev of a couple weeks ago) it was always freed (even in app_server) when quitting Web+ 3) That memory usage problem might be just a consequence of the second problem (garbled audio) so I'll wait for that first ; and HTML5 is not all that high a priority for me, so if it's the only one that leaks memory I'll wait some more to file a ticket on that, let's take these problems one at a time =)

• Regarding the video problems: it is probably because of ffmpeg update. It is working fine for video here, but I have no soundcard driver so I don't know about sound. On jua's laptop we get working video, and broken sound. We can revert to ffmpeg 0.11 (or some version that still works in between) until the problem is identified and fixed; binary searching ffmpeg versions to see when the problem appeared would be useful.

Actually we might have a shot at finding a 'shortcut' for fixing this. Explaining:

From my past experiences with mis-matching sample formats (yea I do that a lot *g*), the audio I get on e.g. the TTSystems channel sounds like 32 bit samples are being read as 16 bit samples. It could be something else (endianness mismatch, signedness mismatch..) but I from what I hear, Dane's voice is faintly audible if you listen carefully, though it's drowned into a lot of white noise. That clearly sound to me like 32 vs 16 bit mismatch. Maybe that hints at the problem ?

[jua]

Do you have a good reproducible testcase (e.g. a website URL) for the memory leak? Had no luck trying to reproduce it here so far with e.g. playing HTML5 video on youtube...

[pulkomandy]

• Renaming the ticket because HTML5 is so much more than just the audio and video elements
• As for the audio problem, mismatched formats is also my best guess so far. It may be that I missed one change in the API when adjusting the ffmpeg media plugin to work with ffmpeg 2.8. I don't think it is specific to WebPositive (which only sends the bytes to the media kit). Compiling others audio/video apps with gcc4 and ffmpeg 2.8 would allow to test that.

[ttcoder]

A test case that (apparently) exhibits a 32 vs 16 bit sampling mismatch:

• ​https://www.youtube.com/user/TTSystems/
• click ​CommandCenter fly-over

Black screen/view, garbled audio. Oddly, memory usage is fairly stable across the board in that case

[ttcoder]

A test case that is more general (on this computer at least), where memory usage grows super fast and I have a few seconds to react and close the tab, or kill W+, and even after that the memory is not reclaimed (until I reboot):

• ​http://www.youtube.com/embed/nLtmEjqzg7M (no man's sky trailer)

This is the behavior I see on all (other) videos I tried so far. Good idea about compiling other apps against gcc4 for testing purposes, could be a learning experience :-)

[vidrep]

Replying to jua:

Do you have a good reproducible testcase (e.g. a website URL) for the memory leak? Had no luck trying to reproduce it here so far with e.g. playing HTML5 video on youtube...

Try the video link in the description for ticket #11384. It won't play and pegs memory usage every time using x86_gcc2. 64 bit appears to work ok, using the older Web+ and ffmpeg.

[jua]

About the memory leak: Fixed in hrev49843, which plugs the memory leak and also brings further performance improvements. Please note that due to the new alpha mask caching, you can see app_srver grow by up to 8 MiB which are not free'd when Web+ is quit, that is because a regular time-based cache cleanup is not yet implemented.

Not closing this ticket, the remaining problem is the garbled audio... that's most probably due to the new ffmpeg version, maybe PulkoMandy can take a look at it.

[pulkomandy]

Fixed in hrev49844.

[pulkomandy]

Reopening as it's still not working. What I did in hrev49844 fixes "playfile" built for gcc4, that was an easy one to build for the other arch on my system. However, there are still issues with youtube videos.

[ttcoder]

Confirming; video is (unexpectedly?) fixed by that changeset, it seems to work 100% here now.

Sound, however, is still corrupt.. But in a 'different' way now: it now goes "klik klik klik". Oddly, the loudness of the "kliks" is constant, even if I change the master volume (in the deskbar). If I run Cortex to look at the Web+ connection to the mixer (not sure how useful that is..) I see it's a 32 bit float connection, 44100 Hz, little endian.

Edit: a newer hrev fixed sound a few days ago by invoking the swr_xxx() ffmpeg function, but as pulkomandy posted in another ticket there are still issues with video playing at 2x (pts/dts issue).

[pulkomandy]

Fixed in hrev50086.

[philcostin]

I just tested the difference on my Asus EEE901.

Sound is perfect in youtube now. Audio didn't even drop out once, even though the video was choppy due to hardware / accelerant constraints.