MediaConverter doesn't convert to OGG or MP3 and converts to WAV with video

[Kev]

I tried converting some MP4 videos to audio-only ogg vorbis or MPEG w/ AC-3 (same as MP3?). Every file I tried resulted in a small file (only a few kb) quickly created.

Raw audio in a WAV container worked; however, despite "no video" in the video encoding section, the WAV files, when opened in MediaPlayer, show the original video somehow. (I didn't even know WAV could do that.)

The MP4s in question play fine in MediaPlayer, which also plays other OGGs just fine.

hrev45413 gcc4hybrid. AC '97 Audio, Dolby 5.1 (6-channel)

[Kev]

As a side note, the WAVs are "normal" enough that BeOS' CDBurner app successfully burns audio CDs with them.

[ttcoder]

ping (hrev47850)

I too have a use-case for converting a wav file to something more compact, like Ogg/Vorbis -- I'd like to encode with the media-kit instead of always relying on /bin/lame :-)

In the latest hrev I get this when trying to generate an Ogg/vorbis from a wav file:

.../New folder> MediaConverter 
[libvorbis @ 0x187e2038] Specified sample_fmt is not supported.
[ogg @ 0x187e6738] No extradata present
[libvorbis @ 0x187e2440] Specified sample_fmt is not supported.
Error writing audio frames: Initialization failed

and MediaConverter then crashes in converter_thread.

Augustin said now is the right time to resurface that kind of media-kit/ffmpeg issue, so there =)

(PS @Kev I think AC3 is lossless compression, different beast from mpeg/layer3)

EDIT: Should mention that I didn't try a gcc4 image of Haiku as it apparently wouldn't make a difference over the gcc2h I'm using.

EDIT2: heard that mpeg/layer3 encoding is not available in Haiku for patents reasons, so Ogg/Vorbis is the alternative that comes to mind

[ttcoder]

Might as well schedule for alpha5.. Quite a few people will want to not only read (decode) media but also write/encode it =)

[ttcoder]

... ping..! (in case it's relevant to the 3-changeset in hrev48039)

[pulkomandy]

It's relevant, but does not really fix. What I did is add FLAC (which works) and remove all other formats which are crashing. FLAC is a lossless codec which means it doesn't compress sound as well as Vorbis or MP3. Unfortunately I'm not sure there is a way to fix this unless we revert to an older ffmpeg - probably from alpha3 days. And this would remove support for decoding some modern video formats (h264 and the like). The other solution is to write an encoder using libvorbis directly (or musepack, or APE or whatever format we decide to support).

Moreover, after fixing this I found there is another bug and some of the files I tested with gets one to two seconds truncated at the end. I have not debugged this part, yet, but MediaPlayer manage to decode the original files until the end so there must be a way to avoid the issue.

[ttcoder]

With the crash fixed, this probably becomes an enhancement ticket.. Update on my side:

Upgraded to 48061 and indeed PCM, Vorbis ..etc are gone, and flac works as you describe Adrien.

ffmpeg: is the bug specific to Haiku (i.e. it fails to encode because it's compiled with our old gcc2) ? Or maybe it's a general bug in ffmpeg, but in that case we could file a ticket with them, telling them it's very good at decoding but fails to encode on all platforms, including Haiku ? I guess I like the idea of getting a half dozen encoders 'for free' in the Media Kit :-) but I'm confused about what the situation is with ffmpeg/encoders in Haiku exactly.

libvorbis: in case ffmpeg is really a non-starter and we have to create encoders separate from it, a Vorbis encoder would be my favorite as closest to mp3 compression probably.. I did a bit of work on haiku Decoders some years back, I guess doing an Encoder might be in my reach.. And if my work is good enough to submit back to trunk that'd be a way for me to give something back to the haiku project.. If you confirm this is the way to go I'll try it out and create a separate ticket (to not spam Kev) to report progress if any.

[pulkomandy]

I didn't investigate the ffmpeg problem deeply, maybe it's possible to debug it. Here is what I know about it:

• Because of our constraint to use gcc2, we are using an old version of ffmpeg. Currently we use version 0.10.14. ffmpeg current releases are at 2.something. They still have maintenance releases for the 0.10 branch, however (and also for the 0.9 branch. Older ones have no updates since 2013).
• Newer version (0.11 and later) won't compile at all with gcc2.
• Even 0.10 is known to not go along well with gcc2. We got it to compile with some patching, but it warns on the console that it has been miscompiled because the compiler didn't align stack variables. Even with this known issue, decoding seems to work fine.
• But I'm not sure the stack alignment issue is actually related to the problem. We compile ffmpeg without SSE2 and that should make it safe. It's easy to check this by testing MediaConvert on a gcc4 system and seeing if it also crashes there (after reenabling the vorbis encoder).

I think the best way forward is:

• First, we need to get MediaConvert to work reliably with FLAC. The problems I've hit in my testing were on the decoding side of things. It seems MediaConvert manages to decode most of the file, but fails at the last "frame" of audio, which is shorter than others because it hits the end of the file.
• Once we have this working, we can see if it helps with some of the other formats.
• If it doesn't, having a vorbis encoder using libvorbis directly is worth a try.

Do you know if using ffmpeg to encode to ogg/vorbis from the command line works? If it does, the problem is not related to gcc2 or the way we build ffmpeg, but to using it in a wrong way in our encoder plugin. Ant it should be possible to fix that.

[ttcoder]

Very clear picture of the situation, allowed me to dig into this. Not going to tackle the input decoding right now (naïve idea: maybe it's just mis-handling of B_LAST_BUFFER status code?), but focused on the encoders still.

It seems the fault is indeed with the gcc2 build of ffmpeg 0.x itself, independantly of the "glue" code that ties it into the Media Kit.

Indeed, using it stand-alone from the command-line I got nothing but errors, failures, and one crash (in malloc()) for these attempts:

ffmpeg -i 01breakin_benjamin__so_cold.mp3  01breakin_benjamin__so_cold.aac
ffmpeg -i 01breakin_benjamin__so_cold.mp3  01breakin_benjamin__so_cold.ac3
ffmpeg -i 01breakin_benjamin__so_cold.mp3  01breakin_benjamin__so_cold.alac
ffmpeg -i 01breakin_benjamin__so_cold.mp3  01breakin_benjamin__so_cold.ogg
ffmpeg -i 01breakin_benjamin__so_cold.mp3  01breakin_benjamin__so_cold.vorbis
ffmpeg -i 01breakin_benjamin__so_cold.mp3  -acodec vorbis 01breakin_benjamin__so_cold.ogg
ffmpeg -i 01breakin_benjamin__so_cold.mp3  -acodec vorbis -aq 100 01breakin_benjamin__so_cold.ogg

This one, however, is pure gold:

ffmpeg -i 01breakin_benjamin__so_cold.mp3  01breakin_benjamin__so_cold.mp2

Tried that encoding (mpeg layer 2) on a few files.. It outputs the same size and audio quality as lame, but where lame takes e.g. 11 seconds to create an .mp3, ffmpeg creates an .mp2 in less than 2 seconds (!). Awesome.

Question...

Could that be the base of a media-kit encoder, technically ? I guess so. As to the "lawyer" side of the equation, I know Haiku can't pay the $$ fee to enable this out of the box, but could it be made an optional build feature, so that some of us can just "jam -q {add-ons,add-ons/x86}/media/plugins/ffmpeg" and use it with the mp2 capability ?

Otherwise I'll ask about the feasibility of using the recent ffmpeg 2.x in a gcc4 or gcc4h build of Haiku :-) I suppose the best place to start would be to look for a recipe for those on Haikuports.

[pulkomandy]

Well, ffmpeg actually is built with support for this, so making it available through the media kit is not going to make the legal side any better or worse. It's just a matter of adding the format to the list (which is as easy as what I did for flac in ​http://cgit.haiku-os.org/haiku/commit/?id=hrev48039).

An ffmpeg 2 based add-on for gcc4 versions is also possible. There is already a recipe for ffmpeg 2.x but I'm not sure if it compiles currently. But it should be possible to port the changes from our previous ffmpeg ports. Then, the media plugin on Haiku side would need to be modified to use the new APIs from ffmpeg 2.x (it would probably beecome an "ffmpeg2" plugin and live next to the existing one).

Finally, another thing you can test:

pkgman install ffmpeg_x86
ffmpeg-x86 -i 01breakin_benjamin__so_cold.mp3  -acodec vorbis 01breakin_benjamin__so_cold.ogg

If that works, we will be sure our problem is compiling ffmpeg with gcc2. If the -x86 (gcc4) version of ffmpeg also crashes, the problem is elsewhere.

If the problem is in ffmpeg, I'll see if I can get vorbis working using libvorbis, so we have at least one lossy encoder. And we can enable MP2 in ffmpeg too, if that one works.

[ttcoder]

That touches on something I'm curious about: seems ffmpeg_x86 is installed "out of the box" on gcc2h, but it does not contain the command line front-end, only the libraries:

~/Desktop> pkgman search ffmpeg
(..)
S       ffmpeg_x86        Audio and video recording, conversion, and streaming library                    
~/Desktop> pkgman install ffmpeg_x86
(..)
Nothing to do.
~/Desktop> query -a ffmpeg_x86
/boot/system/data/ffmpeg_x86
~/Desktop> query -a ffmpeg-x86
~/Desktop> 

Re. enabling mp2 support easily, sounds like a great plan B I could fall back to if needed!

[pulkomandy]

Right, I don't know why the ffmpeg package was setup like this. So at the moment the only way to test the gcc4 ffmpeg is to try a gcc4 version of Haiku. I will see if I can reverse the situation for the gcc2 installs so only the gcc4 version of the binaries is made available. At least we would have something that works.

[waddlesplash]

@pulkomandy Yeah, that sounds like a good idea. And yes it's totally possible to do.

Also, there is a FFmpeg 2.3 recipe in HaikuPorts that I made a few months ago which works on GCC4. I tried ffplay (SDL-based audio/video player that's included by default) and it worked for a few test videos that I had around anyway, so that's a good sign.

But yes, @pulkomandy is right in that we'd have to fork the Media Kit plugin into 2 parts: a GCC2 version for FFmpeg 0.10, and a GCC4 version for 2.3 and beyond as the API is very different (even 0.11 is source/binary incompatible with 0.10).

At any rate, it's certainly possible for @ttcoder to ship a 2.3 ffmpeg binary and library set with TuneTracker and use it to do media conversion internally. It's also possible for us to provide the {{ff*}} commands in the GCC4 package rather than the GCC2 one as well.

[pulkomandy]

I tested this on gcc4 (64bit). Converting with the commands you used above, AC3 works, OGG results in ogg/flac just like MediaConvert does. MP2 also works. The others (vorbis, alac, aac) apparently needs more parameters.

This works fine on both gcc2 and gcc4:

ffmpeg -i test.mp3 -codec:a libvorbis -qcale:a 5 test.ogg

So the ffmpeg library is not at fault, and there is something wrong with our media plugins and/or mediaconvert. This command line makes ffmpeg use libvorbis for audio encoding. There is also a "native" vorbis encoder, which doesn't crash but has bad quality:

ffmpeg -i test.mp3 -codec:a vorbis --strict experimental qcale:a 5 test.ogg

[ttcoder]

@waddlesplash With some luck you won't have to change anything in packaging, and we won't have to investigate gcc4 :-b read on.. (BTW before downloading a gcc4h nightly I looked ​here and ​here but was surprised to see the same ffmpeg package listings as I expected the x86 listing would be different from the x86_gcc2 one, tho the list is not easy to make out for a neophyte so leaving that aside for now :-)

@pulkomandy Thanks for going star wars on me and unleashing the power of that fully-operational (ffmpeg) battle station on my ass! I thought it didn't work... And there I am, with an evening and morning spent doing double-blind tests and the such and amazed at how good ffmpeg sounds *g*. Tried using -qscale and -q and I end up not being able to hear the different between the original .wav and the encoded ogg/vorbis even going as low as -q 0 (which encodes to ca. 64 kbps according to MediaPlayer > File Info menu). This laptop has cheapo speakers and I'm not an audiophile but still, I know I've never seen that kind of performance in years of using lame... That one requires 128 kbps or more..

So still todo..

• in my testing the last 6 seconds of original audio are indeed missing in the encoded vorbis audio, as you had originally mentionned.

And..

• have to look for what's breaking ffmpeg libs in 'libvorbis' mode when used by MediaConverter, or used in TT software (TimeTracker, TunePrepper ..etc);

MediaConverter outputs this to stderr/out: (EDIT: on my 47991 partition which still enables vorbis)

[libvorbis @ 0x187b84d0] libvorbis: buffer overflow.
Error writing audio frames: General system error

by opposed to using ffmpeg command-line, which works freakin' awesome:

ffmpeg -i test.wav -acodec libvorbis -q 0 test.ogg

(Not sure if that matters, but DiskProbe and /bin/cmp show that the 4KB (aborted) output from MediaConverter has some difference with the corresponding chunk in command-line ffmpeg's (correct) output: they both start off with '0ggS', 0x200 ..etc but there are some difference starting at offset 0xe. And starting at offset 0x65 there's even a "shift": the correct ogg file contains the string "vorbis" @ 0x67 whereas the incorrect one has it @ 0x68).

[pulkomandy]

I had a look at the code for ffmpeg 0.11 and noticed the fixed size buffer libvorbis complains about was removed. So I'm trying to write a recipe for ffmpeg 0.11.5 and I'll see if I can get the media kit working with that. I have some gcc2 compatibility issues but I should be able to sort it out. I hope there were enough changes in ffmpeg so we don't hit #8856 again.

[pulkomandy]

Oh and about the repositories layout: the "packages" folder is the same for all repositories, and contains all packages (new and old). This saves a considerable amount of disk space on the repo server since we keep old versions of the repository online.

The "repo" file defines what's available in each version of haiku, but is not human readable.

The packages ending with _x86_gcc2 are for gcc2. The packages ending with _x86 are for gcc4. So for a gcc2 install you will be using: ffmpeg-0.10.14-2-x86_gcc2 and ffmpeg_x86-0.10.14-2-x86_gcc2

And for a gcc4 install: ffmpeg-0.10.14-2-x86.hpkg and ffmpeg_x86_gcc2-0.10.14-2-x86.hpkg

The "x86" architecture means gcc4 compiler. This is a bit confusing at first but it allows to keep a simpler naming scheme in the long term.

[ttcoder]

Another odd thing -- ffmpeg-0.11.5-1-x86_gcc2.hpkg is already listed as done ;-) Didn't open it to see what build is inside though

[pulkomandy]

Yes, as I said I'm working on it. That package is broken, and not listed in the public repositories yet. I will upload a working one when I'm done :)

[pulkomandy]

So, I got ffmpeg 0.11.5 to compile and run after a number of hacks. There are some good news: it seems it doesn't break video and audio decoding. So we could actually update to it (#8856 somehow isn't a problem anymore). But there are also some bad news: the vorbis encoding still fails.

However, the error is more obvious this time. What happens is the libvorbis encoder in ffmpeg only can accept floating-point format as an input. I guess most ffmpeg encoders are designed the same. But our media plug-in doesn't propagate this information, and MediaConvert tries to feed integers to the encoder instead. ffmpeg added an explicit check for this in version 0.11, so they error cleanly, but the error isn't handled properly by our plugin and in the end MediaConvert crashes.

[vidrep]

When using the "pkgman full-sync" command, an error message is generated which refers to missing libraries (see attached screenshot).

[vidrep]

A question, if I may. Is there any reason why we cannot go directly to the latest ffmpeg revision in the gcc4 (32 and 64 bit) builds?

[pulkomandy]

Hi, I noticed this issue and am currently preparing release 1.4.6 of haikuwebkit to fix it. I don't know if the build will be done before I leave for BeGeistert, however.

There is a recipe for the latest version of ffmpeg which works with gcc4, but gcc2 will need to stay at 0.10 for now. I did more testing with 0.11 and there are still problems with h263 and h264 decoding. I tried disabling various things (MMX, SSE, use of asm code, etc) but it still crashes. Later versions are even harder to get to compile with gcc2.

There is no problem with updating the gcc4 version of ffmpeg, but the media add-on must be made to work with either version (or maybe we need an ffmpeg_legacy one for gcc2 and an ffmpeg one for gcc4). This allows the gcc4 world to move on but doesn't solve our issue with legacy apps.

[vidrep]

I updated to hrev48082 using "pkgman update". Any attempt to open a video file or media player causes a crash. For what it's worth, I have attached the debug report and a screenshot. My personal opinion is that gcc2 and hybrid builds should only be developed further for legacy BeOS apps, and that we look to the future now with gcc4 versions.

[pulkomandy]

I already reverted the change. Indeed ffmpeg 0.11 still does not work with gcc2. I'm not trying to get it up to date, I'm trying to get it working to fix this bug here. This is part of the "legacy BeOS apps" support, so we have to do something about it.

Hybrid builds in general are useful for many other reasons, for example they are a first step towards a 64/32bit hybrid, or a R2/R1 hybrid (allowing to run R1 apps on Haiku R2), and a few other cases. The experiments done with the gcc2/4 hybrids are a good training to learn about the problems to face when doing such things, so the future kind of hybrids will work more smoothly. It would be sad to drop general support for hybrid, which took quite some time to develop and get right, and then after R1 is out, notice we need it again for R2 and having to restore it from old commits.

[ttcoder]

@pulkomandy -- Working on adapting/testing my apps in hrev48150 and found one which is allergic to FLAC -- it feeds HDA line-input (instead of a source file .wav) to the encoder, so the samples are 32 bits instead of B_AUDIO_SHORT and 96KHz instead of 44 KHz, and ffmpeg/FLAC replies with "unhandled sample_fmt" or something. If I hack both in the format I pass to BMediaFile::CreateTrack() / CommitHeader() like thus

fmt.u.raw_audio.format = media_raw_audio_format::B_AUDIO_SHORT;
fmt.u.raw_audio.frame_rate = 44000.0;

then I get a B_OK and the file is created, but of course the audible content is mostly junk.

Once I solve this I can wait a while for the rest of the ticket enhancement (ogg/vorbis or other lossy codec).

Questions..

• how much work involved in making flac accept 32 bits samples instead of 16.. and high-KHz sample-rates (though that latter can probably be fixed another way, by changing HDA input in Media Prefs).
• is it possible to re-enable Wav/Pcm-Raw-Audio, in addition to Wav/Flac? It worked well with high sample-sizes and rates in my testing this morning in 47991.

[pulkomandy]

Sample rates should not be a problem, however making it work with 32-bit audio doesn't seem possible on ffmpeg side. What we need to do here is review our ffmpeg plugin so it properly exports the constraints for each encoder. Right now it pretends to accept any kind of input, but actually doesn't.

Then, it should be possible to configure the dcoders to output matching formats, or insert a format conversion step in the process. But without the ffmpeg plugin advertising its actual capabilities and limitations, this is not really possible.

[ttcoder]

Adrien thanks for re-enabling PCM-raw-audio in the hrev48155 changeset, I managed to test that despite stability problems tracked in another ticket, and it works very well for recoding HDA line-input. As to the FLAC codec, I might write a homegrown resampler in a couple weeks to make HDA line-input and the ffmpeg add-on "talk" to each other.. Hardcoding something like "if selected codec == flac then call the resampler/converter before feeding the data to BMediaFile::WriteFrames()" as a temporary hack. But we're in a much better situation now, being able to record line-input in current hrevs. Probably won't hear from me on this ticket for a while :-)

[ttcoder]

@pulkomandy: a data point that should be interesting for you.. I don't see the 6-seconds-before-the-end truncation in my code. When I record the HDA Line-Input for 10 seconds, the result file is 10 seconds long.

That's true of Wav/Pcm, and Wav/FLAC which works since yesterday with some code I've added to convert samples from 32-bit to 16-bit.

So the problem might reside in MediaConverter itself.. (or in the Media Kit, on the decoding side, but after all those years somebody would have noticed ;-).. I guess you could do a bare-bones, mono-threaded 20 lines conversion program, confirm it works correctly (no truncation), then compare its behavior with MediaConverter.

So here's hoping that will save you a bit of hunting when back to this ticket (no sense of urgency here, dsuden is happy with FLAC!)

(completely off-topic: just noticed the recent posts on www.freelists.org and here's a +1 for a separate R1 "legacy" branch, you guys fight the good fight mainly with the new/elegant code, life's too short for spending it 100% on fighting legacy gcc's and legacy ffmpegs *g*) (not to mention it has an impact on third-party devs who spend an inordinate amount of time writing tickets instead of writing elegant/new code *g*)

EDIT: writing this quiet update in late december to track here that the MediaConverter truncation bug was fixed in hrev48552

EDIT2: and AC3 + AAC were added in hrev48554, gotta test that!

EDIT3: a low priority question.. I'm wondering if this ​change does not introduce a potential problem: what if the newly added DecodedFormat() ends up modifying raf->buffer_size ? In some scenarios, the audioBuffer = new uint8[..]memory allocation higher up might be too small to receive the data that will later be loaded by ​ReadFrames(), so the memory/buffer allocation should be moved down below the second DecodedFormat(), or do I understand that wrong ?

[pulkomandy]

Yes, the cut of the end of file is on the decoding side and most likely a problem in MediaConverter itself. But it should still be fixed :)

I have the "transcode" program from Be sample code as well, but it probably shares the same issues as Mediaconverter was derived from it by adding a GUI.

I think it is possible to get the Media Kit to properly negociate formats so no conversion is needed on the application side (we just need the decoder to use a format the encoder can accept).

[pulkomandy]

This should be fixed for good in hrev48552 and hrev48554.

hrev48552 fixed the issue of truncating the end of the sound. This was a bug in MediaConverter only, Media Kit was not at fault. Now files will be converted completely and removed from the list when processing is finished.

The bulk of the fixes is in ​http://cgit.haiku-os.org/haiku/commit/?id=5e0e16894d67e220b4684809b154b85db4229381 . The problem was that libavcodec only accepts some input formats when encoding. The set of formats depends on the encoder used. We were ignoring this, and we forced ffmpeg to use a format it was unable to handle. Now the ffmpeg media plug-in will notice this and adjust the format for media kit negotiation. This got several more formats working, including AAC and AC3, which are lossy formats similar to Vorbis and MP3.

To make this work, there is an extra change in MediaConverter: ​http://cgit.haiku-os.org/haiku/commit/?id=974595bc4673e6e443ae7b6c775cf9461ff2f23d What we do here is tell the decoder about the constraints of the encoder on the audio format. The decoder is able to perform the required format conversion, so it outputs data in the correct format for the encoder. Other applications using the media kit may need to be adjusted in a similar way.

Finally, I tweaked the format families a bit so only some working combinations are possible. Matroska (MKV) seems to be the most reliable container and can hold all supported formats: flac, aac, ac3, and raw audio. Other containers (avi, dv, mpeg, ogg, wav) can only use some of the formats.

I'm leaving this ticket open as there is still no Vorbis or MP3 output, however it works well enough for Beta1 so I'm moving out of the milestone.

[dsuden]

Sounds really promising, Pulkomandy! I did quite a bit of testing last week with various formats and containers, and Matroska did indeed seem consistently good across audio formats. If there was a down side, it was that files I did with Matroska on Haiku didn't seem to be playable or editable in Mac or Windows. I'll get another hrev sometime later this week though and do some more testing.