YouTube and FastStart/flattening for MP4/MOV

BibbityBoo wrote on 6/2/2010, 11:16 PM
YouTube has finally tracked down at least one factor that seemed to trigger a lot of audio sync issues, mainly with Mac-based encoders from the Final Cut, QT Pro and other tools. Verbiage is here

http://www.google.com/support/youtube/bin/answer.py?hl=en&answer=55744

and here

http://www.google.com/support/youtube/bin/answer.py?answer=182687

Video that do not conform to the "flattened" format they claim is optimal now get the following message on upload:

"The MOV/MP4 file you uploaded contains the index at the end. We prefer you flatten the file before uploading so that the index is in the front of the file. See this article for more information." "This article" links to:

http://www.google.com/support/youtube/bin/answer.py?hl=en&answer=165543
and maybe http://support.apple.com/kb/HT2438

I can understand this message being triggered by MOV containers where there's an option to render with FastStart or other options. Does this exist, though, in MP4s where I'm getting the same warning, regardless of the data included as part of the render (so far as I can tell, at least?) Is there some utility to "flatten" MP4s that use either the MainConcept or Sony templates? And is it really necessary? I'm seeing the same mediocre quality as usual from uploads to YouTube as ever, so I'm tending to want to save my time and leave well enough alone. But if there IS some marked improvement to be made by altering the finer points of the encoding, it WOULD perhaps be worthwhile to pursue this.

All comments welcome, but I'd especially appreciate any feedback the devs might have on this, since I've wasted a huge amount of time beating my head against the Google wall of silence when it comes to hunting down audio problems, of which there are so many once YouTube transcoders get their claws into one's uploads, no matter how pristine they are in local playback.

Hope the hasn't triggered the TL;DR factor for everyone.

Videos that have triggered the Warning since YouTube's recent addition of more verbose format warnings so far have included:

1)
2)
3)

These are listed in reverse order of upload. The first one was rendered using a MainConcept profile, the other two in Sony AVC/AAC -- #2 @ 720p and #3 @1080p in presets that I've used consistently with decent results (for YouTube) and better results on Vimeo (big surprise) for about a year now... about 4 or 5 months in the case of the 1080p version (#3).

If it's of any interest, I can readily provide MediaInfo profiles for all of these, though I expect I may add them to the video descriptions very soon and that will save from another excessively long post here?

Comments

DGates wrote on 6/2/2010, 11:24 PM
You know, I got that same exact message on my recent YouTube upload. But it was still added to the playlist anyway and the audio seems fine. Mine also was an MP4.
musicvid10 wrote on 6/2/2010, 11:28 PM
What they are trying to tell us with that cryptic message, is to use a codec / application that enables streaming. Of course, you may get another message, which I did with the streaming option enabled in HB, but YouTube is still learning.
BibbityBoo wrote on 6/2/2010, 11:31 PM
Yes, I'm one of the TCs on the YT Help Forum, so I've heard a little bit more from staff on this than some others. The message shouldn't be leading to any (or many) encoding problems if your profile was working before. In fact I think it might be a glitch in interpretation... I should know more maybe by the end of the week, if lucky.

But it IS implicated in a lot of the audio sync issues that have been reported regularly, especially from the Mac-married crew. Aside from some stuttering/glitching in audio at between 26 and 30 seconds in, which seem to be universal YT problems, I'm not seeing anything hugely defective with these test vids... at least nothing that hasn't been defective for years.

Still, I should probably at least try flattening/FastStarting if I can find a trustworthy utility that's not coded strictly for Mac users.
BibbityBoo wrote on 6/2/2010, 11:37 PM
Yes, there were several other messages, apparently, added in the same maintenance push that enabled the one I quoted. I know what they're *trying* to tell us. I just am not convinced with their "one bit of code fits all" approach, that the messages are going to be free of false positives this early in their development, and considering the "Help" pages they've added, and how badly they're written, I'm beginning to suspect that the only thing Google manages to get with their rumored insistence on a 4.0GPA from all applicants is a population skilled enough to hack the school records database without getting nailed.
musicvid10 wrote on 6/2/2010, 11:40 PM
Yeah, try Handbrake, check the streaming option, and ignore the second message, which is a bit of nonsense IMO.

Did I miss something in your first post?
BibbityBoo wrote on 6/2/2010, 11:52 PM
I should have thought of Handbrake.

Though I'm still curious to know whether there is ANYTHING actually "wrong" for YouTube with the file as rendered straight from Vegas, aside from having uploaded it to YT at all? ;)

I figured I might get a better chance at an informed response here, rather than from YouTube (including my staff contacts).

When you say ignore the 2nd message, just what message were you referring to?
musicvid10 wrote on 6/3/2010, 12:05 AM
Something about extra information in the file. As I said, it's just a niggle.
What YouTube is telling us with the first message, if I understand correctly, is that they would rather not do the extra processing for streaming, assuming (probably incorrectly) that the average joe would know how to do it himself.

Logical from the standpoint of saving server time, but not entirely realistic. YouTube would better spend its time learning how to handle anything that is thrown at them, rather than provide conditions for arguably clueless users.
BibbityBoo wrote on 6/3/2010, 12:27 AM
Well, I tried Handbrake and that only seemed to decrease the fidelity of the video, not surprising from re-rendering a video in a lossy codec, though. And it did generate another, different (new) warning about the video containing lists. I may have missed some key setting, since I rarely use Handbrake... but it does seem like another instance of needing to double my rendering time and getting worse results than Vimeo. Then again, that's not really a change.

Here's the quicky (and defective in many respects) Vegas+Handbrake version. I realize that I would probably have done better to have rendered this in full HDV .M2TS form -- in fact I may do that and upload the raw file to YT to see what happens, time permitting. Maybe that will eliminate all spurious "warnings"?



I tend to agree with you about YT's likely reasons for playing CYA with these messages. As long as there's no appreciable difference in their transcoding between the past (when the same defects existed with no warnings) and the present I think I've taken this about as far as I need to for my personal curiousity. Reminding myself that Eugenia Loli-Quero long ago commented that she only uses YouTube for uploading test renders and other junk video. :) Thanks for the input.
musicvid10 wrote on 6/3/2010, 12:31 AM
I have tried many different options, and I find Handbrake the highest quality solution for creating Youtube videos. It uses the x264 codec, and with a few tweaks, outperforms dozens of other alternatives I have tested. As I stated, the message about lists is a crock, IMO.

I will be happy to provide you with my starting settings for HD, if you are interested in pursuing this further.
BibbityBoo wrote on 6/3/2010, 12:51 AM
Thanks for the offer. Don't go out of your way though... I really am on the verge of just ignoring YouTube altogether at this point, unless I start rendering for pay for someone who seriously demands it... and in that case I expect my current regimen will be more than adequate for such clients. There are a few other renders from Vegas that I may try first, since my videos involve a lot of Vegas editing and tweaking, and I may start looking into Vp8 as a codec, if other, better sites wind up following YouTube's move to use it in connection with HTML5. I don't want to waste too much more time banging my head against the Google wall of silence, in any case.

I did notice, though, that "something" (maybe) in the Handbrake render seemed to have prevented that 30 seconds-in "hiccup" so this might bear more exploring. I've seen so many people reporting problems with x264 renders than I'm hesitant to start trying to deduce the perfect way to render it for YouTube... too many times in the past I'd used codecs that were not *exactly* explicitly on the charts with them and wound up having something work great for a few weeks only to get trashed by some new brainfart as YT engs messed around with the transcoders. It's just way more pain than I want to enjoy, and largely pointless, at least for now, when I can get reliable results from Vimeo without doing handstands and somersaults.

But thank you for your input!
A. Grandt wrote on 6/3/2010, 3:43 AM
BibbityBoo, you mention you have some technical contact in YT, can I get you to try and figure out why YouTube's 1080p mode, isn't 1080p but 1080i with one field missing (=1920x540p)?

1920x1080p test:

The first 5 seconds, you can tell that it isn't 1080p by looking at the text, It's somewhat jagged on the curves. From 5 to 20seconds it's all white with the top and right edges rows/columns alternating between black and white. The right edge is all black, it should not have been, see the test below to see what I were expecting.
The original mp4 is here: [url=http://www.grandt.com/Vegas/YT Aspect test 1920x1080.mp4]

But the 1920x960 (2:1 aspect) works as I expected it to:


I've tried to post this on the TY forums, but no one have reacted.
BibbityBoo wrote on 6/3/2010, 4:39 AM
@A._Grandt I'll see what I can do. I seem to recall a few similar reports of strangeness. I'll try to get one of the more technically adept TCs (epontius) to give this a look too. Please post links to one of your original posts... there's so much spam and garbage on those forums that valid tech question often get lost in all the raw sewage.

Add.: I found your original post in the YT Forums and have noted it for escalation. Hopefully the others there will have some more feedback (and recall some of the other stuff about 1080p? that was brought up just before the watch pages were revised and half the site features seemed to go haywire. I should probably be spending much more time here and on CreativeCow, given that people actually DO talk about technical issues here, not view counts and whining about trolls. ;)

MarkWWWW wrote on 6/3/2010, 6:18 AM
> Is there some utility to "flatten" MP4s that use either the MainConcept or Sony templates?

Try using one or other of these two free moov atom moving utilities:

MP4FastStart
MP4Patch

Mark
BibbityBoo wrote on 6/3/2010, 6:53 AM
Thanks a ton! Tentatively speaking, at least, the first utility seems to work wonderfully well. At least it speeds up the upload/conversion process a bit, though I am still getting that "second message" about lists... could be something in how I rendered, such as project markers and such embedded in the upload file.

Added benefit over twice processing using Handbrake, the conversion is very quick and something I can live with, especially if it cures the stuttering I'd come to assume was a feature of streaming video, not a defect.

Thank you very much, MarkWWWW!!!

First test video with further comments and linkage:



Well, that was shortlived hope... audio now totally garbled in a way it wasn't with the other version. Will have to compare the other utility's impact and do some more troubleshooting.... Ooops... serves me right for commenting before processing is done completely.
A. Grandt wrote on 6/3/2010, 6:55 AM
"given that people actually DO talk about technical issues here, not view counts and whining about trolls. ;)"

There seem to be a lot of that in the YT help forums. Though some of the issues really need to be addressed, there are also some technical problems that sometimes get lost in the proverbial posting forest.

As for the thread topic, I've only seen that warning once, I'll have to try and make a few more uploads.
So far I tend to use the Sony AVC Codec, with Project markers included, though I usually don't have any. But from what I can tell, neither Sony, or MainConcept's AVC profiles have an option to "Flatten" or encode for streaming.
BibbityBoo wrote on 6/3/2010, 7:08 AM
No. They don't (provide options for specifying a FastStart, that is) and my previous comment was a little bit premature. As I type YouTube has only encoded the NoQuality F5 and iPoddish F18 encodings, while telling me that the video has not processed at all. Quite possibly I'm seeing nastiness that my userscript (HDSuite) makes visible, when it would just be hiding those from me if I were running in vanilla YouTube without the userscript. (Update: all encodings by YouTube are completely messed up on audio sync -- I've been trying for months to reproduce these bugs and now that I do it, it's because I'm actually adhering to YouTube's recommendations? How sweet it that?

Right now I'm uploading the same flattened render to Vimeo for comparison. I played the video locally and there were no obvious defects, though I may be imagining some added noise around the 20 second mark, where the audio detaches from the video almost completely. Still have not tried the second utility in the list. I really should probably wait to make these comments until more stuff has been processed and tried. I guess I'm weirdly enthused that I can finally reproduce one of the audio sync problems that others have demonstrated over and over again. Strange, though that the process taken to reproduce the problem seemingly conforms to or tries to "fix" the elements that YouTube is warning me about.

Then again, what am I thinking? This is GooTube we're talking about, right? LOL.

And another bit of awesome, the Vimeo encoding goes off without a hitch, or if there is one, it's too subtle for me to hear or see. Same file uploaded there as in my last YouTube example. Wahoo!

Gadzooks, audio sync exists! (on Vimeo, at least)
CClub wrote on 6/3/2010, 3:06 PM
Is anyone still using the http://renaun.com/blog/2008/02/25/250/QTIndexSwapper[/link] to move the moov atom to the beginning to allow progressive streaming?
BibbityBoo wrote on 6/4/2010, 5:03 AM
@CClub - Haven't tried that one yet, but thanks for adding to the list. I should probably hold off on comment until I've done more upload tests, but it seems both MP4FastStart and MP4Patch mentioned above may eliminate the error warning if you make sure you don't save project markers in your render.

If you leave that on by default, you do get the message about "lists" that seems to at least sometimes trigger some wild audio sync self destruct, especially with MP4FastStart. I'm guessing the same may be true for other utilities that put the index at the head, not the tail?

Hopefully I'll get some kind of better feedback from YT staff on this that I can report in the next few days or weeks. I'll revise this entry if something markedly changes or I find I've been too optimistic again.