HDV to Blu-ray: why is it recompressing?

Comments

malowz wrote on 10/4/2011, 7:43 AM
yes, its like h.264 profiles.

HDV is MP@H-14, and BD requires MP@HL i believe.

and also, yes, i believe just "patching" the header can make HDV 100% compliant, as MP@H-14 is a "lower" level than MP@HL

so, its a matter of how the program handle the file. but its not a requirement, as the file "as is" is incompatible.

EDIT: looks like the level/profile is recorder in 1 bite in the file, in every GOP. so its necessary a software to "process" the file to do this.
Julius_ wrote on 10/4/2011, 11:02 AM
I just tried tsMuxeR..popped the file in DVDA and it still recompressed (will have to re-check my procedure)..

Is this working for everyone?
NickHope wrote on 10/4/2011, 11:51 PM
tsMuxeR can't do it. TMPGenc Authoring Works and Encore are the only 2 I know of that don't recompress HDV.

Thanks for the info Malowz. That explains a lot.
PeterDuke wrote on 10/5/2011, 8:05 AM
DVDA might be lazy and only look at the first GOP of a file. In that case we might only need to patch the byte in the first GOP.

BUT it might confuse a BD player if we did so :(

@malowz,
would you be able to write a simple command line filter to patch all bytes?
malowz wrote on 10/9/2011, 4:37 PM
patch all bytes: i believe no.

my programming skill are 0 to none, the best i do is batch scripts ;)

im looking to see a "pattern" in the area where the bite is found, to be able to do a search/replace with CLI tools. but i think it need a correct analisys of the mpeg file to be able to find the correct point.

i believe i won't succeed :(
malowz wrote on 10/9/2011, 8:57 PM
mmm i may have some sort of success, but not the way i wanted...

even with a hdv file (m2t) with correct profile, DVDA want to recompress.

the solution? export the file from vegas in HDV, but in PROGRAM STREAM. if profile is corrected, it won't recompress,

then, use my patch:
NEW VERSION IN BELOW POSTS

WARNING: make sure to do yourself backup of the files. i guarantee nothing. if you afraid of viruses, test the file in a online scanner:
http://virusscan.jotti.org/en

also, if some people can upload some native captured HDV files for me to test would be nice, as i have only a few.
NickHope wrote on 10/10/2011, 12:55 AM
Lol I like that... Malowz claims to have zero programming skills and then casually writes an app (complete with GUI) to fix a problem that us [HDV > Blu-ray] people have been stuck with for years. If that is zero programming skills then mine are minus infinity :)

Testing now...
malowz wrote on 10/10/2011, 1:00 AM
actually, i didn't write it... its a software to make patches... i just did the string configuration...

i tried with larger files, and it failed... tried dozens software's...

then i find one, that works in command-line, and worked perfectly with small and 5gb files i made in vegas to test...

also, i tried using program stream instead of elementary stream, cause architect need to do a "index" when importing, and may take a lot of time, while program stream it opens instantly.

also, allow us to reopen the file in vegas .P

will post updade soon.
NickHope wrote on 10/10/2011, 1:55 AM
Just tested here in DVDA 5.0b.

I tested with a 16-second 1080-50i HDV file captured with HDVSplit, and with an 8 second 1080-60i HDV file that was rendered from Vegas. In both cases your patch allows the file to pass through DVDA without re-rendering. This is fantastic! Hats off Sir!

Getting the file open is a bit confusing at first. For anyone testing, you have to click "Patch" on the default video.m2v file, and then it allows you to search for the file you want to patch.

If Malowz can do this in his coffee break then surely SCS can manage to get it into DVDA?

In the meantime would someone be able to get this functionality into a custom command?
malowz wrote on 10/10/2011, 2:20 AM

NEW VERSION, tested with larger files. now in command-line.

http://www.mediafire.com/?k6vmee1go6frvyf

(Edit from Nick Hope: Link dead. New link here)

README included, with instructions ;)

working fine so far, but need more testing. "use at your own risk" ;)

tested only in XP32.

NickHope wrote on 10/10/2011, 2:35 AM
Malowz, I much prefer the first version, which I have sitting at C:\Program Files (x86)\HDV Patch\HDV Patch.exe with a shortcut from my start menu.

That first version could access a .m2v file from anywhere and write the patched file to the same location, whereas the new version has to be moved, or the .m2v file moved to it.

Further to that, it looks like the file now has to be renamed to "video.mpg" before it can be patched, whereas the first one could deal with any file name.

Also, isn't .mpg officially the extension for an MPEG program stream? Wouldn't .m2v (or .mpv) be more theoretically correct?

Small point on the readme file... You could mention that 50i is OK as a render template as well as 60i.
malowz wrote on 10/10/2011, 2:42 AM
well... the first patch is not working correctly with larger files... so it is not a option anymore... the patcher cannot handle larger files, so i cannot fix it somehow. so, i needed another way ;)

this first version is more a "test version". i need to know if it works OK first, then i can make other improvements. i can later make an option in the right-click menu in windows explorer (right click on file> patch HDV) or a shortcut in the "Send To" menu ;P

as mentioned, mpeg (yes, program stream) is better to work in architect (more speed) and it is able to re-import in vegas if needed. so, its more practical. DVDA works fine with program stream.

tnks, will note the 50i info.
NickHope wrote on 10/10/2011, 2:54 AM
Apologies. I should have read the readme more carefully. I see now you've changed strategy from elementary to program stream.

At what file size/length was the first version breaking down?

A context menu to patch would be great.

A "Render for DVDA (no-recompress)" custom command would be even better, if such a thing is possible.
malowz wrote on 10/10/2011, 3:14 AM
i was testing with 100mb~250mb files, then i jumped to 5gb, to make sure it was able to handle 4gb+ data. as BD can use 25 to 50gb, it reeeeallyyy need to handle large files. the first program that worked generated a file truncated. the second one entered in a "internal loop" and didn't finish after 20 mins.

this command line version, was the third that worked, but the only one correctly with large files. tested about 12 programs with search/replace functions.

i may add to the "send to" menu, as it is simpler. context menu i may need to change association of .mpg to work correctly.

custom command? pssss, not a chance :P
PeterDuke wrote on 10/10/2011, 3:33 AM
The limit is probably 2GB for signed arithmetic or 4GB if unsigned is being used.

2GB is shorthand for 2,147,483,647 the max 32 bit signed integer
NickHope wrote on 10/10/2011, 5:29 AM
Thanks for all your hard work on this Malowz. It is greatly appreciated.

Wasn't necessarily asking you to code a custom command, but one of the scripting boffins (rosebud, JR, ET etc) might be able to tackle it, now that you've cracked the underlying patch.
Kimberly wrote on 10/10/2011, 8:31 AM
Hey Gurus:

I "think" I've used DVD-A 5.2 (Build 124) to make DVDs and BD-format discs on DVD without recompressing my HDV footage and related audio file. I say I "think" because I may be missing the point of the above discussion.

I followed the instructions in the DVD-A help on Recompression, Avoiding. I saw the no recompression required message for my footage, but my menus recompressed because the media and text was not compliant. It is my understanding that DVD-A menu text will always be non-compliant even if the background media for menus is compliant.

I mention this because there must be a gap in my understanding versus what you all are seeing and I want to understsand this piece.

Regards,

Kimberly
NickHope wrote on 10/10/2011, 9:32 AM
Menus will always recompress in DVDA because you're building a new composite of text, images, video etc..

Recompression is also always going going to occur somewhere in the chain with HDV put on standard definition DVD, because of the downscaling.

But let's say you have a lot of HDV on your Vegas Pro timeline that doesn't have any fx, correction, titles etc.. Then you would want to render the file in such a way that Vegas Pro doesn't recompress those virgin parts, in order to maintain quality (and render faster). HDV would be one such format, although mxf and xdcam-mp4 will also smart render at settings that match HDV ("smart render" means no recompression). The problem is that DVDA then won't smart render those formats. Besides the "video will be recompressed" message, preparing the blu-ray also takes several times longer than if no recompression is required.

Now, if you've really been able to build a blu-ray without recompression, you're the first person I've heard of who has managed to achieve that, and I would love to know what settings you used when you rendered the file from Vegas. I'm on 5.0b, not 5.2, so perhaps there is some new trick there. My help doesn't have a "Recompression, Avoiding" section as such.
NickHope wrote on 10/10/2011, 9:54 AM
I just remembered that both MXF and XDCAM-EX (.mp4) render an HDV timeline at better quality than MainConcept MPEG-2. By that I mean the parts of the timeline that need to recompress (i.e. the transitions, fx, and the parts at the ends of HDV events that are outside the normal half-second GOPs). They'll also smart render the untouched HDV parts. The are basically MPEG-2 streams inside an MXF or MP4 container, but they seem to use a newer, better MPEG-2 codec than the MainConcept one. In addition to that, the XDCAM-EX mp4 render is blisteringly fast.

Unfortunately there are not as many advanced options in the render dialogs for these formats, so it looks difficult to add support for them in Malowz' patch. But if anyone happens to continue the great work that Malowz has done, it would be great to support them. An app that demuxed then patched the MPEG-2 stream from a Vegas XDCAM-EX MP4 file, then remuxed it to .mpg, would be the ideal.
malowz wrote on 10/10/2011, 10:57 AM
i have Architect 5.2 b124 for my tests. in all cases ive tested, it recompress HDV (non-patched)

EDIT: i talked with diablo2oo2's (creator of the program i used in the first patch release) and it said it may change the program to support 4gb+ files ;)

Kimberly wrote on 10/10/2011, 9:48 PM
I have NTSC HDV. I did some tests on this in July. Here are the templates I used for render in Vegas 10d for use in DVD-A.

DVD
This did not of course smart render in Vegas, nor did I expect it to. When I chose the "Optimize Disk" function in DVD-A, I saw a message in DVD-A that no recompression was required.



Blu-ray
I chose the smart render template ( = ) in Vegas. I used no transitions, FX, color correction, etc., to ensure the smart render in Vegas. I also saw the message in DVD-A that no recompression was required when I chose Optimize Disk. Note> I burned a BD format disk on DVD. It played in my BD player which is a newer Sony model.



The help in DVD-A also discusses how to prepare the audio to avoid recompression on those files. I followed the guidance for that and received the no recompression required message on those files too.

When I burned my disks, I saw a recompression message. I did some research on the DVD-A forum. I learned that the menus objects, text, etc. will recompress because they are not compliant.

Based on the above, I assumed my video and audio would not recompressin DVD-A if prepared according to the instructions. But apparently this is not the case?
PeterDuke wrote on 10/10/2011, 10:34 PM
It only takes a moment to render the menu, but about twice real time on my computer to recompress HDV.

What happens if you create a Blu-ray disc image without a menu (single movie)? Do you still get recompress?
Julius_ wrote on 10/10/2011, 10:34 PM
Wow...if there is one thread that everyone (us users) is contributing to finding a solution, this is it!

Great work everyone in trying to find a solution to this problem. I am very impressed with everyone's determination in finding a solution so that we can deliver the best possible quality to our clients.

Keep it up!
Kimberly wrote on 10/10/2011, 10:40 PM
What happens if you create a Blu-ray disc image without a menu (single movie)? Do you still get recompress?

I will give it a try and see. I've never made a single movie -- always had a menu. Never even thought about it but now I'm curious to see what happens.