Main Concept m2t for HDV archive... lossless?

NickHope wrote on 11/5/2006, 5:52 AM
Just got back from my first outing with the Z1 underwater and now I have 30-minutes of 1080i50 HDV clips laid out along my Vegas 7 timeline. There are no effects, just the raw footage trimmed to length with the wobbles and rubbish discarded.

Now I would like to archive these trimmed clips to hard drive using a lossless format for all sorts of future purposes.

I've found out that "copy and trim media with project" does not actually copy the trimmed HDV clips... it copies the whole file.

I'd prefer to avoid the large storage size of Cineform if possible.

So the obvious choice seems to be to render the clips as Main Concept MPEG-2 HDV 1080-50i m2t files which are 20% the size of Cineform AVIs. Bearing in mind there are no effects etc on the clips, my question is... is this a lossless format or will I be sacrificing some quality from the original files? Is it actually re-rendering?

Or is there a better format/method?

Thanks!

Comments

JohnnyRoy wrote on 11/5/2006, 6:36 AM
M2T is not a lossless format and your footage will be re-encoded to re-establish the 15 frame GOP from your edits even though you didn't do anything else to it

No there is no other format that is lossless and small. (in fact, "lossless" and "small" are oxymorons, by definition lossless is large because there is no loss) For lossless 2-3x compression is as good as it gets. Cineform is not lossless but it can withstand a lot of generations before the loss becomes visible just like Sony's DV codec.

Here is an example of the "cost" of lossless: the Huffyuv codec is lossless. I just did a 5 second render of M2T, Cineform and Huffyuv:

M2T = 17MB
Cineform = 57MB
Huffyuv = 248MB!

As you can see, lossless is not small!

~jr
Wolfgang S. wrote on 11/5/2006, 6:40 AM
Even the m2t files from your Sony are not looseless, and nor are the Cineform intermediates. So, what you can do is to store your video as m2t files as received after capturing - with the best quality available since that format derives from your camcorder directly.

Desktop: PC AMD 3960X, 24x3,8 Mhz * RTX 3080 Ti (12 GB)* Blackmagic Extreme 4K 12G * QNAP Max8 10 Gb Lan * Resolve Studio 18 * Edius X* Blackmagic Pocket 6K/6K Pro, EVA1, FS7

Laptop: ProArt Studiobook 16 OLED * internal HDR preview * i9 12900H with i-GPU Iris XE * 32 GB Ram) * Geforce RTX 3070 TI 8GB * internal HDR preview on the laptop monitor * Blackmagic Ultrastudio 4K mini

HDR monitor: ProArt Monitor PA32 UCG-K 1600 nits, Atomos Sumo

Others: Edius NX (Canopus NX)-card in an old XP-System. Edius 4.6 and other systems

NickHope wrote on 11/5/2006, 6:49 AM
Thanks for clarifying that JohnnyRoy.

Wolfgang, I often find myself shooting a minute of something but only want to archive say the best 10 seconds. So archiving the complete original m2t files is going to take up a lot of extra space.

Big hard drives are still expensive here in Thailand so I'm trying to avoid using Cineform, but I do want to preserve my stock footage at the best quality. So for just a single render (one generation), how much "worse" might a Main Concept m2t encode be than a Cineform AVI? I can't really judge this for myself because I don't have an HD monitor yet.

Also is the 2.7 Cineform codec in Connect HD significantly higher quality than the 2.5 one in Vegas?
fldave wrote on 11/5/2006, 7:24 AM
Nick,

One option could be to keep the original m2t (they are relatively small, unless your 30 minutes is out of 4 hour total!), and use the veg file as your media in future projects, aka "Nesting". That way, absolutely no conversion, the veg filters out the footage you don't like or need. There is slight overhead using Nesting, but I find this method works well.

You can experiment with m2t-m2t generational loss, it should hold up pretty well for 2-3 conversions. Try it and see if it still meets your standards, should be fairly quick to do.
Xander wrote on 11/5/2006, 7:26 AM
I would suggest something like TMPGEnc MPEG Editor in your case. This will allow you to trim the .m2t files without having to re-encode. It only re-encodes parts that were changed and leaves the rest untouched. Thus, it will be no worse than the original after editing.
Wolfgang S. wrote on 11/5/2006, 7:38 AM
< So for just a single render (one generation), how much "worse" might a Main
< Concept m2t encode be than a Cineform AVI?

For a pure m2t file, without any effects and corrections? If you render with best, you will not be able to see a difference by eye with Vegas 7.

Yes, there is a drop in quality you can measure - but that does not mean that you will see that at all.

Maybe in addition, why not keep the original footage on tapes?

Desktop: PC AMD 3960X, 24x3,8 Mhz * RTX 3080 Ti (12 GB)* Blackmagic Extreme 4K 12G * QNAP Max8 10 Gb Lan * Resolve Studio 18 * Edius X* Blackmagic Pocket 6K/6K Pro, EVA1, FS7

Laptop: ProArt Studiobook 16 OLED * internal HDR preview * i9 12900H with i-GPU Iris XE * 32 GB Ram) * Geforce RTX 3070 TI 8GB * internal HDR preview on the laptop monitor * Blackmagic Ultrastudio 4K mini

HDR monitor: ProArt Monitor PA32 UCG-K 1600 nits, Atomos Sumo

Others: Edius NX (Canopus NX)-card in an old XP-System. Edius 4.6 and other systems

NickHope wrote on 11/5/2006, 7:55 AM
Thanks everyone for the suggestions.

I could keep the tapes, or the original m2t files, but I do so much diving that this would become unmanageable. The bits of footage I want to archive for stock are buried amongst lots of footage of scuba diving tourists that are of no use after their souvenir DVDs have been made. So yes fldave, it's typically 30 minutes out of a 4 hour total.

Xander, if Vegas/Main Concept has to re-encode to establish the 15 frame GOP, surely TMPEG would have to do the same?

Laurence wrote on 11/5/2006, 8:22 AM
You need to go to womble.com and get yourself copies of MPEG Wizard and MPEG VCR. MPEG VCR will trim the junk off the m2t files you need without rerendering the video. So will MPEG Wizard. Until Vegas adds an MPEG smartrendering feature, these tools are invaluable.
fldave wrote on 11/5/2006, 8:25 AM
Nick, I think you should be fine with one generation. Big tip: In the MPEG2 HDV template, go to the Video tab, set the quality slider to 31, all the way to the right. Then save it as your own template "_HighQuality". Not sure if it makes that much difference, but that is what I always do with my new versions of Vegas. The default on my HDV 1080 60i template was 15, medium.

Here is a comparison of 7 generations of the same frame saved as png files. Compare away. No effects, just straight renders. My HD tv has Cartoon Network on right now, so I can't compare the video footage:)

Generations.zip (16MB) PNG files (sorry, the website seems kind of slow today)
NickHope wrote on 11/5/2006, 9:02 AM
Dave I can't see any difference at all between the generations, which is great. Did you shift the ends of the clip a few frames between each generation so that each time Vegas had to restructure the GOPs?
fldave wrote on 11/5/2006, 10:14 AM
Oops, no. The only new GOP would be between 0 and 1. I'm uploading a new test right now, will post a new link shortly
JohnnyRoy wrote on 11/5/2006, 10:22 AM
> You need to go to womble.com and get yourself copies of MPEG Wizard and MPEG VCR.

I agree. This is absolutely the best way to trim M2T files. MPEG Wizard saved my butt from re-rendering on several occasions. It is definitely worthwhile tool to have in your kit. As Laurence said, it does not re-encode, it copies the file parts. So you get the exact same quality as you did from your camera. Sorry I didn't think of it earlier.

~jr
fldave wrote on 11/5/2006, 10:38 AM
For what it's worth, here are rebuilt pngs from m2ts with moving the timeline a few frames between generations. I also added a 210/210/210 letter "A" on that one frame to add to the comparison. Also included is a V7 difference VEG comparing generations 0 and 6. Open your scopes to really guage the difference between generations.

Generations_RebuiltGOP.zip (17MB) PNG files, V7 Difference VEG
Stuart Robinson wrote on 11/5/2006, 11:16 AM
>You need to go to womble.com and get yourself copies of MPEG Wizard and MPEG VCR.<

I used Womble MPEG Video Wizard for some time and had issues with compliance, a better choice is VideoReDo, it also avoids re-encoding but seems to do a far better job of retaining file compatibility.
Laurence wrote on 11/5/2006, 11:17 AM
I'm scared of rerendering mpeg (m2t). It may look fine on a computer screen or even a less than 37 inch TV, but 50 and 60 inch HD TVs are becoming more and more common and these show off even the smallest of flaws in your video. Especially on raw footage, I wouldn't chance rerendering to less than the Cineform codec.
Wolfgang S. wrote on 11/5/2006, 11:42 AM
Be aware that womble changes a lot of things in the structure of the m2t files - what means that the material is not original footage any more. As far as I know, you cannot print to tape that material (but maybe I am wrong here). If and how far that can cause troubles later - I do not know.

Desktop: PC AMD 3960X, 24x3,8 Mhz * RTX 3080 Ti (12 GB)* Blackmagic Extreme 4K 12G * QNAP Max8 10 Gb Lan * Resolve Studio 18 * Edius X* Blackmagic Pocket 6K/6K Pro, EVA1, FS7

Laptop: ProArt Studiobook 16 OLED * internal HDR preview * i9 12900H with i-GPU Iris XE * 32 GB Ram) * Geforce RTX 3070 TI 8GB * internal HDR preview on the laptop monitor * Blackmagic Ultrastudio 4K mini

HDR monitor: ProArt Monitor PA32 UCG-K 1600 nits, Atomos Sumo

Others: Edius NX (Canopus NX)-card in an old XP-System. Edius 4.6 and other systems

wwaag wrote on 11/5/2006, 12:51 PM
Agree with Xander about TMPGenc's MPEG Editor. It's cheap, will smart-render, and the resulting files remain HDV. You can import them back into Vegas and also "print to tape". I've done this with the HC1. I've also used MPEG editor to combine short HDV clips into a single file, which I've found to be quite useful.

It's truly a shame that Vegas doesn't support smart-rendering or that no plug-in is available unlike the competition. Maybe in the next release.

wwaag

AKA the HappyOtter at https://tools4vegas.com/. System 1: Intel i7-8700k with HD 630 graphics plus an Nvidia RTX4070 graphics card. System 2: Intel i7-3770k with HD 4000 graphics plus an AMD RX550 graphics card. System 3: Laptop. Dell Inspiron Plus 16. Intel i7-11800H, Intel Graphics. Current cameras include Panasonic FZ2500, GoPro Hero11 and Hero8 Black plus a myriad of smartPhone, pocket cameras, video cameras and film cameras going back to the original Nikon S.

farss wrote on 11/5/2006, 1:22 PM
There's no way to trim a mpeg-2 stream that has a fixed 15 frame GOP without decoding and encoding and that will be lossy. If the GOP is anything other than 15 frames it's not compliant. Maybe nothing will complain, I'd imagine the VCRs certainly would. The only way you might be able to do this is if you could see where the GOPs start and end and only trim complete GOPs but I know of nothing that does this. For what Nick wants to do though it would be an ideal solution if it existed. Th e only possibility might be the M25U VCRs, they seem to claim that they can clone HDV but it does sound all a bit vague.
Short of decoding the m2t files to uncompressed, trimming that and archiving it there's no easy solution. All HiDef tape formats apart from HDCAM SR are lossy, even HDCAM SR in normal mode is lossy, it does have a way of making digital clones (sorry forgot the correct terminology). At $800 for a 60 minute tape stacks of HDDs and / or DLT tape sounds very cheap.
I've had a bit of a look at HD stock libraries and most seem to offer delivery on 35mm, I'd guess if you're in the business of selling HD stock footage you'd need to factor in what clients expectations are in the way of delivery formats.

Bob.
Laurence wrote on 11/5/2006, 1:42 PM
The Womble software has keyboard shortcuts that will take you to the previous or next ifull frame. You can cut anywhere, but it's better to cut at complete frames. Womble edited stuff cannot be copied directly back to tape, but this doesn't affect raw footage that is going to be later edited in Vegas. The actual encoding is the same. I use the Womble software this way all the time and it works exactly as it should.
farss wrote on 11/5/2006, 2:01 PM
I assume you mean at GOP boundaries?
If so then this is excellent stuff, another piece of good old Aussie innovation.
riredale wrote on 11/5/2006, 4:48 PM
You might want to calibrate your tolerance for rendering errors by first going to the Cineform site, where they have a white paper showing the benefits of Cineform. In particular, they show m2t degradation after TEN decode/recode cycles. I'd agree that it looks pretty lousy after ten cycles, but the implication to me is that a couple of cycles will look very good, if not identical for all practical purposes.

Of course, the obvious answer is to actually do a couple of cycles yourself, and see if you can live with whatever degradation there is. For me, I'd just cut out those parts of the video you want to archive, and render them back to tape as HDV (m2t).
Laurence wrote on 11/5/2006, 5:01 PM
Yes I meant the GOP boundries. You don't have to do this. You can cut anywhere but it is better if you stick to the GOP boundries because otherwise the Womble software generates a few frames at the in and out points with a weird GOP sequence. Aside from a little data at the in and out points and a new header, the trimmed files are just copies of exactly the same data (minus the video you are trimming off of course).

I never rerender to m2t except to copy back to the camera. I always use Cineform for rerenders along the way.
NickHope wrote on 11/5/2006, 7:52 PM
Thanks again, especially fldave for those png files.

Archiving to tape is not really an option because I'm constantly calling on my stock footage for all sorts of reasons, so it has to be hard drive.

I already have TMPGEnc Xpress 4 and that could do this trimming. It shades the i-frames ("keyframes") in pink so in theory it could be used to trim to whole GOPs.

The problem with all these stanadalone utilities is repetition of work. I already have 100 or so clips trimmed on the Vegas ttimeline and duplicating the cutting in another application would take a long time.

It would be great if Vegas could give an option to render an m2t file that includes a little extra head and tail that encloses the whole GOP at either end of the file, thereby losing no quality. I think this is what we are talking about by smart rendering.
NickHope wrote on 11/8/2006, 10:02 PM
I've been thinking more about a method for archiving in trimmed .m2t without losing quality from the original captured .m2t file...

If one were to use something like TMPGEnc Xpress or one of the other MPEG editors that highlight the I-frames, then one could cross-reference the timecode with Vegas and trim in the Vegas timeline so that the clip starts exactly on an I-frame before archiving. Then in theory there would be less quality loss (if any) than if the clip were trimmed between I-frames and all the I-frames were re-jigged for the whole clip. Right?

As for the end (tail) of the file, it shouldn't matter where it's trimmed to from a point of view of preserving quality.

How does that sound?