AVC or HEVC for Archiving Video?

Seth wrote on 7/30/2015, 10:46 PM
Apologies if this has been covered. Searched HEVC and found nothing directly applicable, but this is the same conversation many content producers were having about AVC vs MPEG2 not too long ago.

I'm in the process of converting dozens of hours of Pf24 HDV into Cineform while HDLink is still publicly available, then using active metadata to apply subtle LUTs as a final sendoff before it is compressed to it's final archive format, and uploaded to cloud storage for redundant backup.

HERE IS THE $6M QUESTION:

What is a better option for archiving footage today, and potentially for the next 5-10 years: AVC [H.264] or HEVC [H.265]? And, of course, WHY?

Comments

VMP wrote on 7/30/2015, 11:28 PM
What bitrate are you planning to use?

Is there a reason you are not considering the Sony MXF format?

There is another thread about archiving and MXF, can't remember what the title of the thread was.
But MXF is also used as a broadcast standard and is good for archiving, even re-edit if needed. (Best would be the source files of course.)

I wouldn't choose AVC due to its brightness level issues which can be off (Not sure if everyone has this issue).
MXF doesen't seem to have that problem.

What ever compressor you use: native resolution + high bitrate= less compression /quality loss.
You could also consider two pass rendering, if available.


VMP
musicvid10 wrote on 7/31/2015, 7:55 AM
I guess it depends on your definitiion of "archiving."
An archival format is either a copy of the source or a lossless render, which may actually be larger than the original.

Lossy, long-GOP compressed formats, like those mentioned, are for storage and playback. Only in special circumstances are they used as all i-frame lossless formats, and in those cases offer no advantage whatsoever over the more conventional archival solutions that have been around for decades. A true archival format is not designed to be played back, but to retain data.

Given that HEVC is designed as a higher-compression alternative to AVC, albeit a LOT slower, I see the discussion of archiving using either format as little more than a red herring.
Seth wrote on 7/31/2015, 3:45 PM
Hmmm, the process I mentioned already includes upconverting to a visually lossless wavelet encoded format- with greater fidelity than JPEG2000 in an mxf wrapper, and far smaller filesize than something ridiculous like HUFFYUV- And since that small step up in size will result in files much larger than the current crop of cloud storage solutions can support [even the business solutions], and is also functionally useless to anyone involved in the metadata tagging/transcription process, there is a need for a relatively high fidelity, but low bandwidth viewable files. This must precede the final archival of the Cineform intermediates. [which may eventually give way to some other format, if a better/more flexible codec can be discovered]

The fundamental problem with "the more conventional archival solutions that have been around for decades" is just that: THEY HAVE BEEN AROUND FOR DECADES. There was no accounting for Moore's law, and little consideration given to what the average household compute power would be in 2015, let alone the obvious trajectory towards the exponential increase we should see in heterogenous, GPGPU, and ARM computing by 2020.

Speaking of red herrings, ever ask a question only to have it answered by more questions?
astar wrote on 7/31/2015, 3:53 PM
I would keep the footage in the native state, as it is already highly compressed. Why lose another generation? DV is a really old by todays standard, but NLE are still supporting the format. HDV will be the same. 12GB/hr is pretty compressed already, and HDD/SSD are getting so big and cheap get the footage off the tape in the native format. You can use Vegas to smart render clips of the media you want to keep vs the whole tape with outtake media you will never use. that will save you more space than converting to some format more compressed than the native.

I find it easier and less time consuming to keep 2 DV devices in the closet, and my tapes in a safe place, if I need to go back to original I can. I only keep final renders in DV format on disk. There is to much DV/HDV/AVCHD out there for NLEs not to support them into the future.
Seth wrote on 7/31/2015, 4:02 PM
AGREED 100% Definitely going to be keeping the physical HDV cassettes in a climate-controlled, safe place for as long as they last, and I have a Canon HV20 lying around for one reason only: logging/capture.

The biggest issue I'm facing now is that there was never an official 1080 24p 3:2 pulldown written into the HDV spec, so the 1080 Pf24 files I'm working with have never been supported fully- even in VP- and are unlikely to receive further development.

Cineform is discontinuing all HDLink products at the end of this year, and they are still the only automated telecine-removal protocol for Pf24 HDV.

This is, in great part, why I'm feeling the urgency to get these interviews into a 24p native format, and prep them for media management in an IT-based workflow moving forward.
musicvid10 wrote on 7/31/2015, 4:18 PM
Yes, Cineform is an archive format.
To be clear, I do not consider AVC or HEVC to be archival in their compressed states, yet you do.
You didn't mention keeping the Cineform intermediates originally, so this is indeed a discussion over a definition.
Seth wrote on 7/31/2015, 4:35 PM
It seems I didn't explain my workflow completely. Sorry about that.

With current cloud storage options there are still file size limitations, so a compressed format is one form of archive: a worse-case-scenario archive.That might put us on the same page.

So, the question I would love to receive input on is regarding AVC vs HEVC.
musicvid10 wrote on 7/31/2015, 8:26 PM
HEVC has more efficient compression than AVC, so it is better at low bitrates.
It also takes much longer to encode.
Steve Mann wrote on 7/31/2015, 11:30 PM
An offsite hard disk storage is the cheapest of all solutions. Cloud storage can be prohibitively expensive.
Seth wrote on 8/1/2015, 1:16 AM
Last year I thought the same thing, but this year looks much more promising: Google Drive business accounts are now functionally unlimited- as long as you are at the $50/month tier.

However an $85/yr Office 365 subscription comes with 1 TB of "ever-growing" OneDrive. Fill up your 1 TB bucket, and the account limit grows to 10 TB overnight. Fill that up, it grows to 20 TB, and so on.

I am using OneDrive, and have been able to upload at my ISP's full bandwidth, day and night.
Seth wrote on 8/1/2015, 1:27 AM
Apparently Nvidia has built in hardware H.265 encoding into the last 3 generations of GPU, Kepler, Maxwell, and Maxwell 2nd gen. But the only software I
m seeing actually leverage this technology is Mediacoder. http://blog.mediacoderhq.com/mediacoder-nvenc/

The x265 project seems unable to use OpenCL, Cuda, or NVENC. I ran a CPU-only test, and it would take roughly a month of nonstop rendering on my fastest machine to get through all of this footage.

There don't appear to be any public benchmarks for NVENC. Has anyone on the forum used it?