Archival ideas

thwil wrote on 6/2/2017, 10:48 AM

Usually after finishing a project, I archive used footage. With 1080p it used to be not so much of an issue, but I'm shifting towards UHD and files are now big. It's very common when out of every ~1-minute-long take I only use 5-10 seconds. Unfortunately, in the Vegas universe it is a popular belief that H.264 files cannot be trimmed, so when I archive my projects, media piles up to quite huge amounts. I cannot really afford to just keep everything so I'm looking for ways to optimise this.

H.264 files have I-frames and there are tools that allow lossless cutting (one of them is free/opensource and is creatively called Lossless Cut). It's not rocket surgery and I really hope that MAGIX developers know that and we will see smarter project export in the future.

Meanwhile, suppose I'm ready to commit myself to doing all the re-trimming manually using an external tool. That's fine. Naturally I can only do this after the project is complete so that I know which pieces I want to keep. But MSP does not seem to show anywhere which parts of the clips are used in the timeline? That makes the job pretty much impossible.

But ok, I heard that intraframe codecs should be fine. In that case I should be able to transcode source media into DNxHR, export the project with trimmed copies, and then transcode the trimmed clips back into H.264 for archival, with relatively small quality loss, right? Right?? No, MSP cannot trim DNxHR either, even though it's an intraframe codec designed specifically for cutting.

So what are your tricks for efficient archival? Please share. Just buying more HDDs is not a trick.

Comments

thwil wrote on 6/2/2017, 1:48 PM

@Cornico yes, does the trimming work for you? Which codec do you use if it does?

thwil wrote on 6/2/2017, 3:27 PM

It does not work, this is why this thread exists. If you have nothing constructive to say please just ignore my post.

Jack S wrote on 6/2/2017, 4:51 PM

thwil

But MSP does not seem to show anywhere which parts of the clips are used in the timeline? That makes the job pretty much impossible.

When you've completed your project, go to Tools and select Clean Project Media.

This will remove all items under your Project Media tab that don't currently exist on your timeline.

You can then use the list of media that remains as your list to archive.

However, you can save yourself a load of trouble by following Cornico's tip regarding the Save As option.

Be prepared to wait though as this can take a long time with a big project.

Which codec do you use if it does?

The trimming is carried out by VMS and requires no intervention from you (ASFAIK).

It does not work, this is why this thread exists.

I know the trimming works in MSP13 but I haven't tried it in VMS14.

My system
Genshin Infinity Gaming PC
Motherboard Gigabyte H610M H: m-ATX w/, USB 3.2, 1 x M.2
Power Supply Corsair RM750X
Intel Core i7-13700K - 16-Core [8P @ 3.4GHz-5.4GHz / 8E @ 2.50GHz-4.20GHz]
30MB Cache + UHD Graphics, Ultimate OC Compatible
Case Fan 4 x CyberPowerPC Hyperloop 120mm ARGB & PWM Fan Kit
CPU Fan CyberPowerPC Master Liquid LITE 360 ARGB AIO Liquid Cooler, Ultimate OC Compatible
Memory 32GB (2 x 16GB) DDR5/5200MHz Corsair Vengeance RGB
MSI GeForce RTX 4060 Ti 8GB - Ray Tracing Technology, DX12, VR Ready, HDMI, DP
System drive 1TB WD Black SN770 M.2 NVMe PCIe SSD - 5150MB/s Read & 4900MB/s Write
Storage 2 x 2TB Seagate BarraCuda SATA-III 6.0Gb/s 7200RPM
Windows 11 Home (x64)
Monitors
Generic Monitor (PHL 222V8) connected to GeForce RTX 4060 Ti
Generic Monitor (SAMSUNG) connected to iGPU

Camcorder
SONY Handycam HDR-XR550VE

thwil wrote on 6/2/2017, 6:26 PM

@JackS I guess it should work at least in some cases, otherwise why would this option be added. But it does not work on the source media that I have, which is predominantly H.264 files in mp4 and mov containers.

Immediately after Saving As with media, I get a message saying that "Some of the project files were not trimmed to preserve quality". And all media is simply copied in its entirety. Nothing gets trimmed.

Anyway, the only workaround I've found so far is:

* create subclips for all events
* for every subclip in the Properties dialog/Media tab a start time code and duration will be displayed
* this information can be manually used to compile a list which then can be fed to a script that uses ffmpeg to create trimmed copies of all events.

Former user wrote on 6/2/2017, 8:49 PM

As thwil has noted (3 times) when using the archive feature, only certain codecs are actually trimmed to their used length. Others are copied over as full files, thus negating the point of using the archive function. You might as well copy over the project folder. I myself would not convert to another format for archiving unless you are using a lossless or uncompressed codec. Any other conversion will cause a loss of quality to some degree.

3POINT wrote on 6/3/2017, 6:03 AM

The Save as option for trimmed copies of source media works only for DV-avi, as far I can remember, a relict from those days when HDspace was rare and expensive. I don't mind to archive a complete clip, since I already try not to shoot unnecessary long (UHD) footage. When I need only a few seconds of a scene, I shoot max only twice as long as needed. The time and effort to archive "trimmed" clips doesn't weight against costs of extra HDspace.

thwil wrote on 6/3/2017, 7:56 AM

Shooting only as much as you need is the smartest thing to do. But I'm mostly filming my kid, and it's always shoot and hope for something to happen. Some takes are just throwaway, some are spot on, but quite often it's a 2 minute take for 5 seconds of action.

Anyway, as an experiment I tried to do as I suggested in my previous post. I created subclips, wrote down a text file with source clip name, in and duration marks, and wrote a .bat file to create trimmed copies. I do not preserve the project structure by doing so, unfortunately, but I get to keep the originals and source archive is 2x smaller. It didn't really take a very long time, although it's really stupid to do such things by hand in 2017. If anyone's interested I can try describing it in more detail.

It's rather strange that I have to go through such hoops, especially because MSP already has the feature that should do all of this automatically, but in reality only works with formats that are irrelevant in this day and age. MAGIX, looking at you.

3POINT wrote on 6/3/2017, 9:31 AM

According Help only following formats are (auto)trimmed:

  • Audio/video events are saved in a new AVI file. You can trim DV AVI,uncompressed AVI, and YUV AVI files. Because of the lossy nature of other videoformats, those video files will not be trimmed, but will simply be copied to the project folder.

Normally I keep all recordings (on external backup HDD) or delete them if there's nothing useable. If there is really an important 10 second part in a one hour recording I would just rerender that part in the same format as the original, the quality loss of the one generation render will be not visible .

 

Mats wrote on 6/3/2017, 11:51 AM

Very good question thwil. I would myself see a solution to this. But I´m sorry that I don´t have an answer.

I have tried Cyberlinks Powerdirector which claims it can "Smart render".

-"Smart Video Rendering Technology (SVRT) makes video production faster by analyzing your project and rendering only segments of footage that have been modified."

That is even for mp4-files.
I have tried it but found that there are some issues with the resulting video file which is not satisfying. The picture "jumps" at those places where the rendering has taken place.

So maybe it is not so easy to get a program to work with mp4-files in the same way as with DV-AVI. Even if it´s 2017 ;-)

(Sorry for my crappy english.)

thwil wrote on 6/3/2017, 4:57 PM

Thanks for your input everyone.

To sum up my findings, here's a sample of what I've got in case someone wants to try it.

The hand-compiled list of cuts looks like so. I put it in a file trim.txt in a subfolder where my project media is located. For example the project media is in C:\Users\Myself\Project1, this will be in C:\Users\Myself\Project1\trimmed\trim.txt. The first column is the file name, the second one is the timecode for cue start, the last one is duration.

C4K_20170527_124113.mp4 12.000     10.0
C4K_20170527_124113.mp4 34.000     14.0
C4K_20170527_124113.mp4 56.000     6.0
C4K_20170527_124838.mp4 10.0     6.0
C4K_20170527_124838.mp4 30.0    8.0
C4K_20170527_124838.mp4 44.0     20.0
C4K_20170527_125612.mp4 2.0    6.0
C4K_20170527_125612.mp4 29.0    25.0
C4K_20170527_125722.mp4 3.0    19.0
C4K_20170527_125819.mp4 0.0    11.0

Next to it there is a .bat file, let's call it trim.bat with such contents:

for /F "tokens=1-3" %%a in (trim.txt) do call :trimfile %%a %%~na %%b %%c
goto :eof

:trimfile
rem fullname nameonly start duration
echo %1 %2 %3 %4
set /a counter=0
:loop
set /a counter+=1
set fo="%2-%counter%.mp4"
if EXIST %fo% goto :loop
C:\bin\ffmpeg\bin\ffmpeg.exe -ss %3 -i ..\%1 -t %4 -c copy %fo%
goto :eof

The key here is of course a relatively fresh build of ffmpeg.

After I run this script, I end up with files trimmed according to the trim list, named like so:

C4K_20170527_124113-1.mp4
C4K_20170527_124113-2.mp4
C4K_20170527_124113-3.mp4
C4K_20170527_124838-1.mp4
C4K_20170527_124838-2.mp4
C4K_20170527_124838-3.mp4
C4K_20170527_125612-1.mp4
C4K_20170527_125612-2.mp4
C4K_20170527_125722-1.mp4
C4K_20170527_125819-1.mp4

It is very easy to see that no re-encoding takes place at all, all output media is direct stream copy.

If you want to try it on just one file without bothering with lists, .bat files etc, the syntax is:

ffmpeg -ss {start-timecode}  -i {input-file} -t {duration} -c copy {output-file}

Or you could do the same manually in GUI using Lossless Cut: https://github.com/mifi/lossless-cut -- which is nice to have, but I prefer scripted approach.

NickHope wrote on 6/3/2017, 10:36 PM

If you use Vegas Pro, this thread explains how you can automate the trimming of files with ffmpeg from the timeline, via scripting.

thwil wrote on 6/5/2017, 7:01 AM

@Nick Hope unfortunately I don't use Vegas Pro. Maybe later on I decide that it's worth it for me, but I don't really see it coming. It's nice to see that we came up with essentially the same idea.