AutoBench script

Howard-Vigorita wrote on 6/30/2021, 9:40 AM

The phenomenal tutorials by @jetdv inspired me to write a little script myself. Never did a Vegas script before, other than tweaking the batch render supplied with Vegas. Also never looked at Visual Studio before but have been meaning to. So this was a learning experience in which the tutorials were crucial.

My goal is to speed up and regularize the Vegas benchmarking I've been doing manually for years. Started with the multi-project render tutorial last week and took it from there. Rendering 5 possible benchmark projects with 12 possible templates. My form ended up looking like this...

Refinements I added include reading graphics hardware and cpu info, doing 2 renders per project to determine the quickest runtime, doing a user configurable programmed delay between renders for cpu/gpu cool-down, and writing out a csv file that I feed upstream to Excel and Visual FoxPro to generate charts and a webpage. Also added a config file to speed up loading defaults on the 5 different machines I'll be running this on.

Cameras: Z-Cam E2 w/Canon EF turbo mount, Canon XF305, JVC GV-LS2, Canon 6D w/L-glass line.
Laptop: Dell XPS15-9570; i7-8750h 32gb (integrated Intel UHD-630 {.8935} & Nvidia GTX-1050Ti {472.47})
Road: Intel NUC8 i7-8809g 32gb (integrated AMD VegaM 4gb {21.8.2 [Mod]} & integrated Intel HD630 {.9415}) Win11
Workstation: i9-9900k 32 gb (Sapphire AMD Radeon VII 16gb {21.q3.1} & integrated Intel UHD630 {.9466})
Workstation2: XeonE5-1650v4 128 gb (Sapphire Nitro+ Vega64 8gb {21.q3.1} & Nvidia GTX-1660 8gb {472.47})
Workstation3: i7-980X 24gb (Sapphire Nitro+ RX580 8gb {21.q3})
under constr: i9-11900k 64gb (Sapphire ToxicLC 6900xt 16 gb {Adrenalin 21.10.2 } & Intel UHD750 {.9955} ) Win11
currently Vegas 17.455, Vegas 18.527, Vegas 19-latest all with Win10 except as noted.

Red Car & Pip Benchmarks: https://drive.google.com/drive/folders/1TK2lyFZSlzU9JQ8xO1mvoj-zl9tyj2lc?usp=sharing
Benchmark Tables: RoundUp21 , RoundUp2020, and earlier
(ffMetrics by fifonik) Encoder Quality Tables by me: http://www.rtpress.com/ffmetrics.htm

Comments

Howard-Vigorita wrote on 7/4/2021, 11:17 PM

Added a CPU field to my form above getting the data from Windows.System similarly to how I pull the Graphics Card info:

       public static String GetCpu()
        {
            using (ManagementObjectSearcher win32Proc = new ManagementObjectSearcher("select * from Win32_Processor"),
                win32CompSys = new ManagementObjectSearcher("select * from Win32_ComputerSystem"),
                win32Memory = new ManagementObjectSearcher("select * from Win32_PhysicalMemory"))
            {
                String procName = "";
                foreach (ManagementObject obj in win32Proc.Get())
                {
                    procName = obj["Name"].ToString();
                }
                return procName;
            }
        }

Regarding the Graphics Card info, is there any way I can read Vegas' config instead of getting an inventory of graphics cards installed from Windows? Also can't figure out how to get the render time off of the Vegas render screen. My script just uses the stopwatch function to time request to completion which is typically a few seconds longer. Any help would be appreciated.

Last changed by Howard-Vigorita on 7/4/2021, 11:19 PM, changed a total of 2 times.

Cameras: Z-Cam E2 w/Canon EF turbo mount, Canon XF305, JVC GV-LS2, Canon 6D w/L-glass line.
Laptop: Dell XPS15-9570; i7-8750h 32gb (integrated Intel UHD-630 {.8935} & Nvidia GTX-1050Ti {472.47})
Road: Intel NUC8 i7-8809g 32gb (integrated AMD VegaM 4gb {21.8.2 [Mod]} & integrated Intel HD630 {.9415}) Win11
Workstation: i9-9900k 32 gb (Sapphire AMD Radeon VII 16gb {21.q3.1} & integrated Intel UHD630 {.9466})
Workstation2: XeonE5-1650v4 128 gb (Sapphire Nitro+ Vega64 8gb {21.q3.1} & Nvidia GTX-1660 8gb {472.47})
Workstation3: i7-980X 24gb (Sapphire Nitro+ RX580 8gb {21.q3})
under constr: i9-11900k 64gb (Sapphire ToxicLC 6900xt 16 gb {Adrenalin 21.10.2 } & Intel UHD750 {.9955} ) Win11
currently Vegas 17.455, Vegas 18.527, Vegas 19-latest all with Win10 except as noted.

Red Car & Pip Benchmarks: https://drive.google.com/drive/folders/1TK2lyFZSlzU9JQ8xO1mvoj-zl9tyj2lc?usp=sharing
Benchmark Tables: RoundUp21 , RoundUp2020, and earlier
(ffMetrics by fifonik) Encoder Quality Tables by me: http://www.rtpress.com/ffmetrics.htm

Howard-Vigorita wrote on 11/7/2021, 1:34 PM

My script has been moderately stable for a while now so I figured I'd publish it. Not that anyone but me is likely to want to use it... but more as an example Vegas scripting project developed in Visual Studio 19. Keep in mind this is my first stab at Visual Studio, c#, and a large scale script like this. Still many things I don't know how to accomplish efficiently so any feedback would be greatly appreciated.

Just copied all the Visual Studio project directories to GitHub... that's a first for me too:

https://github.com/hv60604/AutoBench/tree/main

Last changed by Howard-Vigorita on 11/7/2021, 1:35 PM, changed a total of 1 times.

Cameras: Z-Cam E2 w/Canon EF turbo mount, Canon XF305, JVC GV-LS2, Canon 6D w/L-glass line.
Laptop: Dell XPS15-9570; i7-8750h 32gb (integrated Intel UHD-630 {.8935} & Nvidia GTX-1050Ti {472.47})
Road: Intel NUC8 i7-8809g 32gb (integrated AMD VegaM 4gb {21.8.2 [Mod]} & integrated Intel HD630 {.9415}) Win11
Workstation: i9-9900k 32 gb (Sapphire AMD Radeon VII 16gb {21.q3.1} & integrated Intel UHD630 {.9466})
Workstation2: XeonE5-1650v4 128 gb (Sapphire Nitro+ Vega64 8gb {21.q3.1} & Nvidia GTX-1660 8gb {472.47})
Workstation3: i7-980X 24gb (Sapphire Nitro+ RX580 8gb {21.q3})
under constr: i9-11900k 64gb (Sapphire ToxicLC 6900xt 16 gb {Adrenalin 21.10.2 } & Intel UHD750 {.9955} ) Win11
currently Vegas 17.455, Vegas 18.527, Vegas 19-latest all with Win10 except as noted.

Red Car & Pip Benchmarks: https://drive.google.com/drive/folders/1TK2lyFZSlzU9JQ8xO1mvoj-zl9tyj2lc?usp=sharing
Benchmark Tables: RoundUp21 , RoundUp2020, and earlier
(ffMetrics by fifonik) Encoder Quality Tables by me: http://www.rtpress.com/ffmetrics.htm

wwaag wrote on 11/7/2021, 4:32 PM

@Howard-Vigorita

Regarding graphics card info, here's a method that extracts that information. Rather than writing to a string, I'd first create a List<string> where you simply add entries to the list. Then set the datasource for your combobox to the list and the user has an option to select which GPU was being used.

private void GetGraphics()
    {
        ManagementScope scope = new ManagementScope("\\\\.\\ROOT\\cimv2");
        ObjectQuery query = new ObjectQuery("SELECT * FROM Win32_VideoController");
        ManagementObjectSearcher searcher = new ManagementObjectSearcher(scope, query);
        ManagementObjectCollection queryCollection = searcher.Get();
        string graphicsCard = "";
        foreach (ManagementObject mo in queryCollection)
        {
            foreach (PropertyData property in mo.Properties)
            {
                if (property.Name == "Description")
                {
                    graphicsCard += property.Value.ToString() + " ";
                }
            }
        }
        MessageBox.Show(graphicsCard);
    }

Edit: Here's the link to the original stackoverflow thread. https://stackoverflow.com/questions/37524111/how-to-find-active-in-use-graphic-cards-c-sharp

Last changed by wwaag on 11/7/2021, 4:37 PM, changed a total of 2 times.

AKA the HappyOtter at https://tools4vegas.com/. System 1: Intel i7-8700k with HD 630 graphics plus an Nvidia 1050ti graphics card. System 2: Intel i7-3770k with HD 4000 graphics plus an AMD RX550 graphics card. Current cameras include Panasonic FZ2500, GoPro Hero5 Black plus a myriad of smartPhone, pocket cameras, video cameras and film cameras going back to the original Nikon S.

Howard-Vigorita wrote on 11/8/2021, 11:29 AM

Thanks, @wwaag That looks more elegant than the explicit array assignment I used where I overloaded both gpu and decoder combo box lists and selection indexes into the same function:

        public void FindAllGraphicsCards()
        {
            // find graphic card names
            string[] Decoder;
            List<string> list = new List<string>();
            using (var searcher = new ManagementObjectSearcher("select * from Win32_VideoController"))
            {
                foreach (ManagementObject obj in searcher.Get())
                {
                    list.Add((String)obj["Name"]);
                }
            }
            GraphicsCards = list.ToArray();
            Decoder = list.ToArray();
            // allow for selecting no decoder
            cmbDecoder.Items.Add("none");
            foreach (String card in GraphicsCards)
            {
                cmbGraphicsCard.Items.Add(card);
                cmbDecoder.Items.Add(card);
            }
            // default gpu to the last graphics card and decoder to first
            cmbGraphicsCard.SelectedIndex = GraphicsCards.Length-1;
            cmbDecoder.SelectedIndex = 1;
        }

My code might be a little confusing as some diagnostic lines that have been commented out. Some other lines that didn't work out for me should probably be deleted instead of just commented out. For instance, the release version of the script doesn't actually call a MessageBox.Show on any of the hardware detected... that only appears in the FindGpuDrivers() function that I used during development to probe what gpu data the ManagementObjectSearcher could come up with. The calling line for that function has been commented out. I kept the function in because others might find it useful... but I should probably explain that in a comment... [EDIT: just did all that. hv]

Btw, any chance you know how to capture the elapsed-time variable shown in the Vegas render box? I'm still stuck on that. This script returns slightly longer run times because the timer starts when render is requested and ends when control returns. That elapsed time includes any file system delays that occur before rendering actually begins, which seem to be between 1 and 3 seconds at times. My only work around is to run renders a 2nd time after a programmed cooling-off period and record the lower elapsed time.

Last changed by Howard-Vigorita on 11/8/2021, 2:59 PM, changed a total of 2 times.

Cameras: Z-Cam E2 w/Canon EF turbo mount, Canon XF305, JVC GV-LS2, Canon 6D w/L-glass line.
Laptop: Dell XPS15-9570; i7-8750h 32gb (integrated Intel UHD-630 {.8935} & Nvidia GTX-1050Ti {472.47})
Road: Intel NUC8 i7-8809g 32gb (integrated AMD VegaM 4gb {21.8.2 [Mod]} & integrated Intel HD630 {.9415}) Win11
Workstation: i9-9900k 32 gb (Sapphire AMD Radeon VII 16gb {21.q3.1} & integrated Intel UHD630 {.9466})
Workstation2: XeonE5-1650v4 128 gb (Sapphire Nitro+ Vega64 8gb {21.q3.1} & Nvidia GTX-1660 8gb {472.47})
Workstation3: i7-980X 24gb (Sapphire Nitro+ RX580 8gb {21.q3})
under constr: i9-11900k 64gb (Sapphire ToxicLC 6900xt 16 gb {Adrenalin 21.10.2 } & Intel UHD750 {.9955} ) Win11
currently Vegas 17.455, Vegas 18.527, Vegas 19-latest all with Win10 except as noted.

Red Car & Pip Benchmarks: https://drive.google.com/drive/folders/1TK2lyFZSlzU9JQ8xO1mvoj-zl9tyj2lc?usp=sharing
Benchmark Tables: RoundUp21 , RoundUp2020, and earlier
(ffMetrics by fifonik) Encoder Quality Tables by me: http://www.rtpress.com/ffmetrics.htm

wwaag wrote on 11/8/2021, 4:58 PM

@Howard-Vigorita

"Btw, any chance you know how to capture the elapsed-time variable shown in the Vegas render box?"

No, I have no idea. There doesn't appear to be any type of rendering log that I can find. There is a Vegasaur render assistant tool that also provides render times that don't agree with the Vegas times. Just my nickel's worth, but I wouldn't worry about the 1 to 3 sec difference. I can render 5 times and get 5 different durations, so which one is the best estimate? You could take averages, but I hardly think it would be worth the time or effort given the multitude of things that can affect render performance for any given run.

AKA the HappyOtter at https://tools4vegas.com/. System 1: Intel i7-8700k with HD 630 graphics plus an Nvidia 1050ti graphics card. System 2: Intel i7-3770k with HD 4000 graphics plus an AMD RX550 graphics card. Current cameras include Panasonic FZ2500, GoPro Hero5 Black plus a myriad of smartPhone, pocket cameras, video cameras and film cameras going back to the original Nikon S.

Howard-Vigorita wrote on 11/9/2021, 11:18 AM

@wwaag That's interesting. I just looked at SeMW. It doesn't report elapsed time on its progress screen. But it does report average fps. Which if divided into the total frame count when render completes seems to nail the Vegas-reported elapsed time. Maybe I've been looking for the wrong thing. Perhaps running a timer in conjunction with calls to the RenderStatusEventHandler is the trick.

Cameras: Z-Cam E2 w/Canon EF turbo mount, Canon XF305, JVC GV-LS2, Canon 6D w/L-glass line.
Laptop: Dell XPS15-9570; i7-8750h 32gb (integrated Intel UHD-630 {.8935} & Nvidia GTX-1050Ti {472.47})
Road: Intel NUC8 i7-8809g 32gb (integrated AMD VegaM 4gb {21.8.2 [Mod]} & integrated Intel HD630 {.9415}) Win11
Workstation: i9-9900k 32 gb (Sapphire AMD Radeon VII 16gb {21.q3.1} & integrated Intel UHD630 {.9466})
Workstation2: XeonE5-1650v4 128 gb (Sapphire Nitro+ Vega64 8gb {21.q3.1} & Nvidia GTX-1660 8gb {472.47})
Workstation3: i7-980X 24gb (Sapphire Nitro+ RX580 8gb {21.q3})
under constr: i9-11900k 64gb (Sapphire ToxicLC 6900xt 16 gb {Adrenalin 21.10.2 } & Intel UHD750 {.9955} ) Win11
currently Vegas 17.455, Vegas 18.527, Vegas 19-latest all with Win10 except as noted.

Red Car & Pip Benchmarks: https://drive.google.com/drive/folders/1TK2lyFZSlzU9JQ8xO1mvoj-zl9tyj2lc?usp=sharing
Benchmark Tables: RoundUp21 , RoundUp2020, and earlier
(ffMetrics by fifonik) Encoder Quality Tables by me: http://www.rtpress.com/ffmetrics.htm

wwaag wrote on 11/9/2021, 12:09 PM

I too looked at SEMW extensions and was surprised it didn't report elapsed time. Here's an example of how I check for the render being cancelled.

var status = myVegas.Render(tempWavFile, waveTemplate, loopStart, loopLengthAudio);
if (status == RenderStatus.Canceled)
    {
        DialogResult result1 = MessageBox.Show("Direct Render from Vegas Cancelled", "Stop", MessageBoxButtons.OK, MessageBoxIcon.Stop);
        goto endofcode;
    }
WriteLogFile("Temp audio render completed from Vegas");

If you start your stopwatch just before the start and end it when status becomes Render.Completed, that may work.

Last changed by wwaag on 11/9/2021, 12:11 PM, changed a total of 1 times.

AKA the HappyOtter at https://tools4vegas.com/. System 1: Intel i7-8700k with HD 630 graphics plus an Nvidia 1050ti graphics card. System 2: Intel i7-3770k with HD 4000 graphics plus an AMD RX550 graphics card. Current cameras include Panasonic FZ2500, GoPro Hero5 Black plus a myriad of smartPhone, pocket cameras, video cameras and film cameras going back to the original Nikon S.

Howard-Vigorita wrote on 11/9/2021, 1:27 PM

@wwaag Thanks for taking the time to look at this. But I think the problem is that once I transfer control to Vegas to do the render, my script does not regain control to do anything till it's either done or cancelled. So I have to start the timer before I transfer control. Here's how I do it:

                            timer.Start();
                            DoRender(FullFileName, myRenderer, myTemplate, RStart, RLength, false, false);
                            timer.Stop();

The DoRender() function is basically from @jetdv tutorials and it indeed polls RenderStatus on return:

        public void DoRender(string fileName, Renderer rndr, RenderTemplate rndrTemplate, Timecode start, Timecode length, bool IncMarkers, bool SetStretch)
        {
            // ...
            // transfer control to Vegas requesting render
            RenderStatus status = myVegas.Render(args);
            //...
            if (status == RenderStatus.Canceled)
            {
                MessageBox.Show("User canceled");
                btnRender.Text = "Render";
                mymsg.Text = "User cancelled.";
                mymsg.Refresh();
                btnRender.Refresh();
                exit = true;
                return;
            }
       }

I think the best approach might be some kind of background worker that is created right before DoRender() is called, tests the render progress and starts the timer when it actually begins, perhaps when progress is >0%, stops the timer when progress hits 100%, and is then destroyed. Rather than the way I'm doing it. Ha, ha, easy for me to say, having never messed with background workers in Vegas before.

Also thinking a more simplified approach more within my skill set might be to call myVegas.Render(args) twice, setting the RLength arg to zero the first time before setting it to the actual length and starting the timer for the real deal. Kind of hokey but might minimize any file system setup delays that only seem to precede the very 1st render no matter how many different projects I do in the same run.

Last changed by Howard-Vigorita on 11/9/2021, 1:52 PM, changed a total of 3 times.

Cameras: Z-Cam E2 w/Canon EF turbo mount, Canon XF305, JVC GV-LS2, Canon 6D w/L-glass line.
Laptop: Dell XPS15-9570; i7-8750h 32gb (integrated Intel UHD-630 {.8935} & Nvidia GTX-1050Ti {472.47})
Road: Intel NUC8 i7-8809g 32gb (integrated AMD VegaM 4gb {21.8.2 [Mod]} & integrated Intel HD630 {.9415}) Win11
Workstation: i9-9900k 32 gb (Sapphire AMD Radeon VII 16gb {21.q3.1} & integrated Intel UHD630 {.9466})
Workstation2: XeonE5-1650v4 128 gb (Sapphire Nitro+ Vega64 8gb {21.q3.1} & Nvidia GTX-1660 8gb {472.47})
Workstation3: i7-980X 24gb (Sapphire Nitro+ RX580 8gb {21.q3})
under constr: i9-11900k 64gb (Sapphire ToxicLC 6900xt 16 gb {Adrenalin 21.10.2 } & Intel UHD750 {.9955} ) Win11
currently Vegas 17.455, Vegas 18.527, Vegas 19-latest all with Win10 except as noted.

Red Car & Pip Benchmarks: https://drive.google.com/drive/folders/1TK2lyFZSlzU9JQ8xO1mvoj-zl9tyj2lc?usp=sharing
Benchmark Tables: RoundUp21 , RoundUp2020, and earlier
(ffMetrics by fifonik) Encoder Quality Tables by me: http://www.rtpress.com/ffmetrics.htm

jetdv wrote on 11/9/2021, 2:06 PM

When starting a custom command, you can actually set up a "watch" for: RenderStarted, RenderFinished, and RenderProgress. Your watch routine would be called when these updated.

http://www.jetdv.com/2021/11/08/building-a-custom-command-in-vegas-part-6-watching-for-project-changes/