Betas are a very good idea - see i.e. the Windows 7 release.

BUT you would have to consider the extreme noise from the uber whiners too... who probably would anticipate that any problem found in the Beta would be in the final product too... and then blah, blah, blah blah blah blah...

Grazie, that is a good question. The best scenario would be a net reduction in staff due to fewer followup releases to fix critical bugs that weren't detected in the initial release. But to make this a reality, I would expect significant upfront investment in these two areas:

1. Detailed communications to the beta testers listing all areas that have changed and need to be tested, and solicting input on usability of new features that are still under development.

2. A dedicated web site where testers can see and discuss all issues that have been identified. Ideally, this should also tie back to the change list items in (1) above.

A rough and semi intelligent estimate is the above two items would require 2-4 person-months to accomplish. After this is done, I don't think much if any additional time would be required beyond what the current testing takes. And as I said, if the result is fewer serious bugs make it to released products, the net effort should be lower for SCS. And they don't have to listen to my whining here on the forurm either ;-).

/jerry

Photodex (Proshow Slideshow software), while considerably smaller than Sony has a somewhat middle of the road testing program that seems to work quite well. It is fairly easy to be accepted in their Beta testing program, they have a forum only open to the beta testers and they seem to do a pretty good job of knocking down the problems before the full public release. Their support also seems to be able to get to the developers fairly quickly with issues. No software is perfect (and you couldn't begin to afford it if they had to put the level of testing into it that goes into software for flight control boxes on airplanes ( look up DO-178b for what is involved in that area), however, being able to get information between the developers and the user community (with a reasonable sample of users ) is critical to having good software. Too many companies put the filters on the front end that are so "good" that nothing useful goes back and forth. I am somewhat suspicious of the response I finally managed to get from SCS a while back on the help file issue where they finally said the developers were aware of the problem and to check back in a couple of weeks. I suspect they were talking about the new release of Movie Studio, NOT vegas Pro (although I had made it very clear we were talking about Pro).

mikey

Smaller than Sony?

or smaller than Sony Creative Software? Remember the group that makes Vegas is a relatively small software company in Madison, WI that was bought by Sony. My sense is they tend to focus on one release at time. The Studio Products being the latest release.

rather then a public beta (which most consider a final release anyway), a more open beta program would help more. i've seen public betas be everything from incredibly useful to a complete waste, all because of who was listened to.

IE in Vegas, who should the listen to? Someone who spends thousands a year on computer equipment but always has issues? Someone who claims their system is flawless? Someone who claims they know what they're doing? someone who claims they're a "professional"? Someone who's bought an upgrade every year? Someone who does this on the side?

IMHO a public beta would be more of a waste of time. What SCS does NOT need is a bunch of people complaining it doesn't work when they won't post any system info, can't diagnose simple computer problems, etc. There's enough of that already with the release program.

Well, I don't know the exact head count, but it is not unusual for the owner of the company to show up in the Beta forums with comments and observations and there are only 2 or 3 people that are the active interface with the developers so they are not very big. :-) They actually do a pretty good job of working issues most of the time.

mikey

SVCUs and SVCEs users could be reliable beta testers.

Adobe has, from time to time, made real efforts to reach out to customers for input. We've had teams come to visit us a few times to watch us work, show us their ideas, and ask for input. This program might have started at Macromedia since they were the first ones who approached us but the program remained the same after the acquisition was complete.

SCS might have a similar program but I've never heard rumor of it.

Public betas can serve a variety of purposes. GPL'd software makes real use of public betas and if you want to follow a good example of this start following Blender.

For commercial ventures the public beta is more often a PR move. For instance, Microsoft has gotten a lot of PR mileage out of the Windows 7 beta program. They also got a better product but I think the main value is that they can say that LOTS of customers looked at the product and weighed in to make it better. This is credible because the product IS better.

SCS could have a public beta program. Sure. They could also have a more private beta program where they release a little PR information like the number and demographics of the beta program. If I knew there were a couple of hundred customers in the closed beta program I'd be a little more confident. More so if I knew that some were high end profesionals, some were film school level, and some were 80 year olds who'd never used a computer.

I'd also be pleased to see the results of regular surveys of users. And I'd be pleased to see more white papers (and to see them promoted more).

Rob Mack

I think in part it would work, but at the same time, I'm not sure how many people would be willing to install a Vegas beta. The way it looks to me, the first release and one or two updates of Vegas work like betas. Vegas 9 wasn't acceptable to work with in AVCHD until 9.0c.

If there was a way to run a Vegas beta in a sandbox environment I'd be more than happy to, but I can see how running a Vegas 10 beta alongside 9.0e would start bringing problems for 9.0e as well, so I would stay away from it, unless I read in the forums that it's very stable and has good performance, particularly with AVCHD.

SCS could also set it up so Vegas requires 100% constant internet access so it can constantly stay connected with a Sony server to help diagnose issues & issue fixes w/o the user even downloading them, all 100% automatic.

But I've been part of that type of system too. No matter how many patches there are things are always broken. I got a patch for a program last week that broke part of it. Worked 100% fine but they HAD to add mac support somewhere in the app & that broke things on the windows end...

VideoReDo, an mpeg editor app I use for my TiVo files, uses public betas (and private betas from time to time). The developers are very active on their forum too. SCS could benefit from trying to emulate them.

http://www.videoredo.net/msgBoard/index.php

I'm sure there are many other examples in addition to the ones listed here that can be used for reference.

Actually, the higher you are up in this profession, the less likely you are to be an svcu or svce, since you wouldn't have a need for the certification. It means you're above a newbie and a novice, but those letters don't make you a whiz or a wizard.

I was beta tester for SmartSound's SonicFire Pro a year or two back.

The experience was nothing short of excellent and would do it again in a heartbeat. Before being chosen, they made it clear they were looking for certain systems specs and you had to provide that.

The different ways people use the product as well as the impossible task of setting up a test environment duplicating the variety of disparate systems out there, would, IMHO, make a public beta very cost effective.

The incentive was a free upgrade to the version being tested.

I would most certainly sign up with SCS should they hold one.

Tom

To be blunt it couldn't be made any worse so I fail to see why SCS shouldn't give it a shot.

In general public betas are good from a usability and functionality perspective but not so good for bug finding. One problem is the typical user develops good risk aversion.
How many of us deliberately try to break the program, do you ever deliberately drive through the bad part of town to test that you'll get mugged?
A simple real world example from my own experience. When I first started using After Effects I had many issues with it crashing. I haven't had it crash on me in ages. I'd now be one of those people like many here saying "I've not had any problems, it must be your PC.....". Whatever problems I'd exposed in AE sure haven't gone away but my brain has learned how to use it the way everyone else does.

On the other hand the qualified software test engineers I've worked with know nothing about how to use the code and that is by design. Think of them as a large building full of monkeys bashing away at keyboards. They get a banana everytime they crash the code.

I know two Vegas beta testers fairly well. Last one I met didn't know at first I was even a Vegas user. I played dumb and started asking him about it. Slowly I introduced questions about issues I and others had found. Slowly it's revealed that he's found everyone of them too, theyve been there since V3 at least and they have stratergies to avoid the problems e.g. "Yeah, Vegas's capture is broken, we never use it, you have to use.......".

Bob.

I'm for a public beta cycle as well, if nothing else, it allows SCS to expose their software to the Wide verity of hardware and software configurations we have.

Consider what would have happened, and wouldn't have happened for that matter, had SCS released Vegas Pro 9.0 at the time they did, as a beta, and let us have our fun for a month or so. One thing would be that they'd have a lot of the problems isolated, as they did anyway, however there would have been far fewer angry posts from people who had bought 9.0, installed and tried to render their next project, only to run into a massive set of headaches.

People who test a beta tend to be grateful that they can test the next release, and are rarely pissed off when it crashes, some even find it to be fun to try and BREAK a beta, to be the first to report a new bug :)

... do you ever deliberately drive through the bad part of town to test that you'll get mugged?

My GPS does this for me.

All software trains its users to avoid getting bit. Eventually your beta testers aren't very effective because they know their way around the software's warts.

I think the short answer here is, yes, Vegas could be better if it had a better beta program, or a better managed beta program. This might take money (which is what Grazie was driving at) but it might free up a few people to work on other things.

The question is how do you design a good beta testing program? Figure that this runs in parallel with your internal QA program.

Rob

"The question is how do you design a good beta testing program?"

Based on my experience working around mission critical engineering a functional specification is step one.
I read Vegas release notes that use words like "improved" and the first question that would spring into my mind as tester is "improved from what to what?". How can I test this claim, what are the metrics.

One of the biggest issues I see with the paltry "functional spec" for Vegas is the "unlimited" aspect. That's a very rubbery concept to say the least. Is it any wonder even SCS could not get one project of well over 140 tracks to render for a user. The comment back from them was "We never expected anyone to try such a thing with Vegas". Well if you don't define what your product can do what do you expect will happen?

Bob.

>The question is how do you design a good beta testing program? Figure that this runs in parallel with your internal QA program.

Rob, I agree there needs to be some serious thought put into the design if a beta test program is to be useful. I suspect SCS already does this with their private betas, though I have no experience with their program.

I have beta tested a few commercial programs and a larger number of corporate applications. I think the most effective tests gave the testers a list of specific use cases that they were to execute and report against. There were different sets of use cases and you were assigned the set that mapped to your role as a user. Intuit did that with Quickbooks. The Fortune 250 company I worked for did that with some of their more critical in-house applications. At that company many of the standard use cases then were scripted using Mercury's test suite and the automated regression tests were run against each new point release. SCS may do something like this now - I would say the complexity of Vegas would warrant this type of scripted regresson testing. But at a minimum, many of us agree that broader beta testing of the major releases could provide great value to SCS and us.

EDIT: Just saw Bob's post. His "functional specs" and my "Use Cases" are essentially the same thing.

/jerry

That is one of the big reasons to have a public beta - the designers have their own ideas of how things will be used and it never occurs to them some of the things the users will come up with. They need that feedback. If you are going to sell to the user community, you need to let them "fiddle" with the product to see how they are going to use it and what problems will show up.

mikey

You're assuming that the designers don't talk to or pay attention to the users. This whole thread is full of conclusions reached without supporting facts.

I wasn't saying SCS doesn't listen to the users since I have no knowledge of what they really do. I was speaking in a more generic sense - having written lots of software and applications myself, one of the first things I did was get the users who were going to work with it to bang on it. Never ceased to amaze me on what they could come up with. When they explained the reasoning, it made perfect sense most of the time, it just had not occurred to me to do that. Any application that you expect to work well for a group of users needs to be exercised by that group - they will do things you never thought of no matter how much you try to anticipate what they want (or how tight you have the specs written).

mikey

It sounds to me that releasing the demo a month before release & having a patch ready on release day is more what people are referring to "beta test" here vs an actual test.

They already do that in a way: you have a month after release to get a discount. so use it free for that month & see if the bugs are squashed @ the end of the month. If not don't upgrade.