We are using a Xena 2Ke and have found that our layoffs to a Sony DVW-5000 are not frame accurate; sometimes they are a frame early, other times a frame late. It usually takes two or three attempts to get it right.
I guess you're doing Insert or assemble edits here.
Been a while since I worked with a DVW 500 but I do recall it took a lot of fiddling with the deck and the cabling to get it all humming along no matter how you're feeding it.
Do you have external sync from a SPG feeding both the deck and the card?
From memory trying to lock from the VCR is tricky, having tapes stripped made it easier.
The other way we do it with our DVW 250 which will not do assemble or insert edits is to dump the whole program in one go, that avoids the problem completely.
We have been laying out to DVW500 for years now, and on each successive version of Vegas we have had exactly the same problem as what you describe - a frame late/early - and we use Decklink cards. We have changed the computers we layoff from multiple times and still have exactly the same problems today. We have to do what you are resorting to - try multiple times. I was hoping that V8Pro was going to fix this, but SCS seemed to have handballed the problem to BMD and put the blinkers on. Farss gave some good tips.
What we have noticed (but can also be inconsistent) is the first time you output from the timeline (using non pre-rendered SD YUV format files on the timeline) the "engage delay" works on "6". The subsequent output of the same project - using the temp pre-rendered YUV files from the previous output now works using "engage delay" of "5". (- its too #*@! Complex!!).
This "engage delay" setting will most likely be different on your system - but the principle may apply.
FYI: - Make sure all, and I mean all, non important processes and applications are killed b4 ouputting.
- use a dedicated Hard Drive that doesn't have the operating system on it and is well and truly spec'd above the requirements for SD. (minimun of gauranteed sustained 35MB per sec i/o)