Interesting. My problem seems less dramatic -- a lot of VM is being used, but I have never had it reach 8GB or had a crash. Also, the VM is released after quitting Toast. I've never experienced the problem before Toast 8, either, using pretty an much identical burning situation (same video files, in the same locations, etc.)
The MPEG-2 files are generally ones that I converted myself from DivX or Xvid, using ffmpeg or mpeg2enc at half-D1 size. Different ones have different framerates (29.97 NTSC, 25 PAL, 23.976 FILM), but they are generally 4:3 352x480 (NTSC/FILM) or 352x576 (PAL) muxed with 224 kbps 48KHz MPEG 1 layer 2 audio.
Specifically (ffmpegX generated this for me, but I now run it from the command line), the most common thing I do is this (takes a 23.976 16:9 source and converts it to a letterboxed 352x480 mpv file with 3:2 pulldown enabled, encodes 48K audio, and multiplexes it)
ffmpeg -i inputfile.avi -y -f yu4mpegpipe -s 352x480 -map 0.0:0.0 -r 23.976 -an - | yuvscaler -v 1 -M BICUBIC -n n -m WIDE2STD -O SIZE_352x480 | mpeg2enc -v 1 -f 8 -F 1 -a 2 -b 2300 -q 5 -I 0 -M 2 -V 230 -S 9999 -d -p -o outputfile.mpv
ffmpeg -i inputfile.avi -y -vn -f mp2 -acodec mp2 -ab 224 -ar 48000 -ac 2 -map 0.1:0.0 outputfile.mpa
mplex -V -S 0 -f 8 -O 0 -b 230 -o outputfile.mpg outputfile.mpv outputfile.mpa
This is probably more than you needed to know, but it does mean that pretty much all of the MPEG2 files I deal with are fairly uniform. It's possible, perhaps, that there's something funny about the headers that mplex (which I suspect more) or mpeg2enc (which I suspect less) creates that trips Toast up in some kind of subtle way.
I also have not tried this with demultiplexed files on Toast 8.0.3, perhaps that would make a difference. If I get a chance soon I'll try it and report back.