Depends on which app you want do the encoding. Movie DVDs require MPEG-2. It's what you set top player expects, and is the only format that works for a DVD. Movie Blu-ray disks use MPEG-4 (h.264).
What's probably happening is you're encoding the video twice. It's already compressed out of Media Encoder, and then Toast is compressing the compression. On top of that, bit rate is everything. You're either destroying the video in Media Encoder, Toast, or both.
These are the video settings I use in Media Encoder for Blu-ray (first image below).
The lower the bit rate, the choppier the video will be. Too high, and an older deck may not be able to decode it fast enough, causing the video to pause or stutter during playback.
Let's assume you're going to create your MPEG-4 file in Media Encoder. What you need to do in Toast then is tell it NOT to encode the video again. You've already done that. When assembling the final video in Toast (lower image), click on the Customize button, then the Encoding tab, and then click the Custom radio button. Just below the bit rate and motion sliders, change the Reencoding choice to Never. Click OK.
Toast should then just directly copy/store the already compressed video to the BDMV folder on the Blu-ray disk.