Are your disks produced using Toast's Blu-ray plugin? With the plugin you can create BDMV folders on standard DVD or DVD-DL disks. Roxio advertises (in their plugin promos) that these disks will play in ANY Blu-ray player. Sorry - NOT SO. If this is your problem, they should play on Samsung, LG, Insignia and Sony BD machines on your TV, not in your Mac. And not on other players - not on Panasonic (my issue - but they play fine on my Samsung). Again, these are not players in your Mac - they are set top players connected to your home TV (as you'd play any DVD or BD)
The Mac will not mount BD disks. The fact that you say you could mount some and get to a menu seems surprising to me. My Mac Pro's Pioneer 112's won't mount BD media. Nor will Apple's DVD Player read those disks. But there are work arounds.
One begins by producing a disk image of your video files in Toast prior to burning you disk. (if you can work backwards in Toast to create disk images from your BD disks that might be part of a solution for disks you have - I've never done it but it seems possible - copy a BDMV folder on non-BD media [DVD, DVD-DL] to an image file of the same disk size [DVD, DVD-DL] - then open the image.
This is important as the image file can be opened and mounted and the BDMV folder dragged to VLC or Boxee (after a fashion - VLC is much better). In this way you have the equivalent of Apple's DVD Player, albeit for a BDMV disk image rather than a disk. You can see what your video files look like. The entire disk image will play on VLC, though Toast's themes (menus) are flakey for me - they pop open and closed) . If the menu's don't work in VLC, then open the BDMV folder, then open the Streaming folder, and drag each stream to VLC's icon individually (in the dock or apps folder, wherever you have it). The smallest sized file in the Streaming folder is the menu info.
The image has it's advantages. It takes the same time as burning a disk but can then be used to burn as many disk copies as you need in much less time. But most important, the Toast image can be double-clicked and mounted, and it's contents viewed and checked.
So far this is the only way to play (or mount) BDMV media on a Mac (since Apple, one of the original Blu-ray supporters, has decided we should buy or rent BD media, not make it (as Apple won't get their 30% that way) - so always create a disk image of your file before burning the disk to use in standard players, especially when using the BD plugin.
Hope this added something useful.