I am having a similar problem.
I'm using DVDit Pro 6.3 on WinXP Media Center. While trying to burn my roughly 2-hour project (uncompressed DV AVI with still menus - 7.8 GB total) to a dual-layer disc, I repeatedly get the 'Temporary file exceeds the maximum file size currently allowed on your temporary drive. To transfer files larger than 4GB, please choose a hard drive formatted with the NT File System (NTFS). To learn how to convert a hard drive to NTFS, consult your Windows Help file. -3005' error. The error occurs after the menus are built and before transcoding, ejects the disc and stops the burn process. I haven't had this issue with single layer discs.
All of my hard drives are already NTFS formatted and the ones I've tried as temp drives have between 60 and 130GB free space. Attempting to burn a disc image or a Volume returns the same error.
I have defragmented all my drives and uninstalled and reinstalled DVDit! Pro several times, but this error still happens. My progress with this project is at a standstill because of this error.
Please help!