I ran into this problem a couple of years ago while trying to print a 539MB uncompressed TIF with QTRGui (on Windows). When I compressed the TIF using LZW (the only compression type supported by QTR) the file size went UP to over 700MB (!) but QTRGui was able to handle it. (Sometimes compression can actually increase the size of a file, and this can be normal with LZW, but I had never seen that kind of increase!)
In some further testing on my macOS virtual machine, QTR was able to handle the original uncompressed file just fine.
I had a brief exchange of information with Roy Harrington on this, and it turns out that there are probably some size or compression limitations in the Windows QTRGui (most likely resulting from sub-optimal internal memory handling) that don’t exist in the macOS QTR. This was just a hunch, since no one (including Roy) has access to the QTRGui code anymore which was originally developed by someone else.
So, if you are on Windows, the best bet is to just proceed using LZW compression on your TIF files. If you get to a point where that doesn’t work, then for now the only option would be to try printing with QTR on the mac. The good news is that I’ve not hit this problem again using LZW compression (on Windows), even with some very large TIF files from my Howtek scanner.