The printer doesn't support direct PDF printing which is why sending the PDF file directly to it doesn't work. In the absence of instructions the printer assumes what its being sent is PCL and PCL is specified so that if you send it text, it just prints it. PDF files look enough like text for that to happen.
So you need to create either a PostScript or PCL representation of your PDF file (because that's what your printer supports). My guess is that CUPS is converting it to PostScript for the best quality output, using Ghostscript to do the work. This will mean that all the image data is decompressed, and then recompressed into PostScript, which is probably where the time goes.
You could instead try using the Ghostscript pxlcolor device which produces PXL output, or you could try one of the many various PCL6 devices to produce PCL output and see if that's faster and if your printer likes it.
You could also try using the ps2write device to produce PostScript and try that. Its possible that your setup is using a sufficiently old version of Ghostscript/CUPS that its using the old pswrite device, which produces huge, slow PostScript that takes an age to send to the printer and a long time to print.
In the above cases just use something like :
gs -sDEVICE=ps2write -o out.ps
and then use ipptool to send the resulting out.ps file to the printer. NB you may need to prepend the file with some PJL to switch the language into PostScript, and postfix it with the correct terminator code.
You can do the same with the PXL/PCL output files, but you shouldn't need the PJL wrapped around it there.
Note that if you are writing an application, you need to be careful using Ghostscript, as its covered under the AGPL. But if you can find a way to make that work acceptably you should be able to configure CUPS to do the same, instead of whatever its currently doing.