The way to get what you're looking for is to perform the file-transfer over the text-mode console.
In other words, you need something from the 1980's: ZModem
.
If you have an SSH client that supports ZModem transfers, and they're out there (just not stock OpenSSH), you can transfer files via the sz
command.
This works because the old Modem protocols were character based and needed ways to reliably transmit binary data over noisy connections that may or may not allow 8-bit characters. If your server has that, or has the option of installing it, you can get your file-transfers.