Getting an IllegalBlockSizeException: Data must not be longer than 512 bytes while using RSA

Question

I am trying to decrypt a readme file given for a CTF. I have the public and private keys

But when I try either with openssl or cyberchef, I get an IllegalBlockSizeException: Data must not be longer than 512 bytes. Can someone explain to me why this is happening or guide me? Thanks!

I tried decrypting it with openssl, cyberchef, you name it. But I keep getting the error.

Yes, it's a ransomware situation. They suggested starting with the readme file which is a markdown (encrypted) file, using the private key. The title is the infinite monkey theorem so I guess I'll be stuck for a long time Lol — Phil31, Jul 14 '23 at 23:14
It would be nice if we could play along. I promise I won't give away the answer. — President James K. Polk, Jul 14 '23 at 23:17
Okay, if you want to have fun https://www.mediafire.com/file/k1xln03x2ypa3nq/workspace.zip/file — Phil31, Jul 14 '23 at 23:35

score 0 · Answer 1 · answered Jul 14 '23 at 20:06

What you have is a 4096 bit RSA private key. RSA works with modular exponentiation, and is therefore restricted to messages that are within that size.

By definition in PKCS#1 you can see that actually the ciphertext should be precisely 4096 / 8 = 512 bytes, as the resulting number is simply prefixed with zero bits to get to this (unsigned) encoding.

This restrict the input value for encryption, and in turn that limits the message size that can be processed. As RSA is very inefficient compared to e.g. symmetric encryption usually a hybrid cryptosystem is used: a random key is encrypted which is then used to encrypt the message.

So you have basically two failure modes for this exception (which doesn't get to the unpadding part yet): either the RSA ciphertext is not presented correctly, or you need to look if decryption of the first 512 bytes doesn't result in a 128, 192 or 256 bit key (which is probably an AES key, although other algorithms could be used).

Just decoding the key is obviously not the end of the puzzle, that would be too easy :)

Thank you, I understand what you mean. I am trying to decrypt 512 bytes but I get Error: Encryption block is invalid. According to my research, they used RSAES-PKCS1-V1_5. I wonder if I am using the same padding algorithm for encryption and decryption. — Phil31, Jul 14 '23 at 21:30
That might be the next problem, yes. But the decryption code is not in the question, so I cannot say (sorry, didn't notice the next error in above comment). Or, of course, the data that you are now looking at isn't the RSA ciphertext, but commonly it consists of the first bytes. — Maarten Bodewes, Jul 14 '23 at 21:56
Alright. If you think that the answer helped you then please accept it; I think it does clearly specify why you got the error at the very least. Probably best to leave the puzzle unsolved for the next person otherwise (glad you solved it, but maybe best to delete the previous comment). — Maarten Bodewes, Jul 15 '23 at 18:23

score 0 · Answer 2 · answered Jul 15 '23 at 19:24

You can see your decoded private key here

- Certificate SEQUENCE (9 elem)
  - tbsCertificate TBSCertificate INTEGER 0
  - signatureAlgorithm AlgorithmIdentifier INTEGER (4096 bit) 522619537995600330659362798382170334294884599411537923477852582607882…
  - signature BIT STRING INTEGER 65537
  - INTEGER (4094 bit) 171146993674742119666924704806111657760300490906975096719130752520413…
  - INTEGER (2048 bit) 263740111251701695242154015117783353622712014389724293829853199685866…
  - INTEGER (2048 bit) 198157017343803180079696350367430661633409074561527518264812238754480…
  - INTEGER (2044 bit) 104671869228935732383972808233723622193978050479526509948799635684108…
  - INTEGER (2044 bit) 132312298075359371962212373042384694952133620745722938176422228174864…
  - INTEGER (2045 bit) 274025760907982715469692757023388764574001657891745879566522826382768…

Getting an IllegalBlockSizeException: Data must not be longer than 512 bytes while using RSA

2 Answers2