I want to encrypt a file with the private key using OpenSSL with the RSA algorithm:
openssl rsautl -in txt.txt -out txt2.txt -inkey private.pem -encrypt
Now if I do a decrypt operation:
openssl rsautl -in txt2.txt -pubin -inkey public.pem -decrypt
A private key is needed for this operation
I know that I should use the public key to encrypt, and if I use the private key, I get a signature.
However, I want to do that for studying purposes.
You are using keys wrongly. In public-key cryptography, encryption uses a public key:
openssl rsautl -in txt.txt -out txt2.txt -inkey public.pem -pubin -encrypt
And for decryption, the private key related to the public key is used:
openssl rsautl -in txt2.txt -inkey private.pem -decrypt
The private key (without -pubin) can be used for encryption since it actually contains the public exponent. Note that RSA should not normally be used to encrypt data directly, but only to 'encapsulate' (RSA-KEM) or 'wrap' the key(s) used for symmetric encryption.
But you mention you actually want to study signature. Although historically RSA signature was sometimes described as 'encrypting with the private key', that description is misleading and actually implementing that was found to be insecure. Sign and verify are actually different operations separate from encryption and decryption, and rsautl performs only part of them. For example, you can do:
# hash the data and encode the result in ASN.1
openssl rsautl -sign -in hashenc.dat -out sig.dat -inkey private.pem
...
# on the recipient (with signature and purportedly correct data)
openssl rsautl -verify -in sig.dat -out hashenc.dat -inkey public.pem -pubin
# or often more appropriate use a certificate for the public key
openssl rsautl -verify -in sig.dat -out hashenc.dat -inkey cert.pem -certin
# now either decode hashenc.dat and compare the hash
# to a new hash of the data (which should be the same)
# or compare all of hashenc.dat to an encoding of a new hash
Instead it is better to use openssl dgst which performs the entire signature and verification sequence as specified by PKCS1 e.g. rfc8017. For example for RSASSA-PKCS1v1_5 signature with SHA256:
openssl dgst -sha256 -sign private.pem -in data.txt -out sig.dat
# or can be abbreviated
openssl sha256 -sign private.pem -in data.txt -out sig.dat
# hashes the data, encodes the hash, does type 1 padding and modexp d
...
openssl dgst -sha256 -verify public.pem -in data.txt -signature sig.dat
# or abbreviated
openssl sha256 -verify public.pem -in data.txt -signature sig.dat
# does modexp e and type 1 unpadding, and compares the result to a hash of the data
# notice you don't specify which key is public or private
# because this command knows what to expect
# however it does not accept the public key from a certificate,
# you must extract the public key from the cert first
This form (but not rsautl) also supports the newer and technically better, but not as widely used, PSS padding. This is only referenced on the dgst man page, and mostly documented on the pkeyutl man page, which isn't totally obvious.
On other Stacks where this is more on-topic, see e.g.:
https://security.stackexchange.com/questions/93603/understanding-digitial-certifications
https://security.stackexchange.com/questions/87325/if-the-public-key-cant-be-used-for-decrypting
https://security.stackexchange.com/questions/11879/is-encrypting-data-with-a-private-key-dangerous
https://security.stackexchange.com/questions/68822/trying-to-understand-rsa-and-its-terminology
https://crypto.stackexchange.com/questions/2123/rsa-encryption-with-private-key-and-decryption-with-a-public-key
https://crypto.stackexchange.com/questions/15997/is-rsa-encryption-the-same-as-signature-generation
https://crypto.stackexchange.com/questions/15295/why-the-need-to-hash-before-signing-small-data
I see that you "want to do that for studying purposes." Then, for anyone else looking at this answer, it shall too be used for "studying purposes," and not for any production or purportedly secure system.
As @kelalaka points out, in public-key cryptography, the public key is used for encryption, and the private key is used for decryption. However, if you are studying RSA, you also know that "encryption/decryption" and "signature/verification" use the same underlying mathematical formula:
Further, encryption raises m to the power of e, mod(n) and decryption raises c to the power of d, mod(n), where (e, n) are the public key and (d, n) are the private key. Given the mathematical basis, I think a natural question becomes, how can we raise m to the power of d instead, and vice versa?
It turns out that message signing raises hash(m) to the power of d, mod(n), and message verification raises tag to the power of e, mod(n). If you loosely define "encrypt" as "raise m to the power of a number, mod (n)", then Openssl CLI can "encrypt a file with the private key" (actually, sign to create a tag). The rsautl subcommand can be used like so (Note that the input data must be of the same bit-length as the key used for encrypting/signing, hence the 501 bytes of 'A' padding):
# Create the file to have a `tag` created for:
python3 -c "print('Hello World'+501*'A', end='')" > txt.txt
# Create a `tag` called `txt2.txt` for the `txt.txt` file:
openssl pkeyutl -sign -inkey private_key.pem -in tx.txt -out txt2.txt -pkeyopt rsa_padding_mode:none
Now txt2.txt contains the result of raising txt.txt to the power of d, mod(n).
If you loosely define "decrypt" as "raise m to the power of a number, mod (n)", then Openssl CLI can "decrypt a file with the public key" (actually, verify but see the raw results of an intermediate step). The rsautl subcommand can be used like so:
openssl rsautl -verify -inkey public_key.pem -pubin -in txt2.txt -raw -hexdump
Now stdout contains the result of raising txt2.txt to the power of e, mod(n).
