When we generate a key pair and use a private key from it in the cipher, it works as intended without any exceptions:
KeyPairGenerator kpg = KeyPairGenerator.getInstance("RSA", "BC");
kpg.initialize(1024, new SecureRandom());
KeyPair kp = kpg.generateKeyPair();
PrivateKey privateKey = kp.getPrivate();
byte[] encryptedBytes = "SAMPLE".getBytes();
Cipher cipher = Cipher.getInstance("RSA");
cipher.init(Cipher.DECRYPT_MODE, privateKey);
byte[] decryptedBytes = cipher.doFinal(encryptedBytes);
But after saving of the generated PrivateKey into the AndroidKeyStore and extraction from it:
KeyStore ks = KeyStore.getInstance("AndroidKeyStore");
ks.load(null);
KeyStore.Entry entry = ks.getEntry("alias", null);
PrivateKey privateKey = ((KeyStore.PrivateKeyEntry) entry).getPrivateKey();
the privateKey is not null, but the same code in the first block above (for decryption) throws an exception with message "unknown key type passed to RSA" on the line:
cipher.init(Cipher.DECRYPT_MODE, privateKey);
I know that AndroidKeyStore was not purposed to extract the data about keys saved in it, but I expect it to return a correct Private Key to work with Android's Cipher class correctly at least.