Trying to work with a smart card reader from a VMWare Player virtual machine, which is Windows 8.1 AMD64. The card is a US gov't issued PIV card, as described in the respective NIST standard. The host is Windows 7 AMD64.
I'm working with WinSCard API. VERIFY and GET DATA commands work as expected. However, when I perform GENERAL AUTHENTICATE to generate a digital signature, SCardTransmit() returns error code 1, and in the debug output there are messages:
First-chance exception at 0x77675B68 (KernelBase.dll) in PIVTool.exe: 0x00000001: Incorrect function.
First-chance exception at 0x77675B68 (KernelBase.dll) in PIVTool.exe: 0x0000071A: The remote procedure call was canceled, or if a call time-out was specified, the call timed out.
First-chance exception at 0x77675B68 in PIVTool.exe: Microsoft C++ exception: unsigned long at memory location 0x0113E48C.
And in the system log, there are some messages to that effect too:
Smart Card Service, ID 610: Smart Card Reader 'VMware Virtual USB CCID 0' rejected IOCTL TRANSMIT: Incorrect function. If this error persists, your smart card or reader may not be functioning correctly.
Command Header: 00 87 07 9c
The command header matches what I transmit.
WudfUsbccidDrv ID 11: A Request has returned failure. MsgType: 0x80 ICCStatus: 0x0 CmdStatus: 0x1 Error: 0x0 SW1: 0x0 SW2: 0x0
and then
WudfUsbccidDrv ID 1: An operation has failed (0x0, 0x0, 0x0, 0x0). ScT1Transmit: Failed to send request. HResult: The specified request is not a valid operation for the target device.
and also
WudfUsbccidDrv ID 10: Request [ 0 ] (CLS=0x0,INS=0x87,P1=0x7,P2=0x9C,Lc=266,Le=0,.NETServiceMethod=0x0)
again, that's exactly my request.
The very same code works as expected on the host machine. Same card, same physical reader, same command. The card driver might be different.
I've tried an equivalent piece of Java code against the SunPCSC security provider, just to check for subtle protocol failures; it fails on the VM with a similar message:
javax.smartcardio.CardException: sun.security.smartcardio.PCSCException: Unknown error 0x1
Looks like the VMWare's virtualization layer for smart cards doesn't like this particular command. Any ideas, please?