According to JEP 131, Java 8 should provide a PKCS#11 Crypto provider for 64 bit Windows: https://blogs.oracle.com/mullan/entry/jep_131_pkcs_11_crypto.
With that in mind, I downloaded and built both 32 and 64 bit versions of NSS with NSPR using these instructions: https://developer.mozilla.org/en-US/docs/NSS_Sources_Building_Testing
I downloaded Java 8 for Windows 64 build b118, configured the java.security file and created a nss.cfg file:
Excerpt from java.security file:
security.provider.1=sun.security.provider.Sun
security.provider.2=sun.security.rsa.SunRsaSign
security.provider.3=sun.security.ec.SunEC
security.provider.4=com.sun.net.ssl.internal.ssl.Provider SunPKCS11-NSS
security.provider.5=com.sun.crypto.provider.SunJCE
security.provider.6=sun.security.jgss.SunProvider
security.provider.7=com.sun.security.sasl.Provider
security.provider.8=org.jcp.xml.dsig.internal.dom.XMLDSigRI
security.provider.9=sun.security.smartcardio.SunPCSC
security.provider.10=sun.security.pkcs11.SunPKCS11 /devel/nss.cfg
nss.cfg:
# Use NSS as a FIPS-140 compliant cryptographic token
# SunPKCS11-NSS
name = NSS
#32 bit
nssLibraryDirectory = C:\devel\nss\nss-3.15.3.1\dist\WINNT6.1_DBG.OBJ\lib
#64 bit
#nssLibraryDirectory = C:\devel\nss\nss-3.15.3.1\dist\WINNT6.1_64_DBG.OBJ\lib
#non FIPS
#nssDbMode = noDb
#attributes = compatibility
#FIPS
nssSecmodDirectory = c:\devel\fipsdb
nssModule = fips
I ran the test suite that comes with NSS and it looks like all of the encryption/decryption tests passed (did have some issues with the tests that required hostname/domainname but that has to do with the Windows environment).
So here is the problem. I run my test encryption app on Java 7 32 bit with the 32 bit version of NSS and everything works great. When I attempt to run Java 8 64 bit with 64 bit NSS I get the following error:
java.security.ProviderException: Could not initialize NSS
at sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:212)
at sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:103)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at sun.security.jca.ProviderConfig$2.run(Unknown Source)
at sun.security.jca.ProviderConfig$2.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.jca.ProviderConfig.doLoadProvider(Unknown Source)
at sun.security.jca.ProviderConfig.getProvider(Unknown Source)
at sun.security.jca.ProviderList.getProvider(Unknown Source)
at sun.security.jca.ProviderList.getIndex(Unknown Source)
at sun.security.jca.ProviderList.getProviderConfig(Unknown Source)
at sun.security.jca.ProviderList.getProvider(Unknown Source)
at java.security.Security.getProvider(Unknown Source)
at sun.security.ssl.SunJSSE.<init>(Unknown Source)
at sun.security.ssl.SunJSSE.<init>(Unknown Source)
at com.sun.net.ssl.internal.ssl.Provider.<init>(Unknown Source)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at sun.security.jca.ProviderConfig$2.run(Unknown Source)
at sun.security.jca.ProviderConfig$2.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.jca.ProviderConfig.doLoadProvider(Unknown Source)
at sun.security.jca.ProviderConfig.getProvider(Unknown Source)
at sun.security.jca.ProviderList.getProvider(Unknown Source)
at sun.security.jca.ProviderList$ServiceList.tryGet(Unknown Source)
at sun.security.jca.ProviderList$ServiceList.access$200(Unknown Source)
at sun.security.jca.ProviderList$ServiceList$1.hasNext(Unknown Source)
at javax.crypto.KeyGenerator.nextSpi(KeyGenerator.java:323)
at javax.crypto.KeyGenerator.<init>(KeyGenerator.java:158)
at javax.crypto.KeyGenerator.getInstance(KeyGenerator.java:208)
at STSAESEncryption.generateKeyWithGenerator(STSAESEncryption.java:74)
at Main.main(Main.java:24)
Caused by: java.io.IOException: %1 is not a valid Win32 application.
at sun.security.pkcs11.Secmod.nssLoadLibrary(Native Method)
at sun.security.pkcs11.Secmod.initialize(Secmod.java:210)
at sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:207)
... 36 more
I did post a message to Sean Mullan's blog (linked above) and posted a reply to the question: everything is running 64 bit and I am unable to get it working in non-FIPS mode (same error) but my reply has not shown up on the blog yet (needs approval).
Has anyone else attempted to get NSS working with Java 8 64 bit on Windows 64 bit?
Update 1 based on Alex Pakka comment:
Thank you for the reply. I am using the 64bit NSS Library when I am using the Java 8 64 bit. Been switching back and forth as I test things both 32 and 64 bit.
I attached the code and stepped through but when I try to view the platformPath variable I get "platformPath cannot be resolved to a variable". I am not really that familiar with Eclipse so I am wondering if I am doing something wrong.
I have tried to edit the paths that I am putting in to see what errors I get and when I change the nssLibraryPath to another directory (without the nss library) I get a different error then the win32 one.
I do know that nss works with Java 8 64 bit for Linux (and possibly other platforms) but does has it been verified for Windows 64 bit. I know this is a new feature with Java 8 and Windows 64 bit with Java 7 only supporting Windows 43 bit.
Thanks again for the reply, it has helped and I am still trying to figure this out.