Python script called from ASP.NET runs fine in IIS Express but fails in local IIS

Question

This might ostensibly seem like a duplicate of Python ArcGIS ArcPy RuntimeError: NotInitialized but this is different because

• that uses ArcGIS 10.0 and there were huge differences in later versions such as the 10.3 that I'm using
• that refers to multiple versions of Python installed whereas I only have one
• that refers to uninstalling and reinstalling which I've already done
• that refers to a different operating system (I'm running on Win 2012)

• that refers to an error that occurs consistently whereas I get the error only from IIS

  I have an ASP.NET application that calls a Python script. The code uses a System.Diagnostics.Process object to call the Python.exe and pass it arguments such as the location of the Python script and other arguments. That Process object looks like this in C#

      Process proc = new Process();
  proc.StartInfo.Verb = "runas";     
  proc.StartInfo.FileName = pathToPythonExe;
  proc.StartInfo.Arguments = procArgs;
  proc.StartInfo.RedirectStandardError = true;
  proc.StartInfo.RedirectStandardOutput = true;
  proc.StartInfo.CreateNoWindow = true;
  proc.StartInfo.UseShellExecute = false;
  proc.StartInfo.WindowStyle = ProcessWindowStyle.Hidden;
  proc.Start();
  proc.WaitForExit();            
  errorToConsole = proc.StandardError.ReadToEnd();
  proc.WaitForExit();
  messageToConsole = proc.StandardOutput.ReadToEnd();
  proc.WaitForExit();
  

  The Path and PYTHONPATH environmental variables point to exactly where the Python executable resides.

  When I run the Python script from an ASP.NET C# app in IIS Express in Visual Studio 2015, everything runs fine. When I run the Python script through the command console, everything runs fine. When I run the Python script from IDLE, everything runs fine. When I publish the application to IIS 8.5 and run it, however, an error occurs in the Python script. Also, when I run the app from Visual Studio and use the local IIS instead of IIS Express, the Python script fails again.

So, here is a recap of the conditions where it does work:

1. Run the Python script from an ASP.NET C# app in IIS Express in Visual Studio 2015.
2. Run the Python script through the command console.
3. Run the Python script from IDLE.

Here is a recap of the conditions where it does not work:

1. Run the Python script from an ASP.NET C# app in a local IIS Express in Visual Studio 2015.
2. Run the Python script from an ASP.NET C# app on IIS.

The gist of the error is "RuntimeError: NotInitialized" on the script line, "import acrpy." This is listed under "Error #1" below. The only difference I saw between running it in VS (either local IIS or IIS Express) versus regular IIS was the permissions.

When I run it in Visual Studio, the Visual Studio app has admin privileges. In IIS, the authentication is set with Windows Authentication enabled. Everything else is disabled. Here is the full error message that occurs when the app runs in IIS 8.5 and when run in Visual Studio with the local IIS.

Error #1:

Traceback (most recent call last):
  File "E:\Application Development\PublishServiceDefinition\MapPublisher\MapSdDraftCreator.py", line 1, in <module>
    import arcpy
  File "E:\Program Files (x86)\ArcGIS\Desktop10.3\ArcPy\arcpy\__init__.py", line 21, in <module>
    from arcpy.geoprocessing import gp
  File "E:\Program Files (x86)\ArcGIS\Desktop10.3\ArcPy\arcpy\geoprocessing\__init__.py", line 14, in <module>
    from _base import *
  File "E:\Program Files (x86)\ArcGIS\Desktop10.3\ArcPy\arcpy\geoprocessing\_base.py", line 598, in <module>
    env = GPEnvironments(gp)
  File "E:\Program Files (x86)\ArcGIS\Desktop10.3\ArcPy\arcpy\geoprocessing\_base.py", line 595, in GPEnvironments
    return GPEnvironment(geoprocessor)
  File "E:\Program Files (x86)\ArcGIS\Desktop10.3\ArcPy\arcpy\geoprocessing\_base.py", line 551, in __init__
    self._refresh()
  File "E:\Program Files (x86)\ArcGIS\Desktop10.3\ArcPy\arcpy\geoprocessing\_base.py", line 553, in _refresh
    envset = (set(env for env in self._gp.listEnvironments()))
RuntimeError: NotInitialized

Here is what I've done to troubleshoot so far.

1st attempt at troubleshooting: I made a full uninstall and reinstall of Desktop 10.3 (which includes the Python files and the arcpy files). That didn't help. I got the same error.

2nd attempt at troubleshooting: I installed Desktop 10.3 (and Python.exe) on a different Windows 2012 server and I setup the IIS on that machine (no Visual Studio on that second machine). The same error occurred. Before the install on the second machine, I made sure that any ESRI products had been removed (including from the registry).

3rd attempt at troubleshooting: I added "import arcinfo" as the first line such that the first two lines of the Python script look like:

import arcinfo
import arcpy

That resulted in the following error:

Error #2:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "e:\program files (x86)\arcgis\desktop10.3\arcpy\arcpy\__init__.py", line 21, in <module>
    from arcpy.geoprocessing import gp
  File "e:\program files (x86)\arcgis\desktop10.3\arcpy\arcpy\geoprocessing\__init__.py", line 14, in <module>
    from _base import *
  File "e:\program files (x86)\arcgis\desktop10.3\arcpy\arcpy\geoprocessing\_base.py", line 598, in <module>
    env = GPEnvironments(gp)
  File "e:\program files (x86)\arcgis\desktop10.3\arcpy\arcpy\geoprocessing\_base.py", line 595, in GPEnvironments
    return GPEnvironment(geoprocessor)
  File "e:\program files (x86)\arcgis\desktop10.3\arcpy\arcpy\geoprocessing\_base.py", line 551, in __init__
    self._refresh()
  File "e:\program files (x86)\arcgis\desktop10.3\arcpy\arcpy\geoprocessing\_base.py", line 553, in _refresh
    envset = (set(env for env in self._gp.listEnvironments()))
RuntimeError: NotInitialized

Traceback (most recent call last):
  File "E:\Application Development\PublishServiceDefinition\MapPublisher\MapSdDraftCreator.py", line 1, in <module>
    import arcinfo
  File "E:\Program Files (x86)\ArcGIS\Desktop10.3\ArcPy\arcinfo.py", line 18, in <module>
    gp.setProduct("ArcInfo")
RuntimeError: ERROR 999999: Error executing function.

4th attempt at troubleshooting: Since Windows Authentication uses the local IIS_IUSRS account, I gave full control to that account to • the folder that has the Python.exe file, • the ArcGIS\Desktop10.3 folder • the folder that has the Python script • any folders accessed by the Python script

I removed “import arcinfo” from the Python script. The results were the same as the first time I tried it from IIS.

5th attempt at troubleshooting: Since the error seems to reference environmental variables, I figured that maybe there is something about the local IIS_IUSRS group that doesn’t permit any thread in its process from accessing environmental variables for security reasons. That was just a guess. It seems to me like something Microsoft might do. So, I got the application in IIS to impersonate an account that has local admin rights. When I ran the ASP.NET application again, it seemed to not even try to read the Python script. The C# process that wraps the Python script captures the StandardError and the StandardOutput. Nothing is caught when impersonation is used. Also, the time the debugger stays on the Start() and WaitForExit() methods is negligible, as if the executing thread in the Process object is not even trying to read the Python script. If it were a matter of folder security permissions, then I would have gotten an error with the impersonated account the way I got when IIS_IUSRS didn’t have sufficient permissions. However, I didn’t get any errors. It just appeared that the process running with the impersonated account didn’t even try to do anything.

6th attempt at troubleshooting: Since the Python.exe that has been installed is a 32-bit version of Python, I went to the application pool that the ASP.NET application uses and I made sure that the 32-bit applications setting was enabled in the advanced settings. That didn’t change the results either for IIS impersonation on or off.

The only other odd thing that I saw was that the ASP.NET project properties in Visual Studio shows that it is targeted for .NET Framework 4.5 whereas the application pools are set for version IIS Manager shows that the app pool is set for .NET CLR Version 4.0. When I try to change the .NET Framework version in the IIS Manager, it only offers the two choices of v4.03 and v2.05. I don’t know how important that is.

edit (5/18/2016 4:06 pm): I've done some more testing. I looked at some environment variables under three different conditions. I ran the ASP.NET app in Visual Studio with these three settings:

1. Local IIS , impersonation disabled
2. Local IIS , impersonation enabled , identity = x12345
3. IIS Express

Here are some of the results of those tests.

Local IIS , impersonation disabled:

Environment.UserDomainName = "IIS APPPOOL"
Environment.UserName = .NET v4.5

Local IIS , impersonation enabled , identity = x12345:

Environment.UserDomainName = x12345
Environment.UserName = ABC

IIS Express:

Environment.UserDomainName = x12345
Environment.UserName = ABC

Notice from above how running with impersonation enabled to the same identity that runs with IIS Express results in the identical Environment variables. That tells me that maybe I should be focusing on getting this to work while starting with ASP.NET impersonation enabled since, as I've mentioned before, running with IIS Express is consistently successful. It seems that it would benefit me to try to emulate those known successful conditions as closely as possible. It seems reasonable to me to focus my efforts moving forward on trying to find out why ASP.NET impersonation with the same environment user as IIS Express fails to run the script. Comments and suggestions will be appreciated.

edit (5/19/2016 1:40 pm):

I have tried enabling the, "Allow service to interact with desktop" option in both the "IIS Admin Service" properties, and also in the W3SVC properties. This did not help.

My next attempt will be to create a web service that calls the Python executable and returns the results. I will have the current ASP.NET application call the web service and then get back the results. Unless anyone has comments as to why this would not work, or has comments as how to get the current ASP.NET application to function as desired, then I will proceed with the web service strategy.

edit (5/25/2016)

It appears that I've got this working, finally, after nearly 3 weeks of working on it.

While it is not conclusive exactly what I did to get it working, here follow the two main things that I did.

I created a WCF Service with basicHttpBinding hosted in IIS. The wsHttpBinding did not work for me because when I tried to configure it for Windows authentication and set mode="transport" I received an error stating that ssl would be required.

The more important thing seems to be that I changed the WCF Service's application pool's process identity from ApplicationPoolIdentity to an account with elevated privileges.

Now when I look at the w3wp.exe user name in Task Manager, I see that it is running with that account's name.

My next step is to start sequentially undoing all the previous configurations and testing to see which ones were nonessential. For example, I had added the app pool account in the security access of the folders that contained the Python scripts and executable that were used. Now I think that might have been unnecessary since the w3wp has access to the Python scripts through its new account. I will start to remove the other accounts that I had previously added.

On hindsight, I think that creating the web service might have been technically unnecessary since I might have just been able to change the process identity of the application pool that runs the ASP.NET application. However, there might have been security issues with that since the ASP.NET app is directly accessible by users and the web service is only indirectly accessible even though this application is an intranet app and has no outside facing interface.

Any insights (suggestions and/or constructive criticisms) on what I got to get it to work will be appreciated.

Answer 1

I've done some more research into this and I now understand why my solution worked. This information answers my original question. Here follows the useful and relevant information on this topic that my research has unearthed.

• info #1

When ArcGIS Desktop 10.3 gets installed on a machine, it creates a directory where it stores important information. The default location of that directory depends on the logon account name of the user who initiates the ArcGIS Desktop application installation. For example, if the logon account is MS\AHejlsberg, then the default installation's UNC path will be,
C:\Users\AHejlsberg.MS\AppData\Roaming\ESRI\Desktop10.3\
That directory will contain essential files such as .sde database connections which would by default be located here,
C:\Users\AHejlsberg.MS\AppData\Roaming\ESRI\Desktop10.3\ArcCatalog\

• info #2

When the .NET System.Diagnostics.Process object's Start() method is called, the Process object will run with the same account as the w3wp.exe worker process identified in the application pool in which the ASP.NET application runs. So, if the ASP.NET application is running under the DefaultAppPool, and if that application pool has the identity, ApplicationPoolIdentity, then Windows' Local Security Authority will create a new virtual account with the user SID of the DefaultAppPool and the group SID of the IIS_IUSRS security principal in its access token.
The new .NET Process object will run in the security context of that new virtual account. So, if the new Process object wraps a Python.exe file, that executable will run as if called by the same account identified in the application pool identity.
For another example, if the ASP.NET application is running under some application pool that has the identity set to some specific user such as a domain account, then the .NET Process object spawned from the ASP.NET application will run under the security context of that user. In this case, the Python.exe will run as if called by the specific user set as the application pool's identity.
Even if the ASP.NET application has impersonation enabled such that both the authenticated HTTP context identity and the WindowsIdentity.GetUser() identity returned happen to be the same, the new .NET Process object will still not run under that ASP.NET application identity if the identity of the application pool is different.

note: A programmer might egregiously presume that if ASP.NET impersonation is enabled to run as a certain account and that the identity of the application pool is set to that same account, then the ASP.NET application and the newly spawned .NET Process object will run with exactly the same security context as each other, in actuality, the thread within the ASP.NET application process that attempts to access securable objects will have an impersonation token whereas the thread that accesses securable objects on behalf of the newly spawned .NET Process object will have only a primary access token.

• info #3

If the Python runtime needs to work with the ESRI ArcPy that gets installed with ArcGIS Desktop, then the runtime needs to be able to access the AppData\Roaming\ESRI subfolders which I mentioned in the info #1 section above. That was true in my case when my Python script called the "import ArcPy" command.
An account in Windows has a list of environmental variables associated with it. Any process created from that account will reference that list of environmental variables.
One of those variables will be called "AppData."
The Python runtime will look for the AppData variable in the list of environmental variables associated with the identity of the account that triggered the Python executable.
That AppData variable will need to value the location where ArcGIS Destkop installed its ESRI subfolders. One way to value the AppData variable to the desired UNC path is to run the Process that triggers the Python executable with identity of the same account that was used to install ArcGIS Desktop.
So, for example, if the ArcGIS Desktop was installed using an account called MS\AHejlsberg, then the .NET Process spawned from ASP.NET that will then call the Python executable will need to also run under the identity of the MS\AHejlsberg account. Setting the identity of the application pool that hosts the ASP.NET application pool to that account will accomplish the goal of having the ASP.NET application create the .NET Process object with that desired account.

A person can retrieve the list of environmental variables for an identity running an ASP.NET application like this:

var envVars = Environment.GetEnvironmentVariables();

If ASP.NET impersonation is enabled for MS\AHejlsberg, then the AppData variable will look like this:

// ["APPDATA"] = "C:\\Users\\AHejlsberg.MS\\AppData\\Roaming"

A person can retrieve the list of environmental variables for the identity of a Process spawned from an ASP.NET application like this:

ProcessStartInfo procInfo = proc.StartInfo;

If the application pool hosting that ASP.NET application has its identity set to ApplicationPoolIdentity and the name of the application pool happens to be, ".NET v4.5" then the AppData will look like this:

// Environment = {System.Collections.Specialized.StringDictionary.GenericAdapter}
    // Count = 43        
        // [19] = {[APPDATA, C:\Windows\system32\config\systemprofile\AppData\Roaming]}

On the other hand, if the application pool hosting that ASP.NET application has its identity set to MS\AHejlsberg, then the AppData will look like this:

// Environment = {System.Collections.Specialized.StringDictionary.GenericAdapter}
    // Count = 49     
        // [21] = {[APPDATA, C:\Users\AHejlsberg.MS\AppData\Roaming]}

Now, the Python runtime will be able to locate the subfolders installed by ArcGIS Desktop.

If a programmer is not able to set the identity of an application pool to a specific user account such as a domain account as shown above, then another option is to copy all of the ESRI subfolders to a location that will be visible to the identity of the application pool such as,
C:\Windows\system32\config\systemprofile\AppData\Roaming
as shown above where the name of the application pool was, ".NET 4.5" and the identity was ApplicationPoolIdentity.
I imagine that should work; but I've not tested that. It is possible that other variable settings may also need to be changed.

Notice the second to last line for the trace back of my error:
envset = (set(env for env in self._gp.listEnvironments()))

We can see that the Python runtime was clearly trying to access environmental variables.

When I had the application pool identity set to ApplicationPoolIdentity, the .NET Process object spawned from ASP.NET was running under the identity of the virtual account that Windows created, and so the Python runtime was looking for the AppData in the wrong UNC path. This caused the RunTime not initialized error we see in the last line of the traceback.

When I set the identity of the application pool to the same identity of the same account that installed the ArcGIS Destkop, then the .NET Process object spawned from the ASP.NET application to be able to reference the AppData variable valuing the path that the Python runtime could successfully use for the "Import ArcPy" statement.

That solves the mystery of the Python runtime succeeding with IIS Express but failing with Local IIS.