Custom IronPython import resolution

Question

I am loading an IronPython script from a database and executing it. This works fine for simple scripts, but imports are a problem. How can I intercept these import calls and then load the appropriate scripts from the database?

EDIT: My main application is written in C# and I'd like to intercept the calls on the C# side without editing the Python scripts.

EDIT: From the research I've done, it looks like creating your own PlatformAdaptationLayer is the way you're supposed to to implement this, but it doesn't work in this case. I've created my own PAL and in my testing, my FileExsists method gets called for every import in the script. But for some reason it never calls any overload of the OpenInputFileStream method. Digging through the IronPython source, once FileExists returns true, it tries to locate the file itself on the path. So this looks like a dead end.

Answer 1

After a great deal of trial and error, I arrived at a solution. I never managed to get the PlatformAdaptationLayer approach to work correctly. It never called back to the PAL when attempting to load the modules.

So what I decided to do was replace the built-in import function by using the SetVariable method as shown below (Engine and Scope are protected members exposing the ScriptEngine and ScriptScope for the parent script):

delegate object ImportDelegate(CodeContext context, string moduleName, PythonDictionary globals, PythonDictionary locals, PythonTuple tuple);

protected void OverrideImport()
{
    ScriptScope scope = IronPython.Hosting.Python.GetBuiltinModule(Engine);
    scope.SetVariable("__import__", new ImportDelegate(DoDatabaseImport));
}

protected object DoDatabaseImport(CodeContext context, string moduleName, PythonDictionary globals, PythonDictionary locals, PythonTuple tuple)
{
    if (ScriptExistsInDb(moduleName))
    {
        string rawScript = GetScriptFromDb(moduleName);
        ScriptSource source = Engine.CreateScriptSourceFromString(rawScript);
        ScriptScope scope = Engine.CreateScope();
        Engine.Execute(rawScript, scope);
        Microsoft.Scripting.Runtime.Scope ret = Microsoft.Scripting.Hosting.Providers.HostingHelpers.GetScope(scope);
        Scope.SetVariable(moduleName, ret);
        return ret;
     }
     else
     {   // fall back on the built-in method
         return IronPython.Modules.Builtin.__import__(context, moduleName);
     }
}

Hope this helps someone!

Answer 2

I was just trying to do the same thing, except I wanted to store my scripts as embedded resources. I'm creating a library that is a mixture of C# and IronPython and wanted to distribute it as a single dll. I wrote a PlatformAdaptationLayer that works, it first looks in the resources for the script that's being loaded, but then falls back to the base implementation which looks in the filesystem. Three parts to this:

Part 1, The custom PlatformAdaptationLayer

namespace ZenCoding.Hosting
{
    internal class ResourceAwarePlatformAdaptationLayer : PlatformAdaptationLayer
    {
        private readonly Dictionary<string, string> _resourceFiles = new Dictionary<string, string>();
        private static readonly char Seperator = Path.DirectorySeparatorChar;
        private const string ResourceScriptsPrefix = "ZenCoding.python.";

        public ResourceAwarePlatformAdaptationLayer()
        {
            CreateResourceFileSystemEntries();
        }

        #region Private methods

        private void CreateResourceFileSystemEntries()
        {
            foreach (string name in Assembly.GetExecutingAssembly().GetManifestResourceNames())
            {
                if (!name.EndsWith(".py"))
                {
                    continue;
                }
                string filename = name.Substring(ResourceScriptsPrefix.Length);
                filename = filename.Substring(0, filename.Length - 3); //Remove .py
                filename = filename.Replace('.', Seperator);
                _resourceFiles.Add(filename + ".py", name);
            }
        }

        private Stream OpenResourceInputStream(string path)
        {
            string resourceName;
            if (_resourceFiles.TryGetValue(RemoveCurrentDir(path), out resourceName))
            {
                return Assembly.GetExecutingAssembly().GetManifestResourceStream(resourceName);
            }
            return null;
        }

        private bool ResourceDirectoryExists(string path)
        {
            return _resourceFiles.Keys.Any(f => f.StartsWith(RemoveCurrentDir(path) + Seperator));
        }

        private bool ResourceFileExists(string path)
        {
            return _resourceFiles.ContainsKey(RemoveCurrentDir(path));
        }


        private static string RemoveCurrentDir(string path)
        {
            return path.Replace(Directory.GetCurrentDirectory() + Seperator, "").Replace("." + Seperator, "");
        }

        #endregion

        #region Overrides from PlatformAdaptationLayer

        public override bool FileExists(string path)
        {
            return ResourceFileExists(path) || base.FileExists(path);
        }

        public override string[] GetFileSystemEntries(string path, string searchPattern, bool includeFiles, bool includeDirectories)
        {
            string fullPath = Path.Combine(path, searchPattern);
            if (ResourceFileExists(fullPath) || ResourceDirectoryExists(fullPath))
            {
                return new[] { fullPath };
            }
            if (!ResourceDirectoryExists(path))
            {
                return base.GetFileSystemEntries(path, searchPattern, includeFiles, includeDirectories);
            }
            return new string[0];
        }

        public override bool DirectoryExists(string path)
        {
            return ResourceDirectoryExists(path) || base.DirectoryExists(path);
        }

        public override Stream OpenInputFileStream(string path)
        {
            return OpenResourceInputStream(path) ?? base.OpenInputFileStream(path);
        }

        public override Stream OpenInputFileStream(string path, FileMode mode, FileAccess access, FileShare share)
        {
            return OpenResourceInputStream(path) ?? base.OpenInputFileStream(path, mode, access, share);
        }

        public override Stream OpenInputFileStream(string path, FileMode mode, FileAccess access, FileShare share, int bufferSize)
        {
            return OpenResourceInputStream(path) ?? base.OpenInputFileStream(path, mode, access, share, bufferSize);
        }

        #endregion
    }
}

You would need to change the constant ResourceScriptsPrefix to whatever your base namespace is where you stored the python scripts.

Part 2, The custom ScriptHost

namespace ZenCoding.Hosting
{
    internal class ResourceAwareScriptHost : ScriptHost
    {
        private readonly PlatformAdaptationLayer _layer = new ResourceAwarePlatformAdaptationLayer();
        public override PlatformAdaptationLayer PlatformAdaptationLayer
        {
            get { return _layer; }
        }
    }
}

Part 3, finally, how to get a Python engine using your custom stuff:

namespace ZenCoding.Hosting
{
    internal static class ResourceAwareScriptEngineSetup
    {
        public static ScriptEngine CreateResourceAwareEngine()
        {
            var setup = Python.CreateRuntimeSetup(null);
            setup.HostType = typeof(ResourceAwareScriptHost);
            var runtime = new ScriptRuntime(setup);
            return runtime.GetEngineByTypeName(typeof(PythonContext).AssemblyQualifiedName);
        }
    }
}

It would be easy to change this to load scripts from some other location, like a database. Just change the OpenResourceStream, ResourceFileExists and ResourceDirectoryExists methods.

Hope this helps.

Answer 3

You can re-direct all I/O to the database using the PlatformAdaptationLayer. To do this you'll need to implement a ScriptHost which provides the PAL. Then when you create the ScriptRuntime you set the HostType to your host type and it'll be used for the runtime. On the PAL you then override OpenInputFileStream and return a stream object which has the content from the database (you could just use a MemoryStream here after reading from the DB).

If you want to still provide access to file I/O you can always fall back to FileStream's for "files" you can't find.

Answer 4

You need to implement import hooks. Here's an SO question with pointers: PEP 302 Example: New Import Hooks