C# How to make an in memory object (DataTable) accessible from other applications, asp websites

Question

I basically have a page which needs to have a specific set of data available to it, I also need this data accessible from other applications. SQL is to slow for this, so my first thought is to have a Datatable already populated and use linq to select from it.

How can I have 1 executable house a datatable that I can then access from other applications? Also - is a Datatable the best structure for my data? The data is basically 400k rows by 6 columns.

@Shaharyar everything is on the table, I honestly am not 100% sure what i'm looking for i thought about in memory file but that has it's drawbacks — JL1, Jun 12 '16 at 18:34
Actually I am not so sure about how good or bad is accessing an in memory object is. But when you have a scenario like RDBMS is too slow and you want to select some repetitive data very fast, you build a caching layer in between code and DB like Redis etc. — Shaharyar, Jun 12 '16 at 18:37

score 1 · Answer 1 · answered Jun 12 '16 at 18:59

1

System.IO.MemoryMappedFiles is what you're looking for. You can initialize your data table and serialize it using WriteXml(). Then map that file to memory by calling System.IO.MemoryMappedFiles.CreateFromFile() method.

Then you could access the file from multiple processes and do stuff with it. E.g. you can resemble the data table in another process by calling ReadXml() You can find more info about MemoryMappedFiles here

This is doable in .NET 4.0 and on!

answered Jun 12 '16 at 18:59

Bozhidar Stoyneff

3,576
1
18
28

Bozhidar - I thought about this why do I care about WriteXML()? – JL1 Jun 12 '16 at 19:16
Also isn't the kind of redundant? So I create a datatable, then I create an xml file, then I save the xml file serializing my class then I read the xml file back to a datatable... every time? – JL1 Jun 12 '16 at 19:24
@JoshuaLong - I can't imagine how else you can access the data table. Its called `MemoryMappedFiles` - so you need to serialize your data somehow in order to become a file. `WriteXml()` and `ReadXml()` are working out of the box. But you could implement binary serialization to gain more speed... – Bozhidar Stoyneff Jun 12 '16 at 19:37
Bozhidar - I think I going to pass on this. This method would require me loading the entire collection of a million objects everytime i wanted to use it once. Thanks though1 – JL1 Jun 12 '16 at 19:56

Bozhidar Stoyneff · Accepted Answer · 2016-06-12T20:09:35.337

Another (better, I believe) approach is to employ MarshalByRefObject. This way you could pass the object - actually proxies to the object - between AppDomains so you don't need serialization at all.

Check out MSDN documentation on the subject

And this is live example I developed couple of years ago, with different goal in mind though. It's a custom web server, which needed to update it self whenever update is available, without interrupting its jobs. So shadow copying was the goal, but I think it is (more) appropriate to use MarshalByRef objects in your scenario.

    public void Boot()
    {
        if (KernelPartition != null)
        {
            throw new InvalidOperationException("Kernel partition already exists.");
        }

        Log.TraceEvent(TraceEventType.Verbose, 0, "Initiating startup sequence...");
        KernelPartition = AppDomain.CreateDomain("Kernel", null, new AppDomainSetup() { ApplicationName = "krnl", ShadowCopyFiles = "true" });
        KernelPartition.DomainUnload += KernelPartition_DomainUnload;

        string engineClassName = null;
        string engineAssemblyName = null;

        try
        {
            var engineTypeId = ConfigurationManager.AppSettings["engineTypeId"];
            engineClassName = engineTypeId.Split(',')[0].Trim();
            engineAssemblyName = engineTypeId.Substring(engineClassName.Length + 1).Trim();
        }
        catch (Exception)
        {
            Log.TraceEvent(TraceEventType.Verbose, 0, "Configuration errors detected. Attempting defaults...");
            engineClassName = "Contoso.Kernel.Turbine";
            engineAssemblyName = "Contoso.Kernel";
        }

        try
        {
            // FOCUS ON THE NEXT LINE
            KernelTurbine = (ITcsKernel)KernelPartition.CreateInstanceAndUnwrap(engineAssemblyName, engineClassName);
            Log.TraceEvent(TraceEventType.Verbose, 0, "Kernel connection established.");
        }
        catch (Exception ex)
        {
            Log.TraceEvent(TraceEventType.Verbose, 0, "Failed to establish communication channel with the kernel with the following exception:\n{0}", ex.ToString());
        }

        if (KernelTurbine == null)
        {
            InitializeRestart();
        }
        else
        {
            try
            {
                Start();
                Log.TraceEvent(TraceEventType.Verbose, 0, "Startup sequence completed.");
                KernelTurbine.RebootDelegate = Reboot;
                DisposeRestartSecond();
            }
            catch (Exception ex)
            {
                string message = string.Format("The startup sequence failed with the following exception:{0}{1}", Environment.NewLine, ex.ToString());
                Log.TraceEvent(TraceEventType.Error, 0, message);

                InitializeRestart();
            }
        }
    }

As I remember, you create the object in the other AppDomain by CreateInstanceAndUnwrap(). This object needs to inherit MarshalByRefObject. So compose a class which inherits it and internally use DataTable.

I hope you can handle it from here. I really can't remember the details but I'm ready to help further should you need.

Thanks again Bozhidar - this one is really tough to find an example of — JL1, Jun 12 '16 at 19:54
Maybe what i;m trying to do just isn't possible. I want 1 application to hold everything in memory and then query that object that only returns 'x'. The problem is SQL is just too slow — JL1, Jun 12 '16 at 19:59
@JoshuaLong - of course it is possible. This does exactly what you're after :). I believe this is the way - work through it and see what happens. It is not a trivial task but definitely possible! If not already, gain some knowledge on `AppDomains`. This is your starting point. — Bozhidar Stoyneff, Jun 12 '16 at 20:15
Bozhidar - I find a pertty decent example of a TCP interface that is a ton of less code - i'm going to make another posting and i'll link it here — JL1, Jun 12 '16 at 20:25
@JL1 - glad you made your mind. Just know that this is a copy-paste from a program I wrote once - it contains unrelated to marshaling stuff. Just focus on the line I pointed by comment... — Bozhidar Stoyneff, Jun 12 '16 at 20:30
Bozhidar i'm going to make this the answer not so much for the code but for MarshalByRefObject. Here is the follow up question http://stackoverflow.com/questions/37778854/c-sharp-how-to-pass-linq-in-an-interface-marshalbyrefobject — JL1, Jun 12 '16 at 20:39

C# How to make an in memory object (DataTable) accessible from other applications, asp websites

2 Answers2