3

We have a command line exe that takes input from a text file and produces an output text file. It is used for complex industrial simulations.

The source code for this exe is long gone. Now it was easy enough to create a .NET wrapper which controls the execution of this exe and links in with an external app via a web service.

Unfortunetely a new requirement is to run optimization over this black box model. Now there are various methods to perform black box optimization but they all require calling the executable thousands (millions?) of times. Its obvious that the creation and parsing of disk based text files is the bottleneck of the simulation process.

Is there anyway I can trick this executable into not writing to a physical disk? If we were on Unix I suppose pipes would do the trick, but our deployment server is Windows Server 03.

It just occurred to me that a ramdrive might solve this problem, but I havent played with one of those since MS-DOS 6. Any commercial products worth looking at? Does anyone have any other ideas for emulating a physical drive through code? We are on .NET 3.5.

Alex
  • 3,099
  • 6
  • 41
  • 56
  • 1
    If you already *know* that disk access is the bottleneck, and you know that that's a given, why would you want to optimize the executable to which you don't have source code to begin with? – Alex Aug 31 '09 at 06:15
  • 2
    I dont want to optimize the executable, I want to optimise the model that is implemented by the executable. Even though the model is unknown you can still use fancy techniques based on gradient descent to try and estimate the parameters and optimal solution spaces. Anyway, this is not the topic of the debate. I'm asking about emulating disk access here. – Alex Aug 31 '09 at 06:32
  • How big are your input/output files? Caching disk controllers might be the answer... – DmitryK Aug 31 '09 at 06:36
  • Files are not big at all, we are talking about 3 or 4 files for a total of less than 50KB. Unfortunetely we do not any access to the servers so a hardware solution is out of the question. – Alex Aug 31 '09 at 06:39
  • 1
    There is a RAMdisk driver for Windows 2000 available from Microsoft (source code included). http://support.microsoft.com/default.aspx?scid=kb;en-us;Q257405 Not sure if it will run under 2003 though. – DmitryK Aug 31 '09 at 06:42

4 Answers4

1

RAMDisk with freeware version

Microsoft version

How-to

Prof. Falken
  • 24,226
  • 19
  • 100
  • 173
0

Pipeing output is possible in windows as well - if the executable generates output only to standard out, you can use that.

If it actually writes to a file: since windows caches filesystem writes, I could imagine that simply writing, then deleting small files is almost as fast as a ramdisk. Have you actually tried running the program at a realistic rate, deleting all output post-run? If you keep an eye on CPU usage and the disk queue you should get an idea if plain old disk caching suffices.

Eamon Nerbonne
  • 47,023
  • 20
  • 101
  • 166
  • The same way it does on unix: try `dir | more` for example. Basically, if your program is coded to write directly to disk, this isn't an option. – Eamon Nerbonne Aug 31 '09 at 12:01
  • Unfortunetely this is the case. If we had the source code we wouldnt be writing to disk at all. – Alex Sep 07 '09 at 18:10
0

If you're running on Vista, there's a commercial Ramdisk product that may suit. The x64 version may be needed if your system is already using most of its memory, to make sure you don't end up doing too much page swapping.

Another option is to spend a bit of cash on a 15000 RPM disk or a SSD (solid state disk), although that'll be slower than a RAM disk.

In the long term though it may be cheaper to reverse engineer the processing tool and rewrite it from scratch to avoid the bottlenecks.

devstuff
  • 8,277
  • 1
  • 27
  • 33
  • We are running within a corporate SOE. Accessing or modifying hardware is not an option. Our app also has to be compatible with VMs for testing purposes. Regarding the reverse engineering - blackbox optimization is part of that process. – Alex Aug 31 '09 at 07:39
0

If you run it in a virtual machine, caching for its virtual disks is the responsibility of the host OS... which means, you can run your Windows server inside a VM on more or less anything, and get an extra layer of caching. Does that work in the environment?

Andrew McGregor
  • 31,730
  • 2
  • 29
  • 28