Effectively using Delphi to read unknown-sized blocks from a file

Question

In the past, I have seen this work, but I never really understood how it should be done.
Assume we have a file of known data types, but unknown length, like a dynamic array of TSomething, where

type
  TSomething = class
    Name: String;
    Var1: Integer;
    Var2: boolean;
  end;

The problem, though, is that this object type may be extended in the future, adding more variables (e.g. Var3: String).
Then, files saved with an older version will not contain the newest variables.
The File Read procedure should somehow recognize data in blocks, with an algorithm like:

procedure Read(Path: String)
begin
  // Read Array Size
  //   Read TSomething --> where does this record end? May not contain Var3!
  //   --> how to know that the next data block I read is not a new object?
end;

I have seen this work with BlockRead and BlockWrite, and I assume each object should probably write its size before writing itself in the file, but I would appreciate an example (not necessarily code), to know that I am thinking towards the right direction.

Related readings I have found:
SO - Delphi 2010: How to save a whole record to a file?
Delphi Basics - BlockRead
SO - Reading/writing dynamic arrays of objects to a file - Delphi
SO - How Can I Save a Dynamic Array to a FileStream in Delphi?

Answer 1

In order to make this work, you need to write the element size to the file. Then when you read the file, you read that element length which allows you to read each entire element, even if your program does not know how to understand all of it.

In terms of matching up your record with the on-disk record that's easy enough if your record only contains simple types. In that scenario you can read from the file Min(ElementLength, YourRecordSize) bytes into your record.

But it does not look as though you actually have that scenario. Your record is in fact a class and so not suitable for memory copying. What's more, its first member is a string which is most definitely not a simple type.

Back in the day (say the 1970s), the techniques you described were how files were read. But these days programming has moved on. Saving structured data to files usually means using a more flexible and adaptable serialization format. You should be looking to using JSON, XML, YAML or similar for such tasks.

Answer 2

I'd say you need a method of versioning you file. That way you know what version of the record is contained in the file. Write it at the start of the file and then on reading, read in the version identifier first and then use the corresponding structure to read the rest.

Answer 3

If I understand you correctly your main issue is if TSomething changes. Most important thing is that you need to add version info into your file, this you really cannot avoid.

As for actual storage using Sqlite would most likely solve all your problems, but depending on your situation it might be an overkill.

Except for unexceptional circumstances I wouldn't really worry about extending the class too much.If you add add version number to the beginning of the file you can easily convert the file after the class have changed. All you need to do is implement your solution so that adding conversions would as simple as reasonable.

In order to read/write files I would prefer streams/XML/JSON (depending on situation) instead of blockread/blockwrite as you don't have to implement a hack to store version number.

In theory you could also have unused space for each record so I you could avoid recreating entire file if class changes upto a point (until you have enough unused space). It maybe helpful if TSomething changes often and files are big, but most likely I would not go that route.

Answer 4

This is how I would do it: Include a simple version number in the header. This can be any string, integer or whatever.

Reading and writing the file is very easy (I am using pseudocode):

Procedure Read (MyFile : TFile);
Var
  reader : IMyFileReader;

begin
  versionInfo = MyFile.ReadVersionInfo();
  reader = ReaderFactory.CreateFromVersion(versionInfo);
  reader.Read(MyFile);
end;


Type
  ReaderFactory = Class
  public 
    class function CreateFromVersion(VersionInfo : TVersionInfo) : IMyFileReader;
  end;

function ReaderFactory.CreateFromVersion(VersionInfo : TVersionInfo) : IMyFileReader;
begin
  if VersionInfo = '0.9-Alpha' then
    result := TVersion_0_9_Alpha_Reader.Create()
  else if VersionInfo = '1.0' then
    result := TVersion1_0_Reader.Create()
  else ....
end;

This can easily be maintained and extended forever. You will never have to touch the Read-routine, but only add a new reader and enhance the factory. With a simple registration method and a TDictionary<TVersionInfo,TMyFileReaderClass>, you can even avoid having to modify the factory.