Ada - reading large files

Question

I'm working on building an HTTP server, mostly for learning/curiosity purposes, and I came across a problem I've never had in Ada before. If I try to read files that are too big using Direct_IO, I get a Storage Error: Stack Overflow exception. This almost never happens, but when I request a video file, the exception will be thrown.

So I got the idea to read and send files in chunks of 1M characters at a time but this leaves me with End Errors since most files aren't going to be exactly 1M characters in length. I'm also not entirely sure I did it right anyway since reading the whole file has always been sufficient before. Here is the procedure that I've written:

procedure Send_File(Channel : GNAT.Sockets.Stream_Access; Filepath : String) is
    File_Size : Natural := Natural(Ada.Directories.Size (Filepath));
    subtype Meg_String is String(1 .. 1048576);
    package Meg_String_IO is new Ada.Direct_IO(Meg_String);
    Meg : Meg_String;
    File : Meg_String_IO.File_Type;
    Count : Natural := 0;
begin
    loop
        Meg_String_IO.Open(File, Mode => Meg_String_IO.In_File, Name => Filepath);
        Meg_String_IO.Read(File, Item => Meg);
        Meg_String_IO.Close(File);
        String'Write(Channel, Meg);
        exit when Count >= File_Size;
        Count := Count + 1048576;
    end loop;
end Send_File;

I had the thought to declare two separate Direct_IO packages/string sizes, where one would be 1048576 in length while the other would be the file length mod 1048576 in length but I'm not sure how I would use the two readers sequentially.

Thanks to anyone who can help.

Answer 1

I'd use Stream_IO (ARM A.12.1), which allows you to read into a buffer and tells you how much data was actually read; see the second form of Read,

procedure Read (File : in  File_Type;
                Item : out Stream_Element_Array;
                Last : out Stream_Element_Offset);

with semantics described in ARM 13.13.1 (8),

The Read operation transfers stream elements from the specified stream to fill the array Item. Elements are transferred until Item'Length elements have been transferred, or until the end of the stream is reached. If any elements are transferred, the index of the last stream element transferred is returned in Last. Otherwise, Item'First - 1 is returned in Last. Last is less than Item'Last only if the end of the stream is reached.

procedure Send_File (Channel  : GNAT.Sockets.Stream_Access;
                     Filepath : String) is
   File   : Ada.Streams.Stream_IO.File_Type;
   Buffer : Ada.Streams.Stream_Element_Array (1 .. 1024);
   Last   : Ada.Streams.Stream_Element_Offset;
   use type Ada.Streams.Stream_Element_Offset;
begin
   Ada.Streams.Stream_IO.Open (File,
                               Mode => Ada.Streams.Stream_IO.In_File,
                               Name => Filepath);
   loop

Read the next Buffer-full from File. Last receives the index of the last byte that was read; if we’ve reached the end-of-file in this read, Last will be less than Buffer’Last.

      Ada.Streams.Stream_IO.Read (File, Item => Buffer, Last => Last);

Write the data that was actually read. If File’s size is a multiple of Buffer’Length, the last Read will read no bytes and will return a Last of 0 (Buffer’First - 1), so this will write Buffer (1 .. 0), i.e. no bytes.

      Ada.Streams.Write (Channel.all, Buffer (1 .. Last));

The only reason for reading less than a Buffer-full is that end-of-file was reached.

      exit when Last < Buffer’Last;
   end loop;
   Ada.Streams.Stream_IO.Close (File);
end Send_File;

(note also that it’s best to open and close the file outside the loop!)