how to deal with large loops

Question

i want to put a large string in a TStringGrid where every cell contains 4 chars from the string the StringGrid has 16 columns

nc:=1; nr:=1;     //nc = number of column . nr = number of raw
while fs.Length>0  do   // fs is a large string 
 begin
    if nc>16 then nr:=nr+1; nc:=1; 
 stringgrid.Cells[nc,nr]:=copy(fs,1,4);    
 delete(fs,1,4);
 nc:=nc+1;

PeekMessage(M, Handle, 0, 0, PM_NOREMOVE); // it prevents "not responding"
end;

how do i make it faster :=)

Answer 1

The slowdown comes for the most part from Delete. Delete rewrites the entire string. It is better to save the index where to copy from.

Answer 2

You are never going to get this to scale well to large amounts of data. The problem is that trying to get the string grid control to hold huge amounts of data is asking it to do something it was not designed for. Doing so results in an incredibly inefficient storage of the data.

Instead what you really need is the virtual data paradigm. Instead of having the control store the data that it displays, let the control ask you for data on demand. When it needs to know what to display it asks you. That saves you having to load it up in advance with information, most of which it never uses.

Perhaps the ideal control for your needs would be Mike Lischke's famous virtual tree view. As a simpler demonstration of the power of this paradigm, here's a simple example using TListView.

Some initial declarations:

const
  ColCount = 16;
  CharactersPerCell = 4;
  LoremIpsum = 'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod '+
    'tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, '+
    'quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. '+
    'Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu '+
    'fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in '+
    'culpa qui officia deserunt mollit anim id est laborum. ';

Setting the control's properties, and making a large string:

var
  Data: string;

procedure TForm1.FormCreate(Sender: TObject);
var
  i: Integer;
begin
  while Length(Data)<20*1000*1000 do begin // 20 million characters
    Data := Data + LoremIpsum;
  end;

  ListView1.ViewStyle := vsReport;
  ListView1.OwnerData := True;
  ListView1.OnData := ListViewData;
  ListView1.Items.Count := 1 + (Length(Data)-1) div (ColCount*CharactersPerCell);

  ListView1.Columns.Clear;
  for i := 0 to ColCount-1 do begin
    ListView1.Columns.Add.Caption := IntToStr(i+1);
  end;
end;

Rather than using a global var stuffed full of nonsense, you might load the text from a file.

The code to fetch the data on demand:

procedure TForm1.ListViewData(Sender: TObject; Item: TListItem);
var
  Row: string;
  ColIndex: Integer;
begin
  Row := Copy(Data, 1 + Item.Index*ColCount*CharactersPerCell, ColCount*CharactersPerCell);
  Item.Caption := Copy(Row, 1, CharactersPerCell);
  for ColIndex := 1 to ColCount-1 do begin
    Item.SubItems.Add(Copy(Row, 1 + CharactersPerCell*ColIndex, CharactersPerCell));
  end;
end;

Using a virtual contol gives performance in the display aspect. You will still have an issue with loading the data into memory. If you wish to be able to operate on huge files then you'll need to avoid loading the entire file into memory. Instead load only portions of the file, again on demand.

Answer 3

First of all, I don't know exactly how big this string is. However, there are many other issues in your code which made it not do what you were saying in the first place (It was only putting the last two characters in the first cell).

This is what I believe you're trying to do...

procedure TForm1.BitBtn1Click(Sender: TObject);
var
  i, nc, nr, sp, len: Integer;
  fs: String;
begin
  StringGrid.RowCount:= 2;
  StringGrid.ColCount:= 16;
  for i := 1 to 1000 do
    fs:= fs + 'abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz';
  nc:= 1;
  nr:= 1;
  sp:= 1;
  len:= Length(fs);
  while sp < len  do begin
    if nc >= 16 then begin
      Inc(nr);
      nc:= 1;
      StringGrid.RowCount:= StringGrid.RowCount + 1;
    end;
    StringGrid.Cells[nc,nr]:= Copy(fs, sp, 4);
    Inc(sp, 4);
    Inc(nc);
  end;
end;

A few other notes...

I excluded the PeekMessage line because I don't know where you get M from. But that adds to the performance issues you're experiencing. That's forcing the UI to update and repaint for every single cell you will be placing text in.

The row count should also be pre-calculated and set before the loop starts. Personally my math is not fresh enough to add that to my answer for you.

(Modified from my original answer's code using the information from NGLN's answer which was posted seconds after mine)

Answer 4

As @NGLN has already explained the easiest to simply make it faster, I'll show an alternative that also avoids Delete and automatically adjusts to any length string input.

Here's how I would do it, calculating the number of rows needed based on the length of the input data. Note that I've included some setup code to allocate a string for testing (I've commented that code as such) that clearly wouldn't need to be used in your application. This correctly handles strings that aren't evenly divided in to 64-byte rows to display in the grid as well.

procedure TForm4.FormCreate(Sender: TObject);
var
  NumRows, CurrRow, CurrCol: Integer;
  Len: Integer;
  StrToParse: string;
  i: Integer;
const
  SplitCount = 16 * 4;  // Number of columns * chars per column
const
  Letters = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz';
begin
  // Setup code. Only for demonstration purposes.
  // Grid columns, remove fixed column. Leaves column headers
  StringGrid1.ColCount := 16;
  StringGrid1.FixedCols := 0;
  StrToParse := Letters;
  StringGrid1.ColCount := 16;
  // Allocates 1088 character string for testing
  while StrToParse.Length < 1000 do
    StrToParse := StrToParse + Letters;

  Len := StrToParse.Length;
  NumRows := Len div SplitCount;
  // If it's not evenly divisible, add an extra row for the spillover
  if Len mod SplitCount <> 0 then
    Inc(NumRows);
  {
    Calculate the number of rows we need, allowing
    1 for the fixed header row
  }
  StringGrid1.RowCount := NumRows + 1;
  // Index into string's characters
  i := 1;
  for CurrRow := 1 to NumRows do // Skipping fixed row headers
    for CurrCol := 0 to 15 do
      if i < Len then
      begin
        StringGrid1.Cells[CurrCol, CurrRow] := Copy(StrToParse, i, 4);
        Inc(i, 4);
      end;
end;

Answer 5

if possible i would set the stringgrid to invisible because it is more than 10 times faster:

procedure TForm1.Button1Click(Sender: TObject);
(** SLOW VERSION **)
const ABC = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz';
var s:string;
    i,l,r,c:Integer;
    dt:TDateTime;
begin
  s := '';
  for i := 0 to 10000 do begin
    s := s + ABC;
  end;
  l := s.Length;

  StringGrid1.RowCount := 1;
  StringGrid1.ColCount := 16;

  dt := Now;
  i := 1;
  r := 0;
  c := 0;
  while i < l do begin
    StringGrid1.Cells[c,r] := Copy(s,i,4);
    Inc(i,4);
    c := (c+1) mod 17;
    if c=0 then begin
      StringGrid1.RowCount := StringGrid1.RowCount + 1;
      Inc(r);
    end;
  end;
  ShowMessage(Format('Adding strings took %d msec',[MilliSecondsBetween(dt,Now)]));  // ~ 7000 msec
end;

procedure TForm1.Button2Click(Sender: TObject);
(** FASTER VERSION **)
const ABC = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz';
var s:string;
    i,l,r,c:Integer;
    dt:TDateTime;
begin
  s := '';
  for i := 0 to 10000 do begin
    s := s + ABC;
  end;
  l := s.Length;

  StringGrid1.RowCount := 1;
  StringGrid1.ColCount := 16;

  dt := Now;
  i := 1;
  r := 0;
  c := 0;
  StringGrid1.Visible := false;
  while i < l do begin
    StringGrid1.Cells[c,r] := Copy(s,i,4);
    Inc(i,4);
    c := (c+1) mod 17;
    if c=0 then begin
      StringGrid1.RowCount := StringGrid1.RowCount + 1;
      Inc(r);
    end;
  end;
  StringGrid1.Visible := true;
  ShowMessage(Format('Adding strings took %d msec',[MilliSecondsBetween(dt,Now)])); // ~ 700 msec
end;

if you want your app to response you could add Application.ProcessMessages; in the loop.

  while i < l do begin
    StringGrid1.Cells[c,r] := Copy(s,i,4);
    Inc(i,4);
    c := (c+1) mod 17;
    if c=0 then begin
      Application.ProcessMessages;
      StringGrid1.RowCount := StringGrid1.RowCount + 1;
      Inc(r);
    end;
  end;

there are some things you have to be aware when using Application.ProcessMessages;:

• UI is reacting to user inputs
• UI gets redrawn
• timers get fired
• ...

it depends on your application if it is "safe" to call it. maybe it is enough to set a flag when entering your function like this:

procedure DoSomething;
begin
  if not InDoSomething then begin
    InDoSomething := true;
    while blub do begin
      // ...
      Application.ProcessMessages;
    end;
    InDoSomething := false;
  end;
end;