1

Is it possible to use TFDFBNBackup and TFDFBNRestore for creating and restoring backups from/to a remote server from local files?

I know that this can be done with the local service manager command line tool like gbak also allows this, but I do not want to use these tools in my new Firemonkey application (Windows, OSX, Linux). I want to compile the functionality completely into my application and I only will have access to the server on a Firebird connection basis, no file share.

Mark Rotteveel
  • 100,966
  • 191
  • 140
  • 197
MichaSchumann
  • 1,361
  • 1
  • 17
  • 36
  • 1
    Are you aware TFDFBNBackup is nbackup, which is totally different than gbak? In any case the [Embarcadero documentation](http://docwiki.embarcadero.com/Libraries/Tokyo/en/FireDAC.Phys.FB.TFDFBNBackup) suggests this is not possible. Firebird itself does support remote nbackup. – Mark Rotteveel Dec 09 '18 at 14:15
  • Yes, I am aware of it. Usually ser service behind gbak also only writes files local to the server but somehow gbak on the client accomplishes it. I also would go with the simpler FBIBBAckup if there was a way to do it without calling the command line utilities. – MichaSchumann Dec 09 '18 at 14:17
  • Sorry, I don't know Delphi. I know that Firebird itself supports this, I have no clue if there are Delphi components that support this. – Mark Rotteveel Dec 09 '18 at 14:21
  • Thanks. I also thougth about creating a service on the remote machine that regularily checks for presence of a backup file, compresses it and creates a secondary database witjh a blob field containing the backup. Sounds weird but is pehaps doable. – MichaSchumann Dec 09 '18 at 14:23
  • gbak on the client can work by two methods: original method is just issuing standard SQL commands to the server, receiving standard query resultsets on client, and saving them to FBK file. New method introduced in some later subversions of FB 2.5 is allowing gbak service running on server and FBK file streaming into client by Services API - this new feature was announced in FB 2.5.x relnotes. However for nbackup it is different: 1) nbackup does not work on SQL data level, it does not know about SQL, it mirrors pages - blocks of FDB file. It works low-level. 2) there was no news about – Arioch 'The Dec 10 '18 at 08:55
  • introducing nbackup streaming from server to client via services API like it was implemented and announced for gbak. So seems both gbak avenues are closed for nbackup. I also think I recall some talk, that since nbackup blocks all the database file on low level (all pages become read-only until snapshot of them pages is completed) that devs insist on local (fastest) pages cloning, then you can copy the snapshot file, even using SQL External Tables if you wish, but they very strongly against turning database read-only for long time until you stream nbk over networks, typically slower than HDD – Arioch 'The Dec 10 '18 at 08:58
  • @Arioch'The I seemed to recall nbackup could be streamed through the services API, but maybe I was wrong about that. – Mark Rotteveel Dec 10 '18 at 10:32
  • @MarkRotteveel FB 2.5.8 readme.services_extensions.txt #2 Peshkov: "....backup and restore must be run on localhost..." then at #4 the same file discusses streaming via services, but only for gbak. If that was implemented for nbackup too then it would be dangerous ( network go down without explicit conn-n break while FDB file is locked read-only - then what? ) and did not happen in 2.5 branch – Arioch 'The Dec 10 '18 at 11:08
  • @Arioch: How can I copy the backup file via external tables? As far as I understand the external tables have a fixed format. If this was possible, this would be great. – MichaSchumann Dec 10 '18 at 19:18
  • @MichaSchumann make your external table virtually "array of int32" for example. As long as you `fbk` file sizes are 4-bytes padded - that should work. Or "array of int16". If it would turn out that FBK is not aligned even on two bytes boundary - then `CHAR(1) OCTET` would be needed. Granted, SQL was not designed for bit-blasting, so that would be less efficient transfer than normal protocols like FTP or SMB. OTOH if transfer speed is not priority and you can do it background after backups created then why not. – Arioch 'The Dec 11 '18 at 08:51
  • Re: nbackup, my understanding is that you only can start copying when you finished backup into specific file node. As nbackup is designed for continuous and nested-layers (tree-like) operation. – Arioch 'The Dec 11 '18 at 08:53
  • I was only thinking about an over-engineered solution with GTT table and blob. First step you do on-server: reading ET into GTT BLOB and dropping ET. Then you read the BLOB to client and close/commit freeing GTT's content (BLOB). However frankly I can not come with any efficient ETT-rows to BLOB-column folding solution in PSQL. Another ov-en solution might be: 1) make ET1 of char-octet column, `select count(1) from ET1` perhaps would be file length. Then you can read the tail of the file using ET1. Then you create ET2 with 16KB rows and read beginning up to tail with big chunks. – Arioch 'The Dec 11 '18 at 08:56
  • @Arioch I will give that a try, very good idea and staring point, thanks a lot! – MichaSchumann Dec 11 '18 at 18:25

1 Answers1

2

Thanks to Arioch's suggestion I could solve it and it works well. I used gbak service as it compresses the backup file. Should work with the nbackup flavour as well.

Below please find some example code without any error handling as proof of concept. As Backup only makes sense if it is absolutely reliable a sophisticated error detection and handling is neccessary when implementing this concept for production purposes.

Also, one has to modify firebird.conf on the server to allow external file access in the folder where the database(s) reside. I created backups of some databases in Windows and a binary compare of the files transferred to the local machine.

In the example I feed a label and a progress bar. The backup component should be set to verbose to display the progress although this slows down the backup on the server I prefer being able to give feedback to the user.

procedure TForm1.Button1Click(Sender: TObject);
var
  count: int64;
  fs: TFileStream;
  x: integer;

  procedure dropBackupTable;
  begin
    with FDQuery do
    begin
      sql.text := 'execute block as ' + 'begin ' +
        'if (exists(select 1 from rdb$relations where rdb$relation_name=''BACKUP'')) then ' +
        'execute statement ''drop table backup'';' + 'end';
      execute;
    end;
  end;

begin

  lbl.text := 'Online backup on server...';
  dropBackupTable;
  pb.Value := 2;
  pb.Max := 2000;
  with FDIbBackup do
  begin
    host := '192.168.2.14';
    database := 'r:\databases\office.fdb';
    port := 1216;
    UserName := 'SYSDBA';
    Password := '???????';
    BackupFiles.Clear;
    BackupFiles.add('r:\databases\back.fbk');
    Backup;
  end;

  lbl.text := 'Copying backup file...';

  with FDQuery do
  begin
    sql.text := 'create table backup external ''r:\databases\back.fbk'' (x integer)';
    execute;
    sql.text := 'select count(*) from backup';
    open;
    count := fields[0].AsInteger;
    close;
    pb.Max := count div 1024;
    pb.Value := 0;
    sql.text := 'select * from backup';
    open;
    fs := TFileStream.create('d:\temp\local.fbk', fmCreate, (fmShareDenyRead or fmShareDenyNone));
    count := 0;
    while not eof do
    begin
      inc(count);
      x := fields[0].AsInteger;
      fs.write(x, sizeOf(x));
      if count > 1023 then
      begin
        pb.Value := pb.Value + 1;
        application.processmessages;
        count := 0;
      end;
      next;
    end;
    close;
    fs.free;
    pb.Value := 0;
  end;

  dropBackupTable;

  lbl.text := 'Ready.';
end;

procedure TForm1.FBBackProgress(ASender: TFDPhysDriverService; const AMessage: string);
begin
  if pb.Value = pb.Max then
    pb.Value := 2
  else
    pb.Value := pb.Value + 1;
  application.processmessages;
end;
MichaSchumann
  • 1,361
  • 1
  • 17
  • 36
  • Now, this *is* over-engineering :-) gbak (apart of nBackup) can pass you backup data over the network natively via services API. See the Firebird's file I mentioned to Mark. No need for ET workarounds with gbak - you can just use backup service in store data remotely fashion as it is – Arioch 'The Dec 12 '18 at 07:56
  • Also, "As Backup only makes sense if it is absolutely reliable" - there is a specifics for IB/FB thing "unrestorable backup", because flexibility of them allows to change data structures and constraints on the go, potentially producing logically prohibited states. So it is a good practice exactly for gbak to immediately restore the data (though it is along process) - just to check the FBK is usable in practice if need be. Or even to replace the working database file with the newly created one. – Arioch 'The Dec 12 '18 at 07:58
  • ` The backup component should be set to verbose to display the progress although this slows down the backup on the server` are you sure? I believe it slows things down on the client due to `TMemo` (actually, underlying WinGDI object) low performance in many small additions workload. If you cache log into `TList` and only dump it to memo once every 1/3 seconds and after work over - then it should be still fast I think – Arioch 'The Dec 12 '18 at 08:01
  • Thankls for the comments. The Delphi component seems not to support the remote thing. But I will do some additional research. – MichaSchumann Dec 12 '18 at 09:53
  • For reliability: I package my app with a specific firebird version so I am sure that the backups created with gbak are compatible. Usually I indeed had the backup make a restore to a database copy and so validate the backup file. – MichaSchumann Dec 12 '18 at 09:54
  • It is not about versions compatibility, absolutely not. It is about database transformation sequences. I repeat, flexibility of Firebird provides for creating databases whose structure and data are mutually incompatible. Such a database can be dumped into FBK. But creation of a new database form this FBK would be impossible: structure would be created first and then data would be denied. It is a common point in IB/FB family. – Arioch 'The Dec 12 '18 at 13:38
  • I see. By now my Firebird databases were always recreatable from a backup. Can you give some examples on what could cause this incompatibility? – MichaSchumann Dec 12 '18 at 13:47
  • The easiest example - no more applicable, just to give you a gist of it - you create the data with some columns having `NULL` then you restructure the table and add `NOT NULL` condition over the column. Today Firebird has a special handling of the case, but previously it did not. – Arioch 'The Dec 12 '18 at 13:49
  • In Interbase 5 I personally had was hit by the following thing: I created the table, added an UDF and modified the table adding `computed by` column using the UDF. When I came to my customer with weekly report I could not recreate the database from GBK: IB first created tables, and only then UDFs, which meant it just could not create that table and halted. It was changed in IB6 because of it. See - the causes might be very different, so it is about being on safe side vs "never can happen to me". Also https://sourceforge.net/p/firebird/mailman/message/17826947/ – Arioch 'The Dec 12 '18 at 13:51
  • Variation on the first theme: I create a table and fill it with some data. Then i do `alter table t add /*column*/ Z integer not null` - and what should firebird do? Some servers would block the database, create new table and pump all the data from old one to new one. FB can have up to 255 formats for every table so it does not lock and copy. But when you would try to insert old records from the FBK to the table - it would halt. Today FB has a special case again and would not allow you to add a not-null column without default value. But that was not always so. – Arioch 'The Dec 12 '18 at 14:00
  • P.S. on Firebird API level streaming FBK over net is done AFAIR by specifying a special "magic constant" for the output file name. Maybe that is how it can be done in AnyDAC too, dunno, did not work with the library – Arioch 'The Dec 12 '18 at 14:04
  • I read the docs it states that supplying STDOUT as filename pumps tha data to the client and ths would collide with verbose. But GBAK somehow can do verbose and remote to local. I will have a look into the GBAK code. Firedac only has an event for string messages, I think I have to access the API natively. – MichaSchumann Dec 12 '18 at 14:07
  • IMHO it makes sense to block adding a not null column to a table with existing records. It then requires three steps, add the column, fill it appropriately and then add the not null constraint. Thats how I always did it. – MichaSchumann Dec 12 '18 at 14:09
  • yep, there are two sides to it: 1) collision with verbose happen in one specific case, either backup or restore I don't recall but not both. 2) the collision only happens when using services API ( -se CLI option AFAIR), by using the non-services "old school" remote backup - plain SQL commands sending - it would not happen (but would be generally slower due to more round-trips over the network) – Arioch 'The Dec 12 '18 at 14:10
  • Micha - and as I told it is blocked unless you provide default value. BUT, u miss the point. It is patching the particular use-case, patch this, patch that. Kind of wack-a-mole game. The speed and flexibility of FB provides for the generic conditions for such trap to be created one way or another with every new feature. There actually is only two outcomes - either redesign Firebird pessimistic way (for example - to recreate tables and copy data on any alter-table stmt) or to accept this as a trade-off. The latter case is IB/FB tradition. So you as DB structure developer should be aware of it. – Arioch 'The Dec 12 '18 at 14:13
  • Thanks, as I am working with Oracle a lot, I am very carful with these things. However: Is there anyone who can show me how to mimic gbak's remote backup feature with the service manager? The Firebird burp sources I do not really understand. – MichaSchumann Dec 20 '18 at 19:32
  • by "service manager" u mean CLI utility or DLL Services API ? – Arioch 'The Dec 21 '18 at 08:31
  • I Meant something I could use from multiplatform client to do the backup like GBAK. But anyhow, I understand your point that my solutiuon is overengineering and it also requires additional configuration work to allow external tables. So I went with calling the gbak utility what works fine on Windows, OSX and Linux (working with FMXLinux), displaing a progress bar and a label with the output from gbak. – MichaSchumann Dec 22 '18 at 09:58
  • Well, direct use of fbclient.dll API might be cross-platform enough. Unified Interbase (UIB) might be a thin, close to API library in Delphi, that hopefully has it implemented. IBX2 Lazarus project might be too, dunno – Arioch 'The Dec 24 '18 at 08:37