2

We have a system that generates literally 5k - 10k XML files each day. It's a long story but that system will not change for a while.

Anyway, the system dumps these XML files (3k-20k each) into ONE folder. So imagine how quickly that folder starts getting swamped.

I wrote a program that takes the files and organizes them into a hierarchy of year/month/day format. Then another program goes in and deletes any file older than 90 days.

Here's the problem. Our backup system (again, something that can not be changed) takes HOURS to backup those archived folders because there are close to 1 million files. The backup does a full backup (again, we can not change) and what is happening is that the backup has to open and inspect EACH XML file. So the backup speed is so slow that it actually does not finish before the next night's backup!

So what I've been doing is now taking monthly folders and creating a 7z archive. This works well. 200k files down to one file. However, I have to manually do this.

Also, there is one other catch. We can not archive the current month. So there always needs to be 30 days (x 5k - 10k) of files instantly "searchable".

Any suggestions on how to better handle this?

The following ran through my head:

1) Write a program to take the previous day and dump to SQL. Then add the file to an archive.

2) Some type of "live-archiving" file system that the XML could be moved to.

Thanks.

cbmeeks
  • 243
  • 1
  • 4
  • 11

5 Answers5

2

If they're XML, could they be concatenated before you compress them? They should then compress better (better shared symbol dictionary) You could do almost what you're doing now if you concatenate the day files at the end of each month into one big month XML file, or go even further and concatenate years, or even all-of-history-except-this-one-month. In all cases you then compress them, yes.

It doesn't sound like you're doing too badly, really, other than needing to write some scripts to do things automatically for you.

pjz
  • 10,595
  • 1
  • 32
  • 40
0

do you have logrotate in mind ? It's mostly for log archiving but it could work for you scenario.

About your case, what database do you have ? Can it support the amount of plaintext you mentioned ?

Also why do you have to do the 7zip archive manually ? Why don't you use cron to do it for you ?

Nikolaidis Fotis
  • 2,032
  • 11
  • 13
0

You don't mention your platform...if it's linux, why not script a sorting bit (that you already seem to do) that breaks them into smaller directories, then run a 7z command line archive to compress those directories down to a single file, then back those up?

Alternatively, you could create a "Backup server" that mirrors the dumped directory over IP (something like drdb, or rsync) then you can do the backup off that server, and can free up some resources on your overburdened server.

Really, though, aside from finding ways to crunch down the number of files you're creating, you're kind of limited in options. Every brick wall you name you have the tag added that nothing can be done about it or changed.

If your data for the month must be instantly searchable, maybe you would want to look into a database solution or a way to integrate it with a database. From the sound of it, you're already using the filesystem as an ad-hoc database and actually putting the information into a database should improve your performance (and ability to back it up).

Bart Silverstrim
  • 31,172
  • 9
  • 67
  • 87
0

I know you mentioned that you cannot change your backup system, but with the troubles you're having, it may be worth revisiting the solution to begin with.

There's also rsnapshot available. This uses rsync which will only backup the changed files. In your case, if you run it daily, it will only backup the 5k - 10k worth of files, and you can limit it to the most current month as well.

If you need something searchable, I agree with Bart Silverstrim about throwing the contents into a database, as backups and databases should be treated as two separate systems.

vmfarms
  • 3,117
  • 20
  • 17
0

You can try to use different filesystem (xfs) and tuning it's parameters.

For example: http://everything2.com/title/Filesystem+performance+tweaking+with+XFS+on+Linux

Also you can add some disk drives and/or configure a faster RAID level to improve performance, since it looks like you have a bottleneck in IOPS.

DukeLion
  • 3,259
  • 1
  • 18
  • 19