1

I'm developing an application which uses some large binary files - in the range 1GB - 25GB. The application will run primarily on servers and possibly the odd powerful/modern desktop PC. I could (a) split these large files up so they're always less than 4 GB, or (b) just keep together in one single file.

FAT32 file systems only allow file sizes up to 4 GB. If I don't split up the files, they won't be usable on FAT32 systems.

Do I need to bother splitting these files?

This application is always going to be running on reasonably modern hardware. Are there any modern servers out there which are likely to use FAT32? Are there any other cloud file systems which would have significant limits on file sizes? (e.g. AWS Elastic file system is fine, as it allows single files up to 47 TB).

Open Door Logistics
  • 178
  • 2
  • 14
  • I suppose you are on Windows. NTFS can handle files up to 16 TByte –  Jun 04 '20 at 00:12
  • I you use Linux and choose ext4 there is no limit for the file size. Your file can be as big as the partition. –  Jun 04 '20 at 00:17
  • Not just windows. We want to know what problems could arise from very large files when using common modern cloud technologies. So this would include windows servers, linux servers, and cloud file systems like AWS Elastic Beanstalk, Google Cloud Firestore. – Open Door Logistics Jun 04 '20 at 00:21
  • A.F.A.I.K they all let you format the volumes. –  Jun 04 '20 at 00:31

5 Answers5

4

You can keep files as big as you need them but there is one big question. Do you have to move or copy these files?

If not then I do not see the problem. Huge database files, swap files, and virtual machine image files work just fine. If files have to be copied, moved, and uploaded then I would split them.

First copying, moving, upload, download, and backup are usually file-based. There are some tools that can split files into parts and rebuild files from pieces but you would have to look for them. Uploads and downloads can be problematic too as transfers sometimes get interrupted and most tools just do not support resume.

Optional Option
  • 1,521
  • 13
  • 33
0

In my experience, it wouldn't be recommended to use FAT32 if you are dealing with larger than 4GB files. Fat32 stores 32 bit size so the limit is

2^32-1 ~= 4.29e9. 2^32-1 bytes = 4GB - 1 byte

I recommend to use exFat if you prefer fat file system.

refer: The specifications, features, and requirements of the exFAT file system include: File size limit of 16 exbibytes (264−1 bytes, or about 1019 bytes, which is otherwise limited by a maximum volume size of 128 PiB, or 257−1 bytes), raised from 4 GiB (232−1 bytes) in a standard FAT32 file system

So if you don't want to be limited by FAT32, I recommend to use exFat.

Or if you are going to do it by cloud computing, it's good to use dedicated AWS EC2 instance or Lambda for demand usage.

Bojan
  • 637
  • 7
  • 17
0

FAT32 is a file system designed in 1977.

To answer your question if you should support FAT32 take a look at these similar questions:

  • Should my homepage support Internet Explorer 1.0?
  • Why doesn't Windows 10 support DOS programs?
  • Will latest Firefox run on Windows 98?

So the answer is simple: Don't support FAT32. If you detect a FAT32 file system, tell the user to change it.

Let's take a look at a modern file system like ext4 or NTFS.

These support huge files. So no need for splitting.

You could also consider using no file system at all.

You could use a fixed size partition or the whole disk.

It will give you a better performance because your data is not fragmented.

But the size of your "file" is fixed.

zomega
  • 1,538
  • 8
  • 26
0

If your filesystem is on a SAN with replication and/or snapshotting distributed changes to different areas of your file might create a load on the storage.

On any modern filesystem files with sizes >100GB are not uncommon. Even making copy of a file this size across network is not problematic as long as the network is stable and fast.

Worse is actually if you have to many small files in a directory. Different FS handle a huge amount files very differently and there is always the choice of tools to access a directory with that many files.

Anganthier
  • 126
  • 3
0

Storing 4-25GB shouldn't be an issue even on relatively older hardware, file systems like ntfs and ext2 that have been around since the 90s have no ploblem working with files of that size, in fact it's pretty normal for video production companies to store raw video files as big as several 100GB on ext4/ntfs file systems. (if I remember correctly, the theoretical limit for modern file systems proceeds over 1TB)

That being said, if you're not going to be reading from the files very often, you might want to break your files into several pieces to make storing and moving them easier. Copying large files can be a hassle.

The only real use case fat32 currently that you're likely to come across is on usb flash drives.

cursedclock
  • 23
  • 5