16

On a Linux system, is there any way to concatenate a series of files into one exposed file for reading and writing while not actually taking up another N bytes of disk space? I was hoping for something like mounting these files via loopback/devmapper to accomplish this.

I have a problem where there are split binary files that can get quite large. I don't want to double my space requirements with massive disk IO just to temporarily read / write contents from these files by cating them all together into one enormous file.

I found this project here, but it seems to have a very specific use case and also depends on perl

Ryan
  • 273
  • 2
  • 5
  • I guess FUSE was invented to solve this kind of problem. Probably you have to write your own module but that should not be too difficult. – Hauke Laging Mar 14 '13 at 05:09

4 Answers4

17

You can do that by combining loop devices and device mapper, thanks to "everything is a file". But with one limitation: The file size cannot be changed (as you cannot increase block devices by writing to them). So if you want to append data then you have to create a bigger block device and overwrite the appended dummy data.

# for testing, Create 4 files
echo foo | dd of=block0 cbs=512 conv=block
echo bar | dd of=block1 cbs=512 conv=block
echo bat | dd of=block2 cbs=512 conv=block
echo baz | dd of=block3 cbs=512 conv=block
# Create a loop device for each of the 4 files
losetup /dev/loop0 block0
losetup /dev/loop1 block1
losetup /dev/loop2 block2
losetup /dev/loop3 block3
# Create a device map named "test" using those loop devices
(
    echo "0 1 linear /dev/loop0 0"
    echo "1 1 linear /dev/loop1 0"
    echo "2 1 linear /dev/loop2 0"
    echo "3 1 linear /dev/loop3 0"
) | dmsetup create test
$EDITOR /dev/mapper/test # use overwrite mode only

For extending the file you may create a big sparse file and use that as additional device.

Community
  • 1
Hauke Laging
  • 5,285
  • 2
  • 24
  • 40
3

The answer provided by Hauke Laging is good, but one should note the first number in those echo lines doesn't just automatically increment by one. In this example it does, but if your block0, block1 etc were greater than one block in length, those numbers must increment by the size of the previous echo line. A clean example of this provided by kernel.org is below, assuming $1 is your /dev/loop0 and $2 is your /dev/loop1:

#!/bin/sh
# Join 2 devices together
1=/dev/loop0
2=/dev/loop1
size1=`blockdev --getsz $1`
size2=`blockdev --getsz $2`
echo "0 $size1 linear $1 0
$size1 $size2 linear $2 0" | dmsetup create joined

Source: https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/linear.html

Mr. T
  • 131
  • 3
2

This could be done using named pipe in Linux.

Assume you have files named file0, file1, file2, file3, file4, file5

# create a name pipe
$ mkfifo mynewfile
# cat to named file
$ cat file{0..5} > mynewfile &

In C programatically you can do

mkfifo(mynewfile , 0777);
system("cat file{0..5} > mynewfile");

Then use then use mynewfile as us you would read from normal file.

mynewfile is FIFO file

katta
  • 21
  • 1
  • Nice! This will work well for some use cases - as long as the contents of the "file" only need to be read through once at a time. But if any seeking or other direct file access are needed, it won't work. – Royce Williams Mar 11 '23 at 23:53
0

For reading, you can less multiple files and then use the :n and :p options to go through them.

For writing, without accessing the files directly you will not be able to write to them.

You can vim multiple files as well, and it will just go through the order they were called (ie. vim fileA fileB fileC - fileB opened after fileA closed, fileC opened after fileB closed).

BriGuy
  • 221
  • 3
  • 7