1

I have a typical scenario where there can write requests in parallel, and each file is a few hundred GBs in size.

My test system, a Lustre file system has 4 OSTs (3TB each) and 1 MDS.

What I practically observed is that with striping disabled, Lustre writes all the files onto a single OST unless it is full.

Is it possible to configure Lustre such that when parallel or overlapping write requests arise, the MDS automatically chooses the target that is not currently busy?

I am intrigued as to why Lustre is not doing this by default, or maybe its because I have missed out on something?

U880D
  • 8,601
  • 6
  • 24
  • 40
hrs
  • 487
  • 5
  • 18

2 Answers2

2

If you typically have large files on your filesystem (e.g. > 32MB) you should set the default stripe count for the filesystem appropriately. You can specify the default striping on a per-directory, per-filesystem basis (depending on your permissions and what your usage is), or when an individual file is created.

Use lfs setstripe -c N /path/to/directory to set the default stripe count to N on directory, which will affect all new files in that directory only. If you only have a limited number of OSTs (4 in your case), or all of your files are huge (in the GB range in your case) then you can use -c -1, which means "stripe over all OSTs", rather than an explicit stripe count.

Use lfs setstripe -c N /mnt/lustre (or whatever the mountpoint of your filesystem is) to set the default stripe count for new files in the whole filesystem (unless otherwise specified by a directory-level default layout, or explicitly at file creation).

With Lustre 2.10 and later, you can use composite file layouts so that the stripe count increases as files get larger, like:

lfs setstripe -E64M -c 1 -E4G -c 4 -E-1 -S 4M -c -1

which means that for the first 64MB of a file use only 1 stripe, for the 64MB-4GB part of the file use 4 stripes, and beyond 4GB use a stripe size of 4MB and stripe over all OSTs (which in your case is also 4, but on some filesystems it might be hundreds of OSTs).

LustreOne
  • 339
  • 2
  • 9
1

Yes, if you enable striping, then Lustre will distribute writes to an individual file across as many OSTs as you want (depending on how many stripes you set). Since you have disable striping, then all IO for a file will be limited to a single OST.

If I misunderstood and instead what you are seeing is all IO for all files sent to the same OST, then you have probably inadvertently set the "index" option to a value other than -1. This option will override the default setting which will allow the MDS to balance file creation across all OSTs.

rread
  • 149
  • 3
  • could you tell me more about this "index" option? where can i set it? is it the index we set while preparing a disk as OST with mkfs.lustre ? – hrs Jul 12 '13 at 16:25
  • The `stripe_index` is set with `lfs setstripe -i N /path/to/{dir,file}` option, and indicates which OST index the file will start allocation on (`N=-1` means "allocate on any OST", and is the default). Typically, you should *NOT* specify the `stripe_index` as part of the layout, since this forces the MDS to allocate *all* files on that OST, which will result in imbalanced space usage and reduced performance. The _only_ reason for specifying `-i N` is for testing, such as verifying individual OST functionality/performance, or your application is manually balancing files across all OSTs. – LustreOne Jun 24 '19 at 03:10