Sorting a large amount of data into separate sets

Question

I'm extracting up to 2500 frame files per experiment (not always the same amount), and my process at the moment is to manually divide the total number of frames by three to separate into three subset folders since the file size is too large to convert all into a .mat file. I simply want to automate this.

Once the files are separated into three subsets ('Subset1, Subset2, Subset3'), i run each folder through my code to convert and rename.

from scipy.io import savemat
import numpy as np
import os
arrays = []
directory = r"F:\...\Experiment 24\Imaging\Subset3" # **something here that will look at the while directory and create a different file for each subset folder**
sorted(os.listdir(directory))
for filename in sorted(os.listdir(directory)):
    f = os.path.join(directory, filename)
    arrays.append(np.load(f))
data = np.array(arrays)
data = data.astype('uint16')


data = np.moveaxis(data, [0, 1, 2], [2, 1, 0])

savemat('24_subset3.mat', {'data': data})

How can I automatically sort my frame files into three separate subset folders and convert?

Is there a particular logic to splitting files the files between the 3 subsets?For example do you need to keep the files in a particular order? if you had 9 files numbered 1 through 9, do they need to be split `{1, 2 ,3}, {4, 5 ,6} , {7, 8, 9}`, or would it be ok to be `{1, 4 ,7}, {2, 5, 8}, {3, 6, 9}`? — nigh_anxiety, Jan 27 '23 at 01:20
I'm not sure any of that code is actually relevant to your question, since it's all stuff you plan to do later, *after* achieving the end goal of the question. — CrazyChucky, Jan 27 '23 at 02:20
I want to add something that goes through all the subsets without me having to change it manually! :) — Gala Solis, Jan 27 '23 at 03:24
@nigh_anxiety The data is in numerical order [frame0001, frame0002, frame0003...etc] so yes ideally it would be nice to use the former — Gala Solis, Jan 27 '23 at 03:25

Mateen Ulhaq · Answer 1 · 2023-01-27T04:37:05.750

0

Create subsets from the filenames and copy them to new subset directories:

num_subsets = 3
in_dir = "/some/path/to/input"
out_dir = "/some/path/to/output/subsets"

filenames = sorted(os.listdir(in_dir))
chunk_size = len(filenames) // num_subsets

for i in range(num_subsets):
    subset = filenames[i * chunk_size : (i + 1) * chunk_size]

    # Create subset output directory.
    subset_dir = f"{out_dir}/subset_{i}"
    os.makedirs(subset_dir, exist_ok=True)

    for filename in subset:
        shutil.copyfile(filename, f"{subset_dir}/{filename}")

NOTE: Any extra files that cannot be distributed into equal subsets will be skipped.

edited Jan 27 '23 at 04:37

answered Jan 27 '23 at 04:21

Mateen Ulhaq

24,552
19
101
135

This will leave out file names at the end if the number of files isn't evenly divisible by `num_subsets`. – CrazyChucky Jan 27 '23 at 04:29
@CrazyChucky My guess is that that is desired (see title), but I should have noted it in the answer. – Mateen Ulhaq Jan 27 '23 at 04:35

CrazyChucky · Answer 2 · 2023-01-27T19:01:47.687

If your goal is simply to create your three .mat files, you don't necessarily need to create subfolders and move your files around at all; you can iterate through subsets of them in-place. You could manually calculate the indexes at which to divide into subsets, but more_itertools.divide is convenient and readable.

Additionally, pathlib is usually a more convenient way of manipulating paths and filenames. No more worrying about os.path.join! The paths yielded by Path.iterdir or Path.glob know where they're located, and don't need to be recombined with their parent.

import pathlib

from more_itertools import divided
import numpy as np
from scipy.io import savemat


directory = Path("F:/.../Experiment 24/Imaging/")
subsets = divide(3, sorted(directory.iterdir()))

for index, subset in enumerate(subsets, start=1):
    arrays = [np.load(file) for file in subset]
    data = np.array(arrays).astype('uint16')
    data = np.moveaxis(data, [0, 1, 2], [2, 1, 0])
    savemat(f'24_subset{index}.mat', {'data': data})

Sorting a large amount of data into separate sets

2 Answers2