How to choose some specific images from a large dataset of images?

Question

My dataset has 366 folders means folder for each day covering a duration of 1 year nd each folder containing 51-55 images, out of which I need only 36 images for building neural network. so can I assign some index to those images and select some based on their index..? Can someone suggest me code for doing so..?

Can you add [pandas.DataFrame.head](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.head.html) of your data? So we can guide you to true direction. You gave us no info about your data. There is thousands of way extracting and seperating data from data frame. Maybe there is columns that we can use for seperating images that you wanted. — SeLeCtRa, Oct 30 '21 at 11:23
Actually my dataset has 366 folders means folder for each day covering a duration of 1 year nd each folder containing 51-55 images, out of which I need only 36 images for building neural network....i'm attaching some screenshots for refrence — Shivam Yadav, Oct 30 '21 at 12:10
first you can use `os.listdir()` to get all folders as list, next you can use again `os.listdir(folder)` for every folder to get list with filenames. And every item on list has own number/index. OR you can use `glob.glob()` with `*` to get all filenames in all folders as list. And again every element on list has own number/index — furas, Oct 30 '21 at 13:42
do you want 36 images from all images or 36 images for every day? — furas, Oct 30 '21 at 13:47

score 1 · Accepted Answer · answered Oct 30 '21 at 14:05

Every element on list has own number/index - so first you could create lists with all filenames.

You can use os.listdir() to get all folders and later use os.listdir(folder) for every folder to get list with filenames in folder.

import os

base = '/home/furas/images/2021'

folders = os.listdir(base)

all_filenames = []

for folder_name in folders:

    # add base path to folder name to have full path
    full_path = os.path.join(base, folder_name)
    print(full_path)
    
    # get filenames (without path) in folder
    filenames = os.listdir(full_path)
    
    # add path to filenames
    filenames = [os.path.join(full_path, name) for name in filenames]
   
    all_filenames.append(filenames)
    
print(all_filenames)

This way you have 2D list with all filenames (with full path) and you can select them.

first 36 images in some day

selected = all_filenames[day_index][:36]

first 36 images in every days

selected = []

for day in all_filenames:
    selected.append( day[:36] )

random 36 images in every day

import random

selected = []

for day in all_filenames:
    selected.append( random.choices(day, 36) )

can I make a directory which has the images corresponding to these file paths because I have to do image preprocessing of these images — Shivam Yadav, Nov 03 '21 at 04:34
I don't know what path you means but you can use `os.makedirs(some_path, exist_ok=True)` for this. It will create directory if not exists. And if directory exists then it will skip it (without error message). — furas, Nov 03 '21 at 11:12

How to choose some specific images from a large dataset of images?

1 Answers1