How to make a dataset similar to CIFAR10 with several images

Question

I try to make a dataset that is similar to CIFAR10. I found this tutorial: How to create dataset similar to cifar-10

I already can make a dataset with 1 image, but when I try to use several images I got this error:

tensorflow.python.framework.errors.InvalidArgumentError: Indices are not valid: not lexicographically sorted or containing repeats.

Can anyone help me to solve this problem?

This is my code:

from PIL import Image
import numpy as np

out =np.empty([20,7501])
    for j in xrange(0, 10):
        im = Image.open('%d_receipt.jpg' % j)
        im = (np.array(im))
        r = im[:,:,0].flatten()
        g = im[:,:,1].flatten()
        b = im[:,:,2].flatten()
        label = [0]
        out[j] = np.array(list(label) + list(r) + list(g) + list(b),np.uint8)  


    for i in xrange(0, 10):
        im = Image.open('%d_news.jpg' % i)
        im = (np.array(im))
        r = im[:,:,0].flatten()
        g = im[:,:,1].flatten()
        b = im[:,:,2].flatten()
        label = [1]
        j = i + 10
        out[j] = np.array(list(label) + list(r) + list(g) + list(b),np.uint8)


out.tofile("data_batch.bin")

score 0 · Answer 1 · answered Jun 03 '16 at 12:41

I do it like this:

import numpy as np
import scipy.io 

mat = scipy.io.loadmat('train_32x32.mat')
data = mat['X']
label = mat['y']

R_data = data[:,:,0,:]
G_data = data[:,:,1,:]
B_data = data[:,:,2,:]

R_data = np.transpose(R_data, (2,0,1))
G_data = np.transpose(G_data, (2,0,1))
B_data = np.transpose(B_data, (2,0,1))

R_data = np.reshape(R_data,(73257,32*32))
G_data = np.reshape(G_data,(73257,32*32))
B_data = np.reshape(B_data,(73257,32*32))

outdata = np.concatenate((label,R_data,G_data,B_data), axis = 1)
step = 10000
for i in range(1,6):
    temp = outdata[i*step:(i+1)*step,:]
    temp.tofile('SVHN_train_data_batch%d.bin' % i)
    print('save data %d' % i)

Then, just put it directly in the train code of Cifar10 tensorflow example.

I might be sounding stupid, but can you tell how you obtained the "train_32x32.mat" file? Are these the extracted features of the dataset we want to train? — shivisuper, Oct 15 '17 at 21:15

score 0 · Answer 2 · answered Jun 14 '16 at 21:14

I too tried to follow the tutorial you posted in the question however I couldn't get it to work so I made my own solution. It can be found on my github here: https://github.com/jdeepee/machine_learning/tree/master

The code is commented so should be easy enough to follow. I should note it iterated through a master directory containing multiple folders which contain the images.

score 0 · Answer 3 · answered Dec 11 '16 at 03:00

The below snippet is what I did to adapt CIFAR-10 to GTSRB. More details here. https://github.com/hashkanna/traffic-signs/blob/master/Traffic_Signs_Recognition_binFiles.ipynb

out = {}

for i in range(5):
    bin_val = (i%5) + 1
    #im = Image.open(X_train[i])
    #im = np.array(im)
    im = X_train[i]
    r = im[:,:,0].flatten()
    g = im[:,:,1].flatten()
    b = im[:,:,2].flatten()
    label = [y_train[i]]
    out[bin_val] = np.array(list(label) + list(r) + list(g) + list(b),np.uint8)

for i in range(5,len(X_train)):
    bin_val = (i%5) + 1
    #im = Image.open(X_train[i])
    #im = np.array(im)
    im = X_train[i]
    r = im[:,:,0].flatten()
    g = im[:,:,1].flatten()
    b = im[:,:,2].flatten()
    label = [y_train[i]]
    new_array = np.array(list(label) + list(r) + list(g) + list(b),np.uint8)
    out[bin_val] = np.append(out[bin_val], new_array, 0)

for bin_val in range(1,6):
    out[bin_val].tofile("/Users/kanna/Downloads/data_batch_%s.bin"%bin_val)

How to make a dataset similar to CIFAR10 with several images

3 Answers3