Data size varies in every epoch

Question

I've written a program using "Chainer" to train my model, the data is coming from another function where I am using Python's yield. What I have observed is the size of data changes in every epoch.

def load(file_path):
    with codecs.open(path, 'r', 'utf-8') as read_f:
        for l in read_f:
            yield l.strip()

def data():
    for i1 in load(args.train1)
        x1,y1 = list(map(int,i1.strip().split('\t')))
        tr_data1.add((x1,y1))
    tr_data1 = list(tr_data1)

    for i2 in load(args.train2)
        x2,y2 = list(map(int,i2.strip().split('\t')))
        tr_data2.add((x2,y2))
    tr_data2 = list(tr_data2)

def pos_neg(args):

    p1,n1,p2,n2 = list(), list(), list(), list()
    random.shuffle(tr_data1)
    random.shuffle(tr_data2)
    for i, j in it.zip_longest(range(len(tr_data1)), range(len(tr_data2)), fillvalue='-'):
    '''negative set generation'''
        if len(p)==0 or len(p) <= args.batch and len(p)==0 or len(p) <= args.batch:
            p1.append(tr_data1[i])
            n1.append('''negative for train set1''')
            p2.append(tr_data2[j])
            n2.append('''negative for train set2''')
        else:
            yield p1,n1,p2,n2
            p1,n1,p2,n2 = [tr_data1[i]],['''negative for train set1'''],[tr_data2[j]],['''negative for train set2''']

    if len(p1) != 0 or len(p2) != 0:
        yield p1,n1,p2,n2

def train(args, model):
    for p1, n1, p2, n2 in pos_neg(args):
        print("len of pos_train1 neg_train1 pos_train2 neg_train2:", len(p1), len(n1), len(p2), len(n2))


Main Epoch =  0
len of pos_train1 neg_train1 pos_train2 neg_train2: 17460 17460 17460 17460

Main Epoch =  1
len of pos_train1 neg_train1 pos_train2 neg_train2: 17465 17465 17465 17465

Main Epoch =  2
len of pos_train1 neg_train1 pos_train2 neg_train2: 17407 17407 17407 17407

Can someone please explain to me the reason for the variation in the size of input data?

And is there any way I can avoid this?

Welcome to StackOverflow. Please read and follow the posting guidelines in the help documentation, as suggested when you created this account. [Minimal, complete, verifiable example](http://stackoverflow.com/help/mcve) applies here. We cannot effectively help you until you post your MCVE code and accurately describe the problem. We should be able to paste your posted code into a text file and reproduce the problem you described. — Prune, Oct 23 '18 at 20:56

score 2 · Accepted Answer · answered Oct 24 '18 at 14:07

2

You try running a skip function or some ratio for negative generation, parallel to your positive set.

answered Oct 24 '18 at 14:07

disp_name

1,448
2
20
46

Data size varies in every epoch

1 Answers1