writing arrays to csv

Question

I am looping through 2d array (data) then populate fields using different segments of the arrays that I am looping over. Below is the sample code:

with open('output.csv' , 'wb') as outcsv:
    headers = ['Field1', 'Field2', 'LongField']
    writer = csv.DictWriter(outcsv, headers)
    writer.writeheader()
    for i in range(numArrays):
        row = [{'Field1': data[i][:3],
                'Field2': data[i][3:5],
                'LongField': data[i][5:]}]
        writer.writerows(row)

I am getting the desired output for all fields except the 'LongField' where the number of values are large (hundreds, could be even few thousands) and varying. Here is what my output looks like:

Field1 Field2 LongField
3,4,5  1,2    array([0,0,0,...,0,0,0])
5,3,6  0,9    array([0,0,0,...,0,0,0])

Is this happening because im exceeding some kind of maximum allow values I can write to a cell? Is there a way for me to write all actual values into the cell?

Any help will be greatly appreciated!

because you do not have an ending index maybe it takes the remaining of the array and outputs it like an array. Try puting `len(data[i])` like the following: `[5:len(data[i])]` — deltashade, Jun 23 '16 at 20:00
What is some sample `data`? What do you want the data in the `LongField` column to be? How do you expect to put many values to appear in a single csv column? — martineau, Jun 23 '16 at 20:21
array([[0., 1., 0., ..., 0., 0., 0.,], [0., 1., 0., ..., 0., 0., 0.,], ..., ]]) — user32147, Jun 23 '16 at 22:32
Please read [_How to create a Minimal, Complete, and Verifiable example_](https://stackoverflow.com/help/mcve), provide one and include example `data` _in the question_ please. — martineau, Jun 23 '16 at 23:13

Nate Anderson · Accepted Answer · 2018-11-20T17:43:23.850

I suspect OP wants all those 0's to print, no matter how many there are. No array( nor ) and no ellipsis ...

EDIT: See comment from @John:

For anyone happening upon this in the future, numpy.savetxt() does what the OP wants without having to create the strings by hand.

Guess:

Try ','.join([str(_) for _ in data[i][5:]])

In full:

row = [{'Field1': data[i][:3], 'Field2': data[i][3:5], 'LongField':  ','.join([str(_) for _ in data[i][5:]])}]

Related to this answer.

Help us:

What kind of array are you using? Can you show us more code? Or tell output of: type(data[0])

EDIT

OP clarified what kind of array and how long it is:

It's a numpy.array (or one dimension of the ndarray; same str treatment)

And it's 2,984 long.

Apparently a str version of numpy.array will truncate all but first 3 and last 3 elements when the array has 1000 elements or more

999 elements:

str(numpy.array(range(1000)))
>>[  0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15  16  17
>>  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35
>>  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53
>>  54  55  56  57  58  59  60  61  62  63  64  65  66  67  68  69  70  71
>>  72  73  74  75  76  77  78  79  80  81  82  83  84  85  86  87  88  89
>>  90  91  92  93  94  95  96  97  98  99 100 101 102 103 104 105 106 107
etc...

1000 elements:

str(numpy.array(range(1001)))
>>'[   0    1    2 ...,  998  999 1000]'

So I think my original recommendation should work, forcing it to the string representation you want. I think CSV DictWriter should take care of quoting.

','.join([str(_) for _ in numpy.array(range(2987))])

EDIT

Explaining my answer:

So you are not giving DictWriter a string -- it must infer the string. I assume DictWriter will use Python str function. As the documentation says, the str function will:

Return a string containing a nicely printable representation of an object.

I suppose numpy developers thought that an array with >999 elements would not be very "printable"

So above 1000 they probably have the __str__ function return a "summarized" version, which only shows first 3 elements and last 3 elements.

Instead of letting DictWriter or the str give you a string, you can create exactly the string you want. I believe the string you want is just all the elements of the array separated by a comma, that's what this code does:

','.join([str(_) for _ in [1,2,3]])

','.join() will "concatenate" all the elements on a comma; elements must be strings
the [str(_) for _ in [1,2,3]] will convert the individual elements to strings (required for the join to work)

Sorry for the confusion. Here's the output of type(data[0]): — user32147, Jun 23 '16 at 22:26
here is the output of data: >>>data array([[0., 1., 0., ..., 0., 0., 0.,], [0., 1., 0., ..., 0., 0., 0.,], ..., ]]) — user32147, Jun 23 '16 at 22:33
Thanks, then I think you can join on a comma manually as described above — Nate Anderson, Jun 23 '16 at 23:04
I still don't know why though.. could you help explain please? — user32147, Jun 23 '16 at 23:18
I added an explanation to my answer, if my answer is acceptable can you mark it as the correct answer? (Click the checkbox next to the answer on upper-left) — Nate Anderson, Jun 23 '16 at 23:33
For anyone happening upon this in the future, numpy.savetxt() does what the OP wants without having to create the strings by hand. — John, Nov 20 '18 at 16:51

writing arrays to csv

1 Answers1