Python scrapy change csv cloumn name

Question

Scrapy is indeed powerful. But it lakes a basic feature. When exporting to csv there is not way to change column name. By default it outputs the field name defined in Item. The item name should be python variables. However, at times we need to output with human readable cloumn names like person_name to Person Name

Is there any solution/setting exists. I tried with FEED in the settings but it simply takes which field to output.

Current Output:

id,person_name,uniq_code
D32,John Smith,8923
D89,Sleim,2343

Required Output:

ID,Person Name,Person Code
D32,John Smith,8923
D89,Sleim,2343

You can declare the item as dict and then name the keys to whatever you want. — SuperUser, Nov 07 '22 at 15:05
create multiple custom items use the itemclass section of the feed exports to specify which items go to which feed — Alexander, Nov 07 '22 at 23:24

score 1 · Accepted Answer · answered Nov 07 '22 at 16:56

When using items, the field names have to adhere to python variable names syntax, but you can define the fields as well by overwriting the fields list in the constructor of the item class as below i.e. this will allow you to define field names containing characters such as spaces, full stops etc. which are not valid python variable names.

import scrapy

# define the item class
class MyItem(scrapy.Item):
    def __init__(self):
        super().__init__()
        self.fields["ID"] = scrapy.Field()
        self.fields["Person Name"] = scrapy.Field()
        self.fields["Person Code"] = scrapy.Field()

In your spider callback function, you can then populate the item as below and the column names will be as required.

def parse(self, response):
    item = MyItem()
    item['ID'] = 'D32'
    item['Person Name'] = 'John Doe'
    item['Person Code'] = 8923
    yield item

Thanks for the reply. But the above fix is not viable for my case. I dump into mysql database as well as write to csv. — Shahid Karimi, Nov 07 '22 at 18:24
You can use a pipeline to save the items to MySQL. In the pipeline you can change the column names to match what your tables look like — msenior_, Nov 08 '22 at 02:52

Python scrapy change csv cloumn name

1 Answers1