0

I am trying to run the scrapy exporter with a custom delimiter via CLI like this:

scrapy runspider beneficiari_2016.py -o beneficiari_2016.csv -t csv -a CSV_DELIMITER="\n"

The export works perfectly, but the delimiter is still the default comma(",").

Please let me know if you have any idea how it can be fixed. Thank you!

The code:

import scrapy
from scrapy.item import Item, Field
import urllib.parse

class anmdm(Item):
    nume_beneficiar = Field()

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['http://www.anm.ro/sponsorizari/afisare-2016/beneficiari?
    page=1']
    def parse(self, response):
        doctor = anmdm()
        doctors = []
        for item in response.xpath('//tbody/tr'):
            doctor['nume_beneficiar'] = 
        item.xpath('td[5]//text()').extract_first()
            yield doctor
        next_page =  response.xpath("//ul/li[@class='active']/following-
       sibling::li/a/@href").extract_first()
        if next_page is not None:
            next_page = response.urljoin(next_page)
            print(next_page)
            yield response.follow(next_page, self.parse)
Peter Haddad
  • 78,874
  • 25
  • 140
  • 134
Stefan
  • 1
  • 1

1 Answers1

0

CSV_DELIMITER needs to be changed in settings and not like an spider argument -a.

To change settings on the command line use -s:

scrapy runspider beneficiari_2016.py -o beneficiari_2016.csv -t csv -s CSV_DELIMITER="\n"
eLRuLL
  • 18,488
  • 9
  • 73
  • 99
  • Thanks for your feedback. Sadly adding '-s CSV_DELIMITER="\n"' is not working -- the CSV delimiter is still the default comma. FYI, I am using Scrapy 1.4.0 with Python3 on Mac. – Stefan Jul 11 '17 at 13:14
  • This is the right way to override settings actually. Indeed you should be using -s instead of -a. But With Scrapy 1.4.0 and Python3 I have the same problem. It is still comma in the file. – Mehmet Kurtipek Jul 28 '19 at 23:18