0

I am newbie in Python. I have tried finding the solution everywhere but couldn't get through.

I have made a Scrapy project and because of the project structure, the spiders are by default stored in /spiders directory.

Problem: We generally run the crawlers from the /project directory which contains /spiders inside it. The problem started after including this piece of code:

def implicit():
    from google.cloud import storage

    # If you don't specify credentials when constructing theclient,the
    # client library will look for credentials in the environment.
    storage_client = storage.Client()

# Make an authenticated API request
    buckets = list(storage_client.list_buckets())

import os
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "remote url of the json file"

This code above doesn't catch the remote file if I crawl the spider from the /project directory and throws error

"File json_filename.json was not found."

But when i crawl the spider from /project/spider directory it runs smoothly without any error.

I guess I am missing some fundamentals over here, something to do with crawling location or the environment variables. Thanks people.

Rubén C.
  • 1,098
  • 6
  • 16
Sankalp Nigam
  • 37
  • 1
  • 1
  • 7
  • Where have you inserted the code ? Which file? Are you running scrapy from the command line ? Please clarify some details here, otherwise it is impossible to give you ans answer. – aleroot Apr 11 '18 at 07:40
  • @aleroot inserted this code in a spider which is stored in /spiders directory. Also m running using command line . Scrapy crawl spidername – Sankalp Nigam Apr 11 '18 at 12:33
  • Do you have a json file with the credentials locally saved in `/project/spider`? – Rubén C. May 02 '18 at 11:24
  • @RubénC. yes exactly. – Sankalp Nigam May 11 '18 at 11:24
  • If you are using a relative path in `os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "remote url of the json file"`instead of the absolute one, it would explain why is failing launching from one folder but not from the other. – Rubén C. May 11 '18 at 11:58

0 Answers0