0

I'm having trouble incorporating an IP address into a format string in my Python Scrapy project. I was trying to use python-dotenv to store sensitive information, such as server IPs, in a .env file and load it into my project, instead of hardcoding it.

I added python-dotenv to the settings.py file of my Scrapy project, but when I run a function that should use the values stored in os, I get an error saying that it can't detect dotenv. Can someone help me understand why this is happening and how to properly incorporate an IP address in a format string using python-dotenv in a Python Scrapy project?

Alexander
  • 16,091
  • 5
  • 13
  • 29
rom
  • 666
  • 2
  • 9
  • 31

1 Answers1

2

It is difficult to help figure out what you might be doing wrong with the limited information you have provided. However here is a general example of how one might use python-dotenv with a scrapy settings.py file.

First create a .env file and add your IP

.env

IPADDRESS = SUPERSECRETIPADDRESS

Finally in your settings.py file you need to import dotenv and then run dotenv.load_dotenv() and then you can get the ip address from the environment variable.

settings.py

import os
import dotenv

dotenv.load_dotenv()

IP_ADDRESS_SCRAPY_SETTING = os.environ["IPADDRESS"]

print(IP_ADDRESS_SCRAPY_SETTING)

output:

SUPERSECRETIPADDRESS

Note: Make sure that the .env file is in the same directory as the settings.py file or it is in one of it's parent/ancestor directories.

So if your settings.py file is in

  • /home/username/scrapy_project/scrapy_project/settings.py

then that means the .env file can be in one of the following:

  • /home/username/scrapy_project/scrapy_project/.env
  • /home/username/scrapy_project/.env
  • /home/username/.env
  • /home/.env
  • /.env

Otherwise it will not be able to find the file.

Alexander
  • 16,091
  • 5
  • 13
  • 29
  • Ok but how do you use it inside a spider? – rom Feb 14 '23 at 09:51
  • 1
    @rom The exact same way.... If you included the code you are attempting to use it in, or your previous attempts at trying it would be much easier to show you.... – Alexander Feb 14 '23 at 09:54
  • 1
    After discovering that my Python 3.9 installation was rejecting python-dotenv and even claiming it was not a module, I resolved the issue by creating a fresh virtual environment with Python 3.8. Your code then successfully executed. Thank you very much for your assistance! – rom Feb 14 '23 at 22:26