text substitution {} does not work at scrapinghub

Question

I create a url with {} format to change the url on the fly. It works totally fine on my PC. But once I upload and run it from scrapinghub one(state) of the many substitutions(others work fine) does not work, it returns %7B%7D& in the url which is encoded curly braces. Why does this happen? What do I miss when referencing State variable?

This is the url from my code:

def __init__(self):

    self.state = 'AL'
    self.zip = '35204'
    self.tax_rate = 0
    self.years = [2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017] 

def parse_m(self, response):
    r = json.loads(response.text)
    models = r['models']
    year = response.meta['year']
    make = response.meta['make']
    for model in models:
        for milage in [40000,50000,60000,70000,80000,90000,100000]:
            url = '****/vehicles/?year={}&make={}&model={}&state={}&mileage={}&zip={}'.format(year,make, model, self.state, milage, self.zip)

and this is the url i see in the log of scrapinghub:

***/vehicles/?year=2010&make=LOTUS&model=EXIGE%20S&state=%7B%7D&mileage=100000&zip=35204

Please indent you code properly. Also, if your for-loops are not inside any class method, the self.state will not return anything of value, and will probably not format your string the way you want. — Pax Vobiscum, Oct 23 '17 at 12:03
Fixed indents. Not sure whay you mean by class methods. But as you see the self.zip variable works perfectly fine. Though they have the same formatting and usage as far as I see. — Billy Jhon, Oct 23 '17 at 12:17
That seems really odd... A quick and dirty fix would be to just replace them with a `.replace('{}', '')` at the end of the url. Have you tried running this locally? How do you upload your code to scrapinghub? — Pax Vobiscum, Oct 23 '17 at 16:19
Please post your code, I also think that the problem is your `self.state` variable, might it be possible that it's changed somewhere else in the code? — Wilfredo, Oct 24 '17 at 22:15

score 0 · Answer 1 · answered Oct 23 '17 at 21:09

This is not a scrapinghub issue. It has to be your code only. If I do below

>>> "state={}".format({})
'state={}'

This would end up being

state=%7B%7D

I would add

assert type(self.state) is str

to my code to ensure this situation doesn't happen and if it does then you get an AssertionError

text substitution {} does not work at scrapinghub

1 Answers1