0

I am using the os module to issue a wget request through python. It looks something like this:

os.system("wget 'http://superawesomeurl.com'")

If I issue the wget request straight from terminal, it works, but I have two problems:

  1. When I build this in sublime, it give me the error: sh: wget: command not found
  2. When I enter this into a python shell command line it sends the request but comes back bad: 400 bad request

I noticed that other people don't use the quotes around the url, but in terminal this is the only way it works. I am using python 2.7.8 and running Yosemite.

admdrew
  • 3,790
  • 4
  • 27
  • 39
Elyza Agosta
  • 581
  • 2
  • 5
  • 8

3 Answers3

3

Your code should work if wget is on your PYTHONPATH.

But seriously, do not use wget in Python!

Better use a Python-native function like urlopen: https://docs.python.org/2/library/urllib2.html#urllib2.urlopen It's as simple as this:

from urllib2 import urlopen
response = urlopen("http://stackoverflow.com").read()

Now response contains the whole contents of the html page. You can also use readlines() if you wanted to iterate over it line by line. To save the html file to disk use:

download = open("index.html", "w")
download.write(response.read())
download.close()
Hubert Grzeskowiak
  • 15,137
  • 5
  • 57
  • 74
  • 1
    I don't see how *this works for me* solves the problem – Padraic Cunningham Nov 11 '14 at 14:46
  • You're right, guys. I've just removed the "works for me" part and included an example of urlopen. – Hubert Grzeskowiak Nov 11 '14 at 14:54
  • So I keep getting a bad request when I use urlopen. I'm not sure if this is because the link I'm using is for an API request with facebook, would that change anything? – Elyza Agosta Nov 15 '14 at 02:51
  • urlopen does a simple HTTP GET request by default. For API requests you might need to send some data, which is usually rather done using HTTP POST (see https://docs.python.org/2/library/urllib2.html#urllib2.urlopen). Or you might need to authenticate first (see https://docs.python.org/2/library/urllib2.html#examples - 4th example). – Hubert Grzeskowiak Nov 18 '14 at 12:34
  • Now there is a python package called `wget`: https://pypi.org/project/wget/ – Nagabhushan S N Aug 19 '19 at 07:43
0

I would guess one of two problems:

  1. wget is not in your path
  2. wget is not installed (I'm not trying to insult you)

From a bash terminal type which wget. That will tell you where wget is installed on your system.

[sri@localhost ~]$ which wget
/usr/bin/wget
[sri@localhost ~]$

If which didn't locate wget then use find:

sudo find / -name wget

Once you know the path to wget, try adding the complete path to wget in your call to os.system:

[sri@localhost ~]$ which wget
/usr/bin/wget
[sri@localhost ~]$ python
Python 2.7.5 (default, Feb 19 2014, 13:47:40) 
[GCC 4.8.2 20131212 (Red Hat 4.8.2-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.system('/usr/bin/wget "www.asciitable.com"')
--2014-11-11 09:29:52--  http://www.asciitable.com/
Resolving www.asciitable.com (www.asciitable.com)... 192.185.246.35
Connecting to www.asciitable.com (www.asciitable.com)|192.185.246.35|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html.1’

    [ <=>                                                                ] 6,853       --.-K/s   in 0.006s  

2014-11-11 09:29:53 (1.06 MB/s) - ‘index.html.1’ saved [6853]

0
>>>
shrewmouse
  • 5,338
  • 3
  • 38
  • 43
-1

Make sure wget is on your PYTHONPATH

from subprocess import Popen, PIPE
wget = Popen(["wget", "http://superawesomeurl.com"],stdout=PIPE).read()
print wget
krukita
  • 89
  • 1
  • 10