Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [wget]

GNU Wget (or just Wget, formerly Geturl) is a computer program that retrieves content from web servers, and is part of the GNU Project. Its name is derived from World Wide Web and get. It supports downloading via HTTP, HTTPS, and FTP protocols.

GNU Wget (or just Wget, formerly Geturl) is a computer program that retrieves content from web servers, and is part of the GNU Project. Its name is derived from World Wide Web and get. It supports downloading via HTTP, HTTPS, and FTP protocols.

Its features include recursive download, conversion of links for offline viewing of local HTML, support for proxies, and much more. It appeared in 1996, coinciding with the boom of popularity of the Web, causing its wide use among Unix users and distribution with most major GNU/Linux based distributions. Written in portable C, Wget can be easily installed on any Unix-like system.

Source: wikipedia

Man page

290 questions
2
votes
3 answers

wget and port forwarding

I have a weird problem. I can run this command just fine from various servers: wget --debug '--http-user=USER123' '--http-passwd=PASSWORD' http://GW-BOX:9091/weijhkdsvn/v9_odbc//CRONTAB.2014020 Where "GW-BOX" is the gateway to my network, USER123…
Tony B
  • 254
  • 3
  • 14
2
votes
2 answers

Wget on busybox v1.9.1

I have busybox 1.9.1 runing in huawei router. There is few commands, one of them wget and I want to use it, but there is some strange syntax. When I type wget www.google.com like I do normaly on my debian/mint etc, i get help message in this…
busybox
  • 21
  • 1
  • 1
  • 2
2
votes
1 answer

Wget Minimum Download Rate

Can anyone help with the parameter that ensures that wget will fetch the file at a minimum specified download rate?
user152666
  • 21
  • 1
2
votes
1 answer

Find out what's running wget on my server

In my access.log I see requests coming from 127.0.0.1 and retrieving server's "/" and then all of graphics and CSS referenced in it. This happens regularly, every 5 minutes. I want to find out what, why and how runs wget like this. I changed…
Pēteris Caune
  • 93
  • 1
  • 12
2
votes
3 answers

wget not working with domain on local machine

Basically - I have some PHP scripts that need to be run as cron jobs. Lets say the script needing to be run is: http://admin.somedomain.com/cron_jobs/get_stats If I run the script from the local machine it gives me a 404 Not Found error. So I…
user568829
  • 231
  • 1
  • 3
  • 8
2
votes
1 answer

Stream tar.gz file from FTP server

Here is the situation: I have a tar.gz file on a FTP server which can contain an arbitrary number of files. Now what I'm trying to accomplish is have this file streamed and uploaded to HDFS through a Hadoop job. The fact that it's Hadoop is not…
Charles Menguy
  • 123
  • 3
2
votes
2 answers

how can i use apache log files to recreate usage scenario

Recently i installed a website that had too many requests and it was too slow. Many improvements have been made to the web site code and we've also bought a new server. I want to test the new server with exactly the same requests that made the old…
daigorocub
  • 249
  • 2
  • 10
2
votes
3 answers

The --reject feature in wget doesn't seem to work

I am trying to mirror most of a ftp sever using wget. This particular sever keeps old copies of the data files in a folder called 'backup' inside every subfolder. e.g. '/MaffiaOffShore/backup' '/VeryVeryDodgy/backup', which I don't want but my…
WgetMonkey
2
votes
3 answers

wget application replacement

Is there any other application similar to wget? If yes can you post a link?
zozo
  • 783
  • 3
  • 11
  • 22
2
votes
1 answer

Why does the time it takes to run this script differ so much between user and system and how I might correct the discrepancy?

My application sits behind a load balancer, and every once in a while I like to do a status check on each machine to get an idea of the time it takes to return an index.html document on each machine. The script looks like this: for host in…
editor
  • 383
  • 2
  • 5
  • 21
2
votes
2 answers

WGet or cURL: Mirror Site from http://site.com And No Internal Access

I have tried wget -m wget -r and a whole bunch of variations. I am getting some of the images on http://site.com, one of the scripts, and none of the CSS, even with the fscking -p parameter. The only HTML page is index.html and there are several…
songei2f
  • 1,934
  • 1
  • 20
  • 30
2
votes
1 answer

Using lynx or wget in cron to call a php script?

I'm examining a particular setup, and they have their crontab as follows: 0 * * * * lynx http://www.example.com/cron/scriptA.php Of course, this relies on the 'security-by-obscurity' precept, because any person on the internet, knowing where those…
siliconpi
  • 1,807
  • 6
  • 32
  • 46
2
votes
1 answer

How to prevent external website from being downloaded in wget?

I'm using wget to download useful website recursively: wget -k -m -r -q -t 1 http://www.web.com/ But external website also downloaded along with the website I want. How to prevent this external website from being downloaded?
Serem
2
votes
0 answers

How to use sitemap.xml to create a static mirror of a CMS

Is there a tool to create a static mirror of a content management system (CMS) that provides a sitemap.xml file? Ideally, I would point a tool like wget or curl to a sitemap.xml file and have it automatically sync the static directories using the…
Lee Joramo
  • 21
  • 1
2
votes
2 answers

wget 5xx error code handling

i'm running a shell script that uses wget and pulls something over the web, my question is is there a way to tell wget to abort as soon as it hits 500 error code?
alexus
  • 13,112
  • 32
  • 117
  • 174
1 2 3
…
19 20