Pejman Moghadam / General

wget syntax example

Public domain

---

General

wget -o LOGFILE -bc URL
  -b  go to background
  -c  continue download
  -o  log all messages to LOGFILE
wget -bc -i FILE
  -i  read URLs from FILE

---

Site mirror

wget -bc --mirror URL 
wget --mirror --convert-links --backup-converted -w 1 URL
  -w  Wait the specified number of seconds between the retrievals

---

Changing host header

wget --header="Host: foo.bar" http://localhost/

---

Recursively download special file extensions

wget -r -l0 -N -H -t2 -np -A.mp3 -erobots=off -i url-file.txt

  -r Turn on recursive retrieving
  -l Specify recursion maximum depth level
  -N Turn on time-stamping
  -H Enable spanning across hosts when doing recursive retrieving
  -t Set number of retries
  -np Do not ever ascend to the parent directory
  -A Specify comma-separated lists of file name suffixes or patterns to ACCEPT
  -e robots=off Ignore the robots.txt file/no-follow attribute
  -i Read URLs from
  -D Specify comma-separated lists of domains to be followed.
  --exclude-domains Specify comma-separated lists of domains that are NOT to be followed.

wget -c -e robots=off -np -mirror -R .tar.gz,.tar.xz,.tar.bz2,.html http://slackware.at/data/slackware-14.0/source/

  -R Specify comma-separated lists of file name suffixes or patterns to REJECT

---

BY: Pejman Moghadam
TAG: wget
DATE: 2011-09-02 14:14:39

---

Pejman Moghadam / General [ TXT ]