wget can be a very handy tool during development. Combine wget with other *nix utilities (like time and cron), and you can write some very useful scripts/snippets. For example,

wget -p -H -nv http://www.my-domain.com/cart/ 2>&1

uses -p/–page-requisites to download all the files necessary to render the page while -H causes it to span domains if necessary. If you pass this to grep (after redirecting the output (defaulted to STDERR) to STDOUT), you could look for 403/404’s or any other undesirable response code. The -nv/–no-verbose simply prevents the deluge of information that wget spits out by default.

Here’s part of a shell script run through my crontab that I have set up to time downloads during the development cycle. I can monitor a list of urls on my dev/test servers (passed in $1) and monitor the performance over time.

for i in $(
 # list of urls here
 echo '/product/index.jsp?productId=3633213'
 echo '/cart/index.jsp'
 # ....
); do
time -f "`date` | %E | $1$i\n" -a -o time.log wget -a wget.log -O /dev/null -p $1$i
done