I recently covered how to make a mirror of a website with GNU’s wget command line program and in the comments of that post there were several suggestions for more advanced options which allow you to control your downloading further.
So I’ve decided to follow up on that post and look at some of the more advanced options that wget offers the user.
GNU’s wget command line program for downloading is very popular, and not without reason. While you can use it simply to retrieve a single file from a server, it is much more powerful than that and offers many more features.
One of the more advanced features in wget is the mirror feature. This allows you to create a complete local copy of a website, including any stylesheets, supporting images and other support files. All the (internal) links will be followed and downloaded as well (and their resources), until you have a complete copy of the site on your local machine.
In its most basic form, you use the mirror functionality like so:
$ wget -m http://www.example.com/
There are several issues you might have with this approach, however.
Locating files that you know exist, but you can’t remember where they are can be frustrating. There are a handful of GUI-based search tools for Linux and Unix, but if you prefer to work in the command line, or don’t have one of these special apps installed, you can often fall back on command line tool locate.
The locate command works by querying a database of your files. It won’t be very useful, unfortunately, unless this database is kept up to date. What I recommend you do is set up a cron job to run the /usr/bin/updatedb command (as root) every day/night so that the locate tool becomes useful.
Some distributions have taken the liberty of scheduling updatedb for you already, so you might find this step unnecessary and you can get straight to the searching.
Once you have the database in place, locating a file is as simple as this:
$ locate alostfile
If there are no results, you will be just sent back to a prompt, and you will need to widen or refine your search. Any results will be spat back out as a list of the paths to the files that it has found.
This method of searching works best when you do actually know the filename, or part of it, and as far as I’m aware doesn’t look in the contents of files to search.
Nevertheless, if you have lost that important file and don’t have an alternative desktop search program already installed, give locate a try.
