[Chugalug] spiderring a site question

Robert A. Kelly III bluethegrappler at gmail.com
Sat Nov 9 22:29:08 UTC 2013


On 11/03/2013 12:23 PM, Rod wrote:
> I'm thinking of mirroring a site that is about to shutdown. But first I
> want to determine what kind of storage I need.
> My normal tool is wget. But I don't see a way to gauge file/site size
> without downloading. Any ideas on how spider a site and determine file
> size without downloading?

 wget has an option specifically for this. It's actually called "--spider".


More information about the Chugalug mailing list