website collector program

[Peter Schutze]

Im looking for a program that will download an entire website as a collection of text or html files

any suggestions ?

 

[aramis]

iCab.
www.icab.de

Several different browsers can do it. Icab does it storing the colection as a zipfile.

 

[far-trader]

Fair warning repeated...

Presuming you're thinking along the lines of using it to archive CotI, as long as whatever you're using doesn't work like (or is) a web-spider...

http://www.travellerrpg.com/CotI/Discuss/showthread.php?t=8335

I have no idea how iCab does its magic, and maybe crawlers have advanced in the intervening years to the point they won't bog or crash the site, but if you did your name would be mud here.

 

[aramis]

icab does it exactly like almost every web spider...

It requests page X, examines X for links, follows those links, and grabs those; Repeat.

 

[BeRKA]

Any spiders with bad behavior on the Zhodani Base will be banned. The IP from where that spider came will be banned.

I would also like to add that t.A.T.u. is not evil.

 

[FreeTrav]

With the caveat that I suspect it of being a 'bad' spider - and I have only used it with definitely abandoned sites or with explicit permission to capture sites that were going to be taken down against the owner's desire (financial issues, for example) - HTTrack (or WinHTTrack) would appear to do what you want.

 

[Peter Schutze]

ouch it seems i hit a nerve with that question

my intention is to backup 2 websites that I created and manage .... one for wargaming and one for the 2300 Great Game I am running

 

[far-trader]

ouch it seems i hit a nerve with that question...

Maybe a little. I wasn't too worried by your question but figured a heads up (not just aimed at you) wouldn't hurt. A little reminder for everyone

I hear mutterings now and then about the need to back-up CotI before it is "lost" and there was a chance this might have been one of those, or lead to the idea to do it by someone else.

Backing up your own site(s) is a totally different story. I hope you've found a good program with the leads here and it goes well.

 

[aramis]

Peter, in that case, if you have FTP access to the site, just go in via FTP and copy the whole site, folders and all.

 

[BeRKA]

ouch it seems i hit a nerve with that question

Yes, you did. Since my site has lots of dynamically created content, I am not very fond of bad bots at all.

Other than that, the latest suggestion from Aramis is a good way to do it. If you have telnet access, making a compressed file of everything and then downloading it is probably an even faster way to do it.

A good list for web-masters to check what to ban is:
http://user-agents.org/

 

[Peter Schutze]

got both sites downloaded no probs. thanks guys

oh and neither hosting service seems to allow ftp