Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
ftp-queue
THE SCRIPT IS IN BETA. It will be more optimized.
ftp-check.py is a script used to create item lists for ftp-grab.
How to run
Clone the repo. cd into the repo and create a file with a list of FTPs you want to check.
Use python ftp-check.py listofftpstobechecked to run the discovery. If you scan a FTP that has already been scanned and has an archive file in the archive dir, only new files or files with a different size will be added to the item lists.
If you have scanned a FTP site, let Archive Team know in #effteepee on IRC efnet. Provide the itemlistfiles from the items dir and the new files from the archive dir.