By popular demand, I’ve put up a windows version of the crawler I’ve been using here. Have fun with it, if you like. It’s very much as-is. It’s bunches and snippets of code thrown together late at night. I hope to take a clean run at it over the winter break, providing better integration with a database backend, better crawl management, and some really important stuff like the robot exclusion protocol. But if it is any use to you now, please feel free to take it for a spin. Some extremely rudimentary instructions are included in the zip.
Informicant crawler v0.3a
By popular demand, I’ve put up a windows version of the crawler I’ve been using here. Have fun with it, if you like. It’s very much as-is. It’s bunches and snippets of code thrown together late at night. I hope to take a clean run at it over the winter break, providing better integration with a database backend, better crawl management, and some really important stuff like the robot exclusion protocol. But if it is any use to you now, please feel free to take it for a spin. Some extremely rudimentary instructions are included in the zip.
Share this: