Informicant crawler v0.3a

By popular demand, I’ve put up a windows version of the crawler I’ve been using here. Have fun with it, if you like. It’s very much as-is. It’s bunches and snippets of code thrown together late at night. I hope to take a clean run at it over the winter break, providing better integration with a database backend, better crawl management, and some really important stuff like the robot exclusion protocol. But if it is any use to you now, please feel free to take it for a spin. Some extremely rudimentary instructions are included in the zip.

This entry was posted in Uncategorized and tagged . Bookmark the permalink. Trackbacks are closed, but you can post a comment.

Post a Comment

Your email is never published nor shared. Required fields are marked *

*
*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

  • Tweets

  • Archives