Tim Bray has a great entry on building web robots–or at least robots intended for search engines. I may assign this, along with his paper for WWW, for my class in the spring.