Here’s a cool idea, set up a kind of honeypot for search engines’ bots (and spambots), laid out as a binary tree, and see how the bots behave when crawling the site. The details are written up here. Each page had a node number, written out in English, a comment box (which got populated by spam) and, it seems, a visualization of the populated tree surrounding that node. They’ve tracked how the bots traversed this space over time.
The image here is a thumbnail version of the Yahoo! Slurp bot’s travels over the 2 million pages. Great stuff.
Bot Paths
Here’s a cool idea, set up a kind of honeypot for search engines’ bots (and spambots), laid out as a binary tree, and see how the bots behave when crawling the site. The details are written up here. Each page had a node number, written out in English, a comment box (which got populated by spam) and, it seems, a visualization of the populated tree surrounding that node. They’ve tracked how the bots traversed this space over time.
The image here is a thumbnail version of the Yahoo! Slurp bot’s travels over the 2 million pages. Great stuff.
(via information aesthetics)
Share this: