User-agent: * Disallow: /find/ Disallow: /verify Disallow: /lastfm User-Agent: Googlebot Disallow: /flickr Disallow: /lastfm Disallow: /twitter User-Agent: netEstate FOAF crawler Disallow: *