"If Google crawls 1,000 pages/day, Googlebot crawling many dupe content pages may slow down indexing of a large site. In that scenario, do you recommend blocking dupes using robots.txt or is using META ROBOTS NOINDEX,NOFOLLOW a better alternative?"
Matt Cutts, head of spam at Google, sees blocking pages using robot.txt as a last resort.
I can vouch that our research has pointed out that there is almost never a need to blog pages via robot.txt for duplicate content. Google and other search engines are pretty good at figuring out duplicate content and the static page that should be the considered the main content page.
However, he points out that if you have an incredible weird site with massive amounts of duplicate content, you might find it necessary.
I'd like to add that if that is the case, there is something wrong with your website architecture, and you should talk about creating a better navigation and indexing structure for your website with your web designer.
So don't believe the SEO myth: no need to get freaky about duplicate content, Google, Bing and Yahoo can handle the issue quite well (I can hear a lot of angry SEO's right now, but I simply have never met an SEO who had credible data that showed that exclusion of pages via robot.txt brought positive results, while I know of 3 cases where it was actually harmful.)
Always here when you need me. I am the forum administrator and a web designer for Online Design Bureau.
