Is crawl Budget a thing for smaller websites?
(Results at the bottom of the page)
I have always read about crawl budget optimization for SEO and how removing low quality pages from your website can have a positive impact for organic traffic. Most notably when Britney Muller, from Moz, no indexed 65k+ pages and within 30 days search traffic increased.
.@britneymuller noindexed 65K+ low quality pages from Moz (profile pgs w/ low contributions). 30 days later, search traffic +13% #Mozcon
— Rand Fishkin (@randfish) July 19, 2017
What would happen if we no indexed low quality pages on a smaller site?
I started helping a friend with a much smaller website and I wondered what would happen if we no indexed some of the low quality pages and removed from Google's index. Those "some" low quality pages turned out to be 50% of the pages that were indexed. The pages that I was targeting to remove were blog tag pages and product tag pages. These tag pages were very low quality and could be considered as duplicate content.
To Index or To no Index and a Plan
I checked the tag pages to see if any were driving traffic and should be kept index. The clicks were in the 10's for all tag pages, so not a big risk here. The plan was to no index all of the product tag pages and all of the blog tag pages. Actually no indexing the pages was super easy thanks to the Yoast SEO plugin.
No index! I Can't Lie I was kinda Nervous
Just de-indexed/no indexed a handful of URLs on a smaller site I am working on. Interested to see what happens! 😬
— elliott. (@originalelliott) February 22, 2018
I added no index to all of the tag pages on 2/19-2/20. After adding the no index tag, I created a "html" sitemap, that listed the the tag pages, and submitted to be crawled/indexed in Google Search Console. The idea was for Google to pick up the No index tag faster and remove these pages from Google. Hat tip to AJ Kohn for mentioning this top at a recent Bay Area SEO meet up.
Results (Not Scientific)
Its been about 30 days and interesting enough traffic has increased! I think there are many factors that could be involved here but still! Is crawl budget a thing for smaller sites? In this case is it less about crawl budget and more duplicate content? Anyways it was fun doing this and it always feels good to help a friends website grow 30%+ :)...thanks for reading.