The growing pains of crawling the web

IndexNow provides many benefits. It allows webmasters to notify all search engines at once. It democratized indexing and leads to fewer resources needed by search engines to crawl the web.

Search Engines have been struggling with crawling for a while. Challenges include spam, javascript rendering, but also the increased use of nofollow tags, which is one of the reasons Google started treating nofollow more as a suggestion than a directive.

I don’t think web discovery through links is the best approach and expect more search engines to lean on webmasters to bring new content to them through APIs like IndexNow.

Kevin Indig, Growth Memo