.Google.com's John Mueller answered an inquiry about why Google.com indexes web pages that are disallowed from creeping by robots.txt as well as why the it is actually secure to disregard the related Look Console files about those creeps.Robot Traffic To Query Guideline URLs.The individual inquiring the question chronicled that crawlers were actually producing hyperlinks to non-existent inquiry criterion URLs (? q= xyz) to pages along with noindex meta tags that are actually also obstructed in robots.txt. What prompted the question is actually that Google is actually crawling the web links to those web pages, obtaining blocked out by robots.txt (without seeing a noindex robots meta tag) at that point receiving turned up in Google Explore Console as "Indexed, though obstructed by robots.txt.".The person asked the observing concern:." But right here is actually the major question: why will Google.com mark pages when they can not also see the material? What is actually the perk in that?".Google.com's John Mueller confirmed that if they can't creep the web page they can't view the noindex meta tag. He also creates an intriguing reference of the internet site: search driver, urging to neglect the end results given that the "normal" individuals won't view those results.He wrote:." Yes, you are actually appropriate: if our experts can't crawl the web page, we can not see the noindex. That pointed out, if we can not creep the pages, at that point there is actually certainly not a lot for us to index. So while you might see some of those web pages along with a targeted site:- query, the normal user will not see all of them, so I definitely would not bother it. Noindex is actually also alright (without robots.txt disallow), it only indicates the URLs will definitely end up being actually crawled (and also wind up in the Browse Console document for crawled/not recorded-- neither of these statuses result in issues to the remainder of the internet site). The vital part is actually that you don't create all of them crawlable + indexable.".Takeaways:.1. Mueller's answer validates the restrictions being used the Site: hunt evolved hunt operator for analysis explanations. Among those reasons is considering that it's certainly not connected to the normal search index, it's a distinct thing entirely.Google.com's John Mueller talked about the web site hunt operator in 2021:." The quick solution is that a web site: query is certainly not implied to be total, neither used for diagnostics purposes.A site question is actually a particular type of search that confines the outcomes to a specific website. It is actually essentially merely the word site, a bowel, and then the web site's domain name.This question confines the end results to a particular website. It's certainly not meant to be a complete compilation of all the webpages coming from that web site.".2. Noindex tag without making use of a robots.txt is actually great for these sort of conditions where a crawler is connecting to non-existent pages that are obtaining discovered by Googlebot.3. URLs along with the noindex tag will definitely produce a "crawled/not listed" item in Browse Console which those won't have a bad effect on the rest of the web site.Read the concern as well as address on LinkedIn:.Why would Google.com index web pages when they can not even find the content?Included Photo through Shutterstock/Krakenimages. com.