.Google.com's John Mueller answered an inquiry concerning why Google.com indexes webpages that are disallowed coming from crawling through robots.txt and why the it's secure to neglect the related Browse Console reports regarding those crawls.Bot Visitor Traffic To Concern Criterion URLs.The person talking to the inquiry chronicled that crawlers were actually creating links to non-existent inquiry specification URLs (? q= xyz) to web pages with noindex meta tags that are additionally blocked in robots.txt. What urged the inquiry is actually that Google.com is creeping the web links to those webpages, getting obstructed by robots.txt (without envisioning a noindex robotics meta tag) then acquiring shown up in Google.com Look Console as "Indexed, though blocked by robots.txt.".The individual inquired the following inquiry:." But here is actually the large concern: why would certainly Google.com mark webpages when they can't even find the information? What is actually the benefit in that?".Google's John Mueller validated that if they can't creep the webpage they can't see the noindex meta tag. He additionally produces an exciting acknowledgment of the site: hunt operator, urging to dismiss the outcomes due to the fact that the "ordinary" customers won't observe those end results.He wrote:." Yes, you're correct: if our team can't crawl the web page, our experts can not see the noindex. That mentioned, if our company can't creep the webpages, after that there is actually certainly not a great deal for our team to index. Therefore while you may observe a few of those webpages with a targeted site:- inquiry, the common customer will not see all of them, so I definitely would not fuss over it. Noindex is likewise alright (without robots.txt disallow), it simply suggests the Links are going to find yourself being actually crawled (as well as find yourself in the Browse Console record for crawled/not catalogued-- neither of these conditions trigger concerns to the rest of the website). The fundamental part is that you do not make all of them crawlable + indexable.".Takeaways:.1. Mueller's answer confirms the restrictions in using the Internet site: hunt advanced hunt operator for analysis main reasons. Some of those causes is since it's not attached to the frequent search index, it is actually a different trait completely.Google's John Mueller commented on the site hunt driver in 2021:." The quick response is that a website: question is not implied to become total, nor made use of for diagnostics objectives.A website question is actually a specific kind of hunt that confines the end results to a specific web site. It is actually essentially simply the word site, a colon, and afterwards the website's domain name.This inquiry restricts the end results to a specific website. It's certainly not indicated to become a thorough compilation of all the pages coming from that internet site.".2. Noindex tag without utilizing a robots.txt is actually fine for these sort of conditions where a robot is connecting to non-existent pages that are obtaining found through Googlebot.3. URLs along with the noindex tag are going to create a "crawled/not catalogued" item in Explore Console and that those will not have an unfavorable effect on the remainder of the site.Read through the concern and also address on LinkedIn:.Why would Google.com index web pages when they can't also see the information?Featured Picture by Shutterstock/Krakenimages. com.