Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GoogleOther to spider list #5972

Open
Fighter456 opened this issue Aug 2, 2024 · 0 comments
Open

Add GoogleOther to spider list #5972

Fighter456 opened this issue Aug 2, 2024 · 0 comments

Comments

@Fighter456
Copy link
Contributor

66.249.68.4 - - [02/Aug/2024:22:25:38 +0200] "GET REDACTED HTTP/1.1" 200 168162 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.182 Mobile Safari/537.36 (compatible; GoogleOther)"
66.249.68.6 - - [02/Aug/2024:22:25:13 +0200] "GET REDACTED HTTP/1.1" 200 51797 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.182 Mobile Safari/537.36 (compatible; GoogleOther)"

Please visit https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers?hl=en#googleother for more information.

GoogleOther is the generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Minor Task
Development

No branches or pull requests

2 participants