Google Crawler

Follow

Steve Hoyer

• September 08, 2021 16:41

I was just about to upgrade to 12.2 and I noticed one (three, actually) of my active connections appeared to be from the google bot crawling our website. While I don't expect them to get a lot of information, if it's possible I'd like to put an end to this. Can we add a robots.txt to to block (for certain values of block where crawlers pay attention to that) that traffic or would this be an IP by IP effort?

0

• 

  Dana Anderson

  • September 08, 2021 16:50

  Hi, Steve. 

  You can add a robots.txt file, if you want to stop typical search engine web crawlers from accessing the HTTPS web client, you can drop a robots.txt file in

  C:\Program Files\Cerberus LLC\Cerberus FTP Server\webadmin\client

   

  0

  Comment actions Permalink

Please sign in to leave a comment.