Answer by user27849 for How to make "crawl-delay" works for all "user-agent"...
I know you previously asked to block the bots and now you just want to slow them down.The only way to force them to slow down is not to serve them content if they are going too fast. The problem with...
View ArticleHow to make "crawl-delay" works for all "user-agent" in robots.txt or in...
I have this line: Crawl-delay: 50in the robots.txt file at the root of my website.But it seems that it is not working at all. I get 400 visits from the Yandex user agent in a few minutes.I don't want...
View Article
More Pages to Explore .....
