Nirjonadda
Customer
I think that a great option if DBSEO Can Spiders Ban , Who Spiders can allow for Pages Crawled Search Engines?
Upvote
0
Legacy Ban Spiders
- Status
Nirjonadda
Customer
This option ability to ban a certain spider (such as Yandex) and ban all spiders from accessing a certain page.
The second half of that should be accomplished by robots.txt files: The Web Robots Pages
Banning certain spiders can be done, so I'll place that under consideration
Banning certain spiders can be done, so I'll place that under consideration
neounix
Customer
Yandex and other aggressive spiders do not honor robots.txt
We used to have mod code that blocks those spiders, but we moved much of that code to the Apache2 configuration.
In my view, i's not really a "core SEO" functionality as much as it is a "manage user agents" application; and there is a lot that can be done with managing user agents:-
-- banning user agents
-- redirecting user agents
-- triggering mobile apps based on user agents
-- hiding content from certain user agents
-- banning certain pages based on user agent
-- restricting certain forum features from user agents
-- statistics based on user agents
-- displaying user agents on the site (all, not only bots)
-- managing user agents in the whoisonline bot display)
etc.
We used to have mod code that blocks those spiders, but we moved much of that code to the Apache2 configuration.
In my view, i's not really a "core SEO" functionality as much as it is a "manage user agents" application; and there is a lot that can be done with managing user agents:-
-- banning user agents
-- redirecting user agents
-- triggering mobile apps based on user agents
-- hiding content from certain user agents
-- banning certain pages based on user agent
-- restricting certain forum features from user agents
-- statistics based on user agents
-- displaying user agents on the site (all, not only bots)
-- managing user agents in the whoisonline bot display)
etc.
Last edited:
Nirjonadda
Customer
Have any plan to adding this Features in next Version?
We have no current plans to add this functionality because it would be better served by modifying .htaccess files or Apache configuration.
Here's an example:
Here's an example:
Code:
SetEnvIfNoCase User-Agent "^Yandex*" bad_bot
SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
SetEnvIfNoCase User-Agent .*exabot.* bad_bot
SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
SetEnvIfNoCase User-Agent .*semrushbot.* bad_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>- Status
