Question robots.txt

[bidorbuy]

I have dragonbytes seo pro installed on vbullein 4
the robot.txt file looks like this:

# Comment the following line (add '#' at the beginning)
# to disable mod_rewrite functions.
# Please note: you still need to disable the hack in
# the vBSEO control panel to stop url rewrites.
RewriteEngine On

# Some servers require the Rewritebase directive to be
# enabled (remove '#' at the beginning to activate)
# Please note: when enabled, you must include the path
# to your root vB folder (i.e. RewriteBase /forums/)
#RewriteBase /

#RewriteCond %{HTTP_HOST} !^www\.yourdomain\.com
#RewriteRule (.*) http://www.yourdomain.com/forums/$1 [L,R=301]

##RewriteRule ^((urllist|sitemap_).*\.(xml|txt)(\.gz)?)$ vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]

RewriteCond %{REQUEST_URI} !(admincp/|modcp/|cron|vbseo_sitemap|api\.php)
##RewriteRule ^((archive/)?(.*\.php(/.*)?))$ vbseo.php [L,QSA]​

I need to add the following, but it seems to break the site:

User-agent: *
Disallow: /editpost.php
Disallow: /inlinemod.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /payments.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /report.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showpost.php
Disallow: /usercp.php
Disallow: /usernote.php​

Please advise how I can add these so that crawlers don't look at these pages

 

[Fillip H.]

robots.txt and .htaccess are two separate files.

User-agent: *
Disallow: /editpost.php
Disallow: /inlinemod.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /payments.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /report.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showpost.php
Disallow: /usercp.php
Disallow: /usernote.php

goes in robots.txt

<IfModule mod_rewrite.c>
	RewriteEngine On

	# If you are having problems or are using VirtualDocumentRoot, uncomment this line and set it to your vBulletin directory.
	RewriteBase /

	# If you are having problems with the rewrite from content/ to content.php, uncomment this line to turn MultiViews off.
	Options -MultiViews

	RewriteCond %{REQUEST_URI} !(admincp/|dbseocp/|modcp/|cron|mobiquo|forumrunner|api\.php|reviewpost/|classifieds/|photopost/)
	RewriteRule ^((archive/)?(.*\.php(/.*)?))$ dbseo.php [L,QSA]

	RewriteCond %{REQUEST_FILENAME} !-f
	RewriteCond %{REQUEST_FILENAME} !-d
	RewriteCond %{REQUEST_FILENAME} !/(admincp|dbseocp|modcp|clientscript|cpstyles|images|reviewpost|classifieds|photopost)/
	RewriteRule ^(.+)$ dbseo.php [L,QSA]
</IfModule>

goes in .htaccess (replace everything from your OP with this.

 

[bidorbuy]

O gosh .... super amature mistake, as you said it, it hit me.

Sorry about that, thank you for the assistance