Question Crawl Errors from Google

[Bullmama DeLano]

Obviously since installing DBSEO I am received thousands of crawl errors, and since I can only delete 1000 at a time I've been slowly marking them as "fixed".

As the weeks have passed, however, I noticed I am still getting crawl errors from new rewritten threads. However, these threads were pruned. We list bulldogs for adoption on our site and after a year or so, we delete the threads.

I thought doing a forum and thread rebuild under maintenance would remove the URL's from the sitemap or index, but doing so did not help. So I searched google and found I probably should never delete or prune threads at all since there is no way of removing from the index. The URL's that are showing up are in fact ones I have delete recently. But some are VERY old too. Do they eventually understand the page does not exist?

I do notice when I try to view the URL's I get the vbulletin error message "No thread specified, if you followed a valid link, notify the administrator".

I am not even sure what my question is, but I am wondering if you know of any way to resolve this problem, or if DBSEO does something for these I don't know about. But from what I am reading on vbulletin.com and other threads about crawl errors, they are not good for your SEO and google will index you less due to being unstable if they get too many crawl errors.

If DBSEO doesn't filter out hard deleted threads, maybe this would be a wonderful addition to the product, or perhaps you already have a separate modification I don't know about that does this. :up:

As always, any help in understanding SEO world is welcomed and I am grateful for it!

 

[Fillip H.]

Firstly, let me reassure you that DBSEO does not include non-existant threads in the Sitemap. Each sitemap is built according to what exactly is in the database at the time, so if a thread has been physically removed it will not be added.

Secondly, I don't believe there's anything we can do regarding the crawl errors. The reason why they are showing up as errors is because DBSEO will list missing threads as 404 Not Found, which is actually a good thing. I can't comment on whether this actively harms your SEO, and I suspect this changes frequently as they fine-tune their algorithms.

Thirdly, as an alternative to deleting the threads, why not use the Thread Prefix system in vBulletin to mark the old threads as "Adopted" and then close them? This gets around the crawl errors issue, but it may be undesirable that these threads actually show up in search results. It's up to you, really

 

[Bullmama DeLano]

Thank you for helping me understand. The errors are 404 errors that I am seeing.

Yes I actually thought deleted threads would be beneficial- less to store, especially if not needed.... but I was so very wrong! We already prefix them as adopted, we just did this to 'clean up'. I also deleted threads started by some members I no longer wanted to see that have been banned. (oops!)

So if someone starts a thread in one forum, but we move it to another forum that it should be in (without a redirect), is that also bad? Will the original URL still 'exist' once it has been moved to the new URL?

 

[Fillip H.]

You should not receive any errors when doing so, even if the forum component of the URL changed. The forum component of the URL is there just for show, the "meat" of the URL is in the thread ID, which won't change

In short, it should be entirely safe to move them

Do keep in mind that if the threads are moved to a forum guests cannot see, then the crawl errors will simply change from 404 to 403 (No Permission) instead.

 

[Bullmama DeLano]

Okay that is good news, thank you!

For some reason I am getting a repeated crawl error on entry.php

Should I put that on my robots.txt?

Also getting soft 404 on images/buttons and images/buttons/misc on all my forums.

 

[Fillip H.]

If you're getting 403 errors on entry.php that means some blogs are now no longer accessible via usergroup permissions, and if it's 404 then it means those blog entries have been deleted.

If blogs are not guest accessible on your forum, you should deselect it from the sitemap in the settings.

 

[Bullmama DeLano]

If you're getting 403 errors on entry.php that means some blogs are now no longer accessible via usergroup permissions, and if it's 404 then it means those blog entries have been deleted.

If blogs are not guest accessible on your forum, you should deselect it from the sitemap in the settings.

Okay I will check if I get another one. They are guest viewable, and the only return is "entry.php" not "entry.php?b=368". I sure wish my blog section would work on rewrites.