Legacy Options for Multiple Sitemap Indexes and Sitemap Directories

[neounix]

We installed the 1.0.0.9b update today and all went smoothly. Thanks. Great job.

We have custom code in place (just a quick hack, no options, settings, yet) which changes the directory of the sitemaps based on the name of the index file.

$neo_sitemap_path = $vbulletin->options['dbtech_dbseo_sitemap_path'];

($hook = vBulletinHook::fetch_hook('neo_dbseo_sitemap_start')) ? eval($hook) : false;

if ($vbulletin->GPC['fn'])
{
        $sitemap_filename = preg_replace('#[^a-z0-9_.]#i', '', $vbulletin->GPC['fn']);
        $sitemap_filename = preg_replace('#\.{2,}#', '.', $sitemap_filename);

        if (substr($sitemap_filename, -4) != '.xml' AND substr($sitemap_filename, -7) != '.xml.gz')
        {
                $sitemap_filename = '';
        }
}
else if (file_exists($neo_sitemap_path . '/dbseo_sitemap_index.xml.gz'))
{
        $sitemap_filename = 'dbseo_sitemap_index.xml.gz';
}
else if (file_exists($neo_sitemap_path . '/dbseo_sitemap_index.xml'))
{
        $sitemap_filename = 'dbseo_sitemap_index.xml';
}
else
{
        $sitemap_filename = '';
}

if ($sitemap_filename AND file_exists($neo_sitemap_path . "/$sitemap_filename"))

.... blah blah blah

Basically, we just read the name of the sitemap file and change directories based on the name. We do this because we like to keep certain sitemaps seperate so we can control (to a small degree) how the site is indexed.

Our "hack" is working fine; but we have to rehack your code during each release, so if you could add something like this the future, that would be great.

Also, on another note, we had a bit of a problem (in both releases we have used) in some areas of our forums when the plugin files were included with the statement:

include(DIR. '/dbtech/dbseo/blahblah.php');

so we globally changed the XML upload file to hardcode the exact path:

include('/var/www/dbtech/dbseo/blahblah.php');

... and the PHP error problem in some areas of our forums went away...

No big deal, it's easy for us to hard code this in before updating.

 

[neounix]

OBTW, our hook code above is really simple:

if (preg_match ('/thread|tag|forum/',$vbulletin->GPC['fn'])) 
   { 
    $neo_sitemap_path = '/website/www/dbseo_sitemaps'; 
   } 

else if (preg_match ('/post/',$vbulletin->GPC['fn'])) 
   { 
    $neo_sitemap_path = '/website/www/dbseo_sitemaps_posts'; 
   } 
else 
   { 
     $neo_sitemap_path = $vbulletin->options['dbtech_dbseo_sitemap_path']; 
   }

I should have put the preg_match() pattern and the corresponding sitemap_path (s) in a vB options page..... but it was just a hack, so I've not cleaned it up yet.

 

[Fillip H.]

Could you explain a bit more why having separate directories improves control over how your site is indexed?

When you submit an index file to a search engine, which is what DBSEO does at the end of building the sitemap, the search engine opens the index file and looks for the corresponding files.

Do you not auto-submit to search engines, and only manually submit individual files?

 

[neounix]

Yes, we manually submit our sitemaps... and we have a different index for the threads, forums and tag... and a different index for the posts.

We do this because it is easier for us to track what GoogleBot is doing at a "high level" and also since indexing threads are a higher priority than posts, we have seen problems where GoogleBot will be indexing posts in the sitemap (even with a lower priority set) instead of threads.

So, we prefer to separate these out with at least two different indexes; and because your code removes the old sitemaps, we don't want to overwrite our old sitemaps each time, so there is reason for us to keep them in distinct directories.

Finally, it may just be me; but we've always submitted sitemaps manually using webmaster tools. I'm not comfortable with an automated process that submits them automagically, I guess because I don't understand what is going on since it's not using our webmaster tools login information.

I assume the sitemaps will show up somehow in our Google and Bing webmaster tools when submitted automatically; but since I don't understand that process, we just submit manually.

...

 

[Fillip H.]

For the record, if you submit a sitemap to Webmaster Tools then it will automatically update every time DBSEO submits a new sitemap. I observe our Webmaster Tools sitemap and notice that it re-processes them every day because our sitemap updates every day.

Furthermore, automatically submitting a sitemap will, in my experience, not cause them to show up in Webmaster Tools.

I've also found that the priority in the sitemap is related to searches that would return both pages. For instance, if you have a custom page set at priority 0.5 and threads set at 0.9, if a keyword "kernel" appears on both those pages then the thread will appear first in the search results.

Search engines may use this information when selecting between URLs on the same site, so you can use this tag to increase the likelihood that your most important pages are present in a search index.

(Reference: sitemaps.org - Protocol )

I've found nothing to suggest that the priority affects when content is indexed.

 

[neounix]

ROTFL, yes, you just supported by view. Since priority in the sitemap does not effect indexing, we control it by submitting our threads and having those completely indexed before posts.

Anyway dude, I don't hope to convince you otherwise; we we have been doing this for a long time and know what works good for our site; as we have done it "both ways" and we simply don't submit one huge single "across the entire site" sitemap; and we find doing it this way works very well for us over the years; and we current show an index in Google's realm of:

About 1,370,000 results (0.22 seconds)

.. of course, this changes all the time since 25 to 50% of all our threads are visited each day by GoogleBot.

We can only infer what goes on by observation since the exact algorithms are not published, and we do look at a lot of statistics; so no problem if you don't see the value in our method... we will continue to do it our way and hack the code as required to make it happen.

((I'm not keen at trying to convince others to do things our way.... we can easily write the hooks into the code to do it as well like, or just not update dbseositemap.php file, if not necessary (a bug fix) as it's a pretty straight forward process.))

It's all good!

Cheers.

 

[Fillip H.]

I'll add a hook just above

if ($vbulletin->GPC['fn'])

called

dbtech_dbseo_sitemap_view_start

that you can use to override the

$vbulletin->options['dbtech_dbseo_sitemap_path']

variable.

The files have also been updated to add this new hook right now.

 

[neounix]

it would also be helpful to do something lik this:

$dbseo_sitemap_path = $vbulletin->options['dbtech_dbseo_sitemap_path'];

.. before your new hook and then change all the instances of

$vbulletin->options['dbtech_dbseo_sitemap_path']

to

$dbseo_sitemap_path

in the rest of the code in that file.

(thanks)

 

[neounix]

.. if we overwrite this option.. $vbulletin->options['dbtech_dbseo_sitemap_path']

... it will change in the DBSEO AdminCP based on the bot activity at the moment; so I think it is better not to overwrite $vbulletin->options['dbtech_dbseo_sitemap_path'] , but to assign it to a local variable first; then work the the local variable.

 

[Fillip H.]

No, it will not change the AdminCP value. $vbulletin->options is a local array of the contents of the datastore cache for vBOptions, it's not a reference that will also update the corresponding setting.

 

[neounix]

OK, thanks.

 

[neounix]

.. and thanks for the hook! much appreciated!

 

[neounix]

Thanks again for the hook. Installed working fine..

if (preg_match ('/thread|tag|forum/',$vbulletin->GPC['fn'])) 
   { 
    $vbulletin->options['dbtech_dbseo_sitemap_path'] = '/website/www/dbseo_sitemaps'; 
   } 

else if (preg_match ('/post/',$vbulletin->GPC['fn'])) 
   { 
    $vbulletin->options['dbtech_dbseo_sitemap_path'] = '/website/www/dbseo_sitemaps_posts'; 
   }

I might create some vB options and make it a bit more generic and cleaner in the future; but it does what we need now and we are in no hurry to clean it up.

Cheers and Happy Holidays!

Thanks again for your continuing support for DBSEO version 3.8. We have no plans to ever upgrade to any of the Internet Brands versions of vB; unless we see a different philosophy in design, support and development (we are not holding our breathes for this, LOL); so we will stick with 3.8 and really appreciate your support for DBSEO.

Have a warm and safe Christmas holiday.