Legacy Ability to view more detailed Sitemap Statistics

[bzcomputers]

I would like to be able to view more detailed sitemap statistics.

Currently there is just a log of sitemap hits, for when a spider views a sitemap .xml.

It would be more useful if there was a sitemap statistics view where users could see stats from each sitemap build with a build date/time. Stats like total urls and an increase/decrease count in urls for each xml produced since last build. A "hit count" from all spiders to the sitemaps since the last build would also be useful to be able to compare search engines responsiveness over time.

Then if users have the sitemap set to run at a regular interval they will be able to get some decent data to compare their site activity day-to-day, week-to-week and month-to-month.

 

[neounix]

If DBSEO implements features which require more writes to the DB when a thread or post is visited for such a feature, we only ask that you insure we can "turn it off" if we do not want to use it.

Also, we already update every thread, post and tag DB when a thread, post or tag is visited by GoogleBot or BingBot and we run a cron once an hour to show us real-time stats of how our sites are being crawled.

Although we do get a lot of benefit from this view, it does degrade performance a bit and will we turn off our internal "statistics" code when we are happy that our recent SEO efforts have been effective.

Sample of our "bot stats":

GoogleBot Thread Index Count: 152024 - 107% of sitemap - Last 7 Days: 106% - Last 24 Hours: 33% - Last Hour: 1% (1560)

BingBot Thread Index Count: 110958 - 78% of sitemap - Last 7 Days: 78% - Last 24 Hours: 23% - Last Hour: 1% (1084)

GoogleBot Post Index Count: 142444 - 22% of sitemap - Last 7 Days: 22% - Last 24 Hours: 2% - Last Hour: 0% (6)

 

[bzcomputers]

If DBSEO implements features which require more writes to the DB when a thread or post is visited for such a feature, we only ask that you insure we can "turn it off" if we do not want to use it.

What I had in mind wouldn't require anymore writes to the DB except a few stats when the sitemap is actually produced. The rest of the info would come from the sitemap log, so it should be negligible even for the largest of sites

Now what you have going on there is a whole other story. I never thought I would ever say this but ..."You've got too much information!" That is definitely a load of statistics, that'd be a full-time job just to keep on top of them all! I assume that is a custom in house mod?

 

[Fillip H.]

We'll take it into consideration

 

[neounix]

That is definitely a load of statistics, that'd be a full-time job just to keep on top of them all! I assume that is a custom in house mod?

Yes, we write most of our own mods (for many years) when needed.

It's pretty simple really. We just key on the GoogleBot and BingBot UA strings when a thread, post or tag is read and we update the thread, post and tag DB tables with the UNIX timestamp of the access. Then, we run an hourly cron against the thread, post, and tag(list) tables to generate any and all the stats we wish.

We know precisely when each thread, post or tag has been indexed by the bots we are interested in, currently only GoogleBot and BingBot; and were thinking to add graphs, but we are too busy with other tasks at the moment.

 

[Fillip H.]

bzcomputers neounix

In v1.1.0, I'll be creating a "sitemap build log" which will track when the sitemap was built, how many URLs were added for each content type, and count how many spider hits and sitemap hits are registered for each build.

The new spider hits and sitemap hits will be persistent, and not pruned every X days like the existing spider log and sitemap log is.

The "Enable Sitemap Hit Log" setting will also be used to control whether these new hit counters should be used.