I'm having some trouble with 'bots again.

This time, however, it's my fault.

Basically, the 'bots are indexing the pages that contain dynamic 'blog entries.

For example, Google might index my index2.shtml page and note that it contains the word "Freeze" - but once I type a few more entries, the one about Polly's Freeze is no longer displayed on index2.shtml because it's not one of the ten newest entries anymore.

This means that somebody can Google the words "double poo-poo" and get led to my main page, but when they get there those words are nowhere on that page. That's just too much disappointment for me to want to take responsibility for. I mean, when you want to read about double poo-poo you just shouldn't have to wait.

What I need to do is have the 'bots follow the links on pages like index2.shtml, but not index those pages themselves.

That way the links to the single entries, like this one, are followed, and only those (static) single-entry pages are indexed.

So here's what I've done:

1. I put this line into the code for my non-static pages:

This tells the 'bots that honor this type of line to follow any links found on the page, but not to index the page itself.

2. I put this line on all of my single-entry pages:

This does the exact opposite - it tells the 'bots that it's okay to index the page but not to follow any further links. These pages are a dead-end, in other words.

Of course these modifications only work if the 'bots are well-behaved. The ones that aren't I try to take care of with my robots.txt and .htaccess files as described in this old entry.

The whole thing would make Rube Goldberg proud.

I really need to simplify my 'blog configurations when I do my next site redesign. Until then I'll probably just do some minor tweaks like the one I made tonight.