fyi- msnbot

[renegade600]

Just got a surprised. Was checking one of my website statistics for the first time in a few months and found bandwidth was being eaten by the msnbot. It found my site in January. Over 200 megs last month because of the msnbot. Had to do a quick robot.txt before it got too bad.

Just wanted to mention that if you have a website and have not check your stats recently, best do so. Might get a little surprise.

[degsy]

After the recent MSN Search release it's not surprising.

Thanks for the info

[Spider]

I think Bill is trying to catch up on Google's 5 years of indexing, and he wants to do it in 90 days :eek:

I think he forgets that most of us don't have our own backbones and terabyting our bandwidth may
make us sell our houses to pay the BW bill. :mad: :rolleyes: :laugh:

I see his bot a lot but I figured the extra BW bills now may bring more traffic later and as we all
know...traffic is the most valuable currency of cyberspace and more traffic (surfers anyway) is
never enough.

[renegade600]

I think Bill is trying to catch up on Google's 5 years of indexing, and he wants to do it in 90 days :eek:

I think he forgets that most of us don't have our own backbones and terabyting our bandwidth may
make us sell our houses to pay the BW bill. :mad: :rolleyes: :laugh:

I see his bot a lot but I figured the extra BW bills now may bring more traffic later and as we all
know...traffic is the most valuable currency of cyberspace and more traffic (surfers anyway) is
never enough.


:rotflmao:The extra traffic would be nice for normal sites but the site I do is basically a private informational one for a local organization. Its lucky to get 2 megs of usage a month from real visitors. Fortunately I had the bandwidth to cover it but ... Oh Well, I wonder who will be next with their bots. :dizzy:

[Spider]

basically a private informational one for a local organization.
noindex,nofollow on the meta will help keep out a lot of those crawlers and spiders.
Your right though, the robots.txt needs the crawler name to.

That just makes me wonder...I wonder if Bill has his crawlers and spiders paying attention
to the noindex,nofollow tags? I bet he is grabbing everything and just ensuring with a ping-bot
the page exists. He probably talks to himself in his dreams lately...

"Bill...BILL!! INDEX THE WORLD, Build the database and they WILL COME!" :rotflmao: :laugh: