Update on Pingdom Referrer Spam

Update, 11/16/2006: See this comment from Pingdom on the issue. They claim they are tweaking the GIGRIB bot for better behavior and will soon have a working monitoring page on the other end of the referrer. (Preview of that here.)

Update, 2007: Well, the public monitoring page never happened, and Pingdom’s spoofed referrers still forward to their main page. So yeah, they’re still referrer spammers.


My entry on Pingdom’s referrer spam gets a response in the comments from Pingdom themselves:

What Pingdom says:

Crawling the Internet for statistics and information is nothing new. Search engines and other statistical tools have been doing this since the start of the Internet. The referrer link you mention will be leading to a stats page as soon as the service goes live. The page will contain very basic statistics about your site, such as IP address, country of your server, etc. We are not collecting any visitor statistics (this is impossible for us or anyone else to do anyway) or any private information.

I respond to that in the comments, but here it is again just for purposes of clarity.

My response:

The issue, “Pingdom,” isn’t the crawling, it’s the spoofed referrer in the headers. You say that pingdom.com/monitor/whatever.com will eventually point to a working page, but right now it just rewrites to the front page of Pingdom, a page which does not actually link to the site being crawled. So the name of Pingdom and the name of the crawled site are being search-engine optimized in Pingdom.com’s favor in public referrer logs without any current reciprocal link.

That’s referrer spam, and that’s unethical. I would not be complaining about this if the referrer field in the uptime crawler were blank, or led to a current, working page with an actual link to the site, the way the Whois.sc crawler does it. As it is, Pingdom’s in my blacklist, until such time that the monitor link actually returns a real page.

I must add that it’s even worse for Pingdom’s reputation that their monitor bot is working from a server with a questionable history, and hosted by EV1, a company which Michael Pollitt has previously complained about as being lax on spammer activity.

Guys at Pingdom, I’m trying to help you out here. I’m not hating or trying to be an irritant; I’m pointing out where your bot practices are falling short of what people would expect of an ethical company. Uptime monitoring is fine, crawling web content for legitimate reasons is fine, in every way Pingdom appears to be a fine and legitimate service, except for those spoofed referrers. It’s the same reason I’m blocking RSSMicro’s bot.

Other complaints about the Pingdom bot’s behavior on various forums and weblogs follow. A lot of people are having trouble figuring out the purpose of the bot because of the referrer redirect, and are getting suspicious:

Also check out this Google search for the Pingdom referrer string, showing how the spoofed referrer is coming up on public logs everywhere.

On a related note, the original entry on Pingdom is a huge magnet for all sorts of comment spam now, mostly of the standard pharmaceutical and pornographic variety. MT’s spam filters have been catching all of it so far, but it’s interesting that the overwhelming majority of link spam I’ve been receiving is now targeted at that single entry, even more than at my older entries on referrer, comment, and trackback spam attacks.

Comments

  1. Pingdom says:

    In retrospect, it is really unfortunate that we have used “production settings” (including the page link) for the crawler before the site was ready. Your criticism is valid and we agree that we should have done things differently. We wanted to make sure that everything worked correctly and also clearly identify ourselves and what we were doing. In your example you mention whois.sc as having an acceptable practice by providing a link to a page, and we will be there very, very soon, within one or at most two weeks. To give you an idea, we have a small entry about this in our company blog which explains how the initial pages will work. http://royal.pingdom.com/?p=46.

    Small comment about EV1. We have also gotten EV1 to rename the server in question, a name that must have been left from the previous user of that server. It should now be listed as bot-12.pingdom.com. As for general practices of EV1, that is not for us to comment on. They are just one of the hosting companies we are using to lease servers.

    If you have questions, you are always welcome to contact us.

  2. Paulo says:

    Thanks, Pingdom. I’ve updated the entries involved to reflect your responsiveness to the issue, and look forward to the GIGRIB service.

  3. Eeve says:

    Well pingdom… anytime now. I’m still getting your hits in my logs. Get lost. And change that referrer… now! I’m still waiting….

    For all webmasters: I recommend password protecting ALL traffic stats pages.