Looking at Bots from Another Angle

By Brian Proffitt 
Managing Editor, BotSpot
BotSpot Bi-Weekly Newsletter
January 31, 2005

I was sitting on a chair in the lobby of the Fairmont Hotel in San Francisco, thinking Gee, this guy doesn't look like an evil man...

And yet, here Ian Nandhra was, talking about technology that could bring all this talk about bots around us to a screeching halt. His reception at BOT2001 was certainly a mixed one, once people found out what his company was doing.

Nandhra is the President and CEO of FINDbase.com, a private company located in the Sierra Nevadas in California. He was visiting the one-day seminar on bots this day not to reconnoiter the enemy, but to fuel a genuine interest in this field, because Nandhra has a great fascination with intelligent agents and their ilk.

"I think bots are wonderful. I love bots," Nandhra said.

He just wants to put the smack-down on bots running amok across the Internet, grabbing any data they can find with out the consent of the data owners.

That, in a nutshell, sums up the mission of FINDbase.com, which uses analysis products to locate, track, and if need be, stop bot activity hitting protected Web sites.

On the surface, this mission seems counter to everything the bot community is trying to do, but after some examination, one finds not a company full of anti-bot fanatics, but rather an organization concerned with maintaining some form of control on what has largely been an uncontrolled area of technology.

Nandhra uses the analogy of a Web site as a bank to describe what his company does. A bank, he describes, has locks, which are typically very effective but rather dumb. Locks are indiscriminate in who they keep out, after all, so not only do they keep out potential thieves, they will also keep out potential customers.

Thus, banks will hire bank guards, who can make judgment calls on who to let in the bank, and who to ask to leave, such as "the sweet little old lady who pulls out a gun in the lobby," Nandhra chuckled, "That's someone the bank guard might let in, but will certainly ask to leave."

On a Web site, the lock is the firewall, which will not let any traffic into the backend of the site, but will not control access if the visitor is visiting the "safe" areas of the site. The problem is, many commerce and price comparison bots stay well within the confines of the authorized areas of a Web site, and yet can still do damage to the site and its owners.

FINDbase.com, main product, INFOtector, is the security guard for the site, Nandhra explained. Instead of trying to identify who or what's coming in to the site, a nearly impossible task given some bot's lack of identification information,  INFOtector, monitors the activity of the incoming visitor and analyzes it against known bot activity.

If INFOtector, discovers a bot, it can then implement one of several responses, depending on the desires of the site's owners. It can deny access, delay access to the point where the bot's user will give up and move on, change the data being delivered, or relay the bot's request to a welcoming page for the site.

Why is it important to block bots like this? According to Nandhra , even the most benign bot can cause a problem for a Web site.

At the top of Nandhra's most wanted list, though, are the price comparison bots, which Nandhra describes as "bottom feeders who bring in the worst customers into a commerce site." If they bring a customer at all, he added, since a price comparison bot may be sending someone off to a competitor's site.

Other bots may be specialized search bots that a competitor is using to directly mine information about your company, something that seems to happen casually on the Web. "If I called someone and asked them for their inventory and pricing data, I guarantee you they would hang up on me," Nandhra said.

Even a plain-vanilla search bot can cause hassles for a site, since it could deep-link information on your site and bring surfers in through a pathway you did not want them to take, perhaps leading it away from your ad-revenue-generating pages.

Nandhra sees a genuine need for his company's product on the Internet. Traditional bot control methods, such as the honor-system use of a sites' robot.txt file, is simply ineffective and bot IDs are often unreliable.

"I don't want a bot coming in," Nandhra said of Web site owners, "I want you to come to my site and buy something."

One of the more creative measures INFOtector users could take when using this product is actually feeding price comparison bots false information. A customer using a bot may see the commerce site is selling Widget X for $80, for example, far below the site's usual $100.

Is the bot-using customer getting a better deal than the customer who surfed in by themselves from the Web? Not necessarily, Nandhra said. Once the lower price pulls them in, the price in the shopping cart would actually be $100.

Is this a case of false advertisement? Not at all, Nandhra explained, as the owners of the e-commerce site would have already stipulated in their copyright that the information they provide to customers is accurate--for human users of browsers only. Bots would be excluded from this statement, creating a caveat emptor situation for the bot users.

I raised the concerns some might have with what Nandhra is doing. After all, Web sites do put this information up there for all to see, why should his company prevent the fair use of a site's content?

"I suppose the knee-jerk reaction would be this," Nandhra said, "say you have the ability to reach into somebody's wallet and pull money out. Is it right?"

Upon additional reflection, Nandhra continued, "Sites are there to make money. The Internet is freely accessible. Content should not be free."

Nandhra also said that INFOtector, will now give people the ability to track the distribution of any content from a client Web site and now give Web site owners the ability to actually enforce copyright law on the Internet, a process that until now has been very difficult.

By giving Web site owners control over their data, bots would no longer be able to inadvertently skew all-important ad-revenue data and other related traffic data.

Nandhra emphasized throughout the interview that he harbors no ill will towards bots. If a bot comes in, check the robot.txt file, follows its restrictions and politely leaves, then INFOtector would kindly let it pass. It is the anarchic bots, the ones with little regard for a site's restrictions, that Nandhra wants to stop.

And after listening the discussions of BOT2001, Nandhra is more excited than ever about the future of bots. "It is the next level of Internet access," he said. But with this new dawn of bot technology, Nandhra sees some clouds in the future.

"I don't hold out any hope that there will be any behavior controls [for bots]."