Bots, harvesters, crawlers, and scrapers–these are all terms you may have read or heard about before, but do you know what they really mean? Like many things, they can have both legitimate uses or be used for purposes that are considered to be unethical in the Internet marketing world.
Bots
Bots are known under various names, including web robots, internet bots, or just bots. A bot is a software application that performs tasks repeatedly and much more quickly than a person could. Bots are used in many applications. One place you may see bots being used is online auctions, where they will automatically place bids up to a preset amount, or search auction websites for specified sale criteria. Another is automated online customer service. The customer service representative that you are chatting with may actually be a bot. The most prevalent use of bots is for web crawling or spidering. This is a method used by search engines to find and evaluate websites.
There are also uses of bots that many would consider bad or unethical. You may encounter bots being used in online video games to obtain items of value that would otherwise take much longer to find. Bots are used to automate various activities such as gathering email addresses, entering contests online, and casting votes for videos or blog posts to drive up popularity numbers. If you find you can’t get the tickets you want for a concert or event, it may be because someone used bots to buy as many as possible when they became available.
Crawlers
A web crawler or spider is a bot that searches the web automatically. It is usually used by search engines to identify and evaluate new websites and provide updated information on existing ones.
Harvesters and Scrapers
When you type a search term into Google or Bing and get a list of web pages, you have benefited from the use of a web harvester, also called a scraper. A harvester or scraper is a web crawler that is set to find and categorize specific information. If you have used a website that finds pages on specific topics such as legal advice or hobby supplies, a harvester was probably used to find the pages.
Another use of harvesters is that of gathering email addresses. If you have ever gotten spam email, your email address was probably gathered by a harvester.
Bots, crawlers, harvesters, and scrapers are types of internet technology. They have many legitimate functions but can be annoying or harmful when used for the wrong purposes.
Marketing Specialist for Web Hosting
You can follow the author @JulianaPayson on Twitter. Juliana is a Marketing Specialist for InMotion Hosting, a web hosting company from Los Angeles who can provide you with a virtual private server.