From Seo Wiki - Search Engine Optimization and Programming Languages

Jump to: navigation, search
Developer(s) Emil Romanus
Stable release 1.7.0 / June 23, 2009; 146765802 ago
Written in C
Operating system FreeBSD, Linux
Type web crawler open source
License ISC licence

Methabot is a scriptable web crawler designed for flexibility and speed. It is free software written in C, distributed under the terms of the ISC licence.

Methabot has wide support for customization. It can be scripted using Javascript with E4X, configured using its own configuration language, and dynamically switch configuration while running.

Key features

  • Scriptable using Javascript
  • Provides MySQL bindings to Javascript
  • Support for the Robots Exclusion Standard
  • User-defined filetype filtering and sorting, according to custom rules
  • Heavy multi-threading
  • Chaining of custom parsers
  • Converts HTML to real XML for E4X compatibility

External links

Personal tools

Served in 0.189 secs.