Methabot
From Seo Wiki - Search Engine Optimization and Programming Languages
| This article may not meet the general notability guideline. Please help to establish notability by adding reliable, secondary sources about the topic. If notability cannot be established, the article is likely to be merged, redirected, or deleted. (March 2009) |
| Developer(s) | Emil Romanus |
|---|---|
| Stable release | 1.7.0 / June 23, 2009 |
| Written in | C |
| Operating system | FreeBSD, Linux |
| Type | web crawler open source |
| License | ISC licence |
| Website | http://metha-sys.org/ |
Methabot is a scriptable web crawler designed for flexibility and speed. It is free software written in C, distributed under the terms of the ISC licence.
Methabot has wide support for customization. It can be scripted using Javascript with E4X, configured using its own configuration language, and dynamically switch configuration while running.
[edit] Key features
- Scriptable using Javascript
- Provides MySQL bindings to Javascript
- Support for the Robots Exclusion Standard
- User-defined filetype filtering and sorting, according to custom rules
- Heavy multi-threading
- Chaining of custom parsers
- Converts HTML to real XML for E4X compatibility
[edit] External links
| File:Nuvola apps emacs.png | This free software-related article is a stub. You can help Wikipedia by expanding it. |
