From Seo Wiki - Search Engine Optimization and Programming Languages

Jump to: navigation, search
File:ArachnodeDotNetHome.jpg homepage
Developer(s) Mike Anderson
Stable release 2.0 / January 31, 2010; 127502405 ago
Written in C#
Operating system Windows
Type Web crawler
License GPL
Website is a .NET web crawler written in C# using SQL 2008 and Lucene and is released under the GPL.


  • Lucene integration allows for full-text searching through a familiar web interface.
  • Can be configured to run any number of threads and to use as much or as little processor time and memory as required.
  • Plug-in architecture allows for custom pre- and post-request crawl rules and actions without source recompilation. The existing crawl rules and actions architecture easily enables crawling enhancements such as federation, partitioning and distributed caching.
  • Pre- and post-request rules governing address and content filtering, robots.txt behavior, request frequency and crawl depth.
  • 250 stored procedures, views and functions designed for use with SQL Server Analysis Services and other business intelligence software. These procedures and views address trending, popularity, term extraction, phrase extraction and many other common analysis and reporting needs.
  • Pre-configured with several SSIS procedures to extract and prepare key information from collected data for text mining and analysis.
  • Extract, store, and index all discoverable Exif data fields from discovered images.


Content Aggregation: Use for personal content aggregation, crawling intranets of any size or crawling the Internet as a whole. Discovered content is parsed and stored into multiple configurable forms and locations.

Research and Analysis: Extract, collect and parse downloaded content into multiple forms, including XML. SSIS packages and Common Language Runtime functions extract terms and phrases from text content, and provide over 250 stored procedures, views and functions to jumpstart SQL Server Analysis Services or other text mining applications.

Search: Discovered content is indexed and stored in Lucene indexes and can be searched through a familiar Web interface.

Text Mining: Extract words, phrases, tags and text from discovered content.

Education: Learn introductory to advanced crawling techniques, and features of the .NET Framework and SQL Server 2008, including full-text indexing, multi-threading, caching, reflection, interfaces, object-oriented concepts, SQL common language runtime functions and regular expressions.

External links

Personal tools

Served in 0.286 secs.