Web crawlers have become popular tools for collecting large portions of the web that can be used for many tasks from statistics to structural analysis of the web. Due to the amount of data and the heterogeneity of tasks to manage, it is essential for crawlers to have a modular and distributed architecture. In this paper we describe Lumbricus webis, (short L.webis) a modular crawling infrastructure built to mine data from the.it domain and portions of the web reachable from it. The purpose of our crawler is to support gathering of advanced statistics, and advanced analytic tools on the content of the Italian Web. This paper describes the architectural features of L.webis and its performance. L.webis can currently download a mid-sized ccTLD such as .it in about one week. © 2011 IEEE.
Medium sized crawling made fast and easy through Lumbricus webis
Geraci F;Pellegrini M
2011
Abstract
Web crawlers have become popular tools for collecting large portions of the web that can be used for many tasks from statistics to structural analysis of the web. Due to the amount of data and the heterogeneity of tasks to manage, it is essential for crawlers to have a modular and distributed architecture. In this paper we describe Lumbricus webis, (short L.webis) a modular crawling infrastructure built to mine data from the.it domain and portions of the web reachable from it. The purpose of our crawler is to support gathering of advanced statistics, and advanced analytic tools on the content of the Italian Web. This paper describes the architectural features of L.webis and its performance. L.webis can currently download a mid-sized ccTLD such as .it in about one week. © 2011 IEEE.I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
