Crowd crawling

Abstract

The emerging research for online social networks (OSNs) requires a huge amount of data. However, OSN sites typi- cally enforce restrictions for data crawling, such as reques t rate limiting on a per-IP basis. It becomes challenging for an individual research group to collect sufficient data by using its own network resources. In this paper, we intro- duce and motivate crowd crawling , which allows multiple re- search groups to efficiently crawl data in a collaborative way . Crowd crawling is carefully designed by addressing several practical challenges including resource diversity of diffe rent partners, strict request rate limiting from OSN providers, and data fidelity. We implemented and deployed a crowd crawling prototype on PlanetLab, and demonstrated its per- formance through evaluations. We have made the datasets crawled in our evaluation publicly available.