Challenges in Web Crawling

Web crawling brings with it, a load of challenges. Politeness, Freshness, Coverage, Scale, Avoiding Traps, ...the list goes on. In this talk, we present a brief summary of these challenges.