A awasome collection of web crawlers ,spiders and resources in different languages. Tested by myself on Mac OS X with Python 3.
Scrapy: The King of Web Crawler Framework written by Python. A very fast high-level html-content scraping and web crawling framework.
A Complete List of all crawlers can be found under: