BeautifulSoup and lxml are libraries for parsing HTML and XML. Scrapy is an application framework for writing web spiders that crawl web sites and extract data from them.
Scrapy provides a built-in mechanism for extracting data (called selectors) but you can easily use BeautifulSoup (or lxml) instead, if you feel more comfortable working with them. After all, they’re just parsing libraries which can be imported and used from any Python code.
In other words, comparing BeautifulSoup or lxml to Scrapy is like comparing urllib or urlparse to Django (a popular Python web application framework).
No, and there are no plans to port Scrapy to Python 3.0 yet. At the moment Scrapy works with Python 2.5 or 2.6.
Probably, but we don’t like that word. We think Django is a great open source project and an example to follow, so we’ve used it as an inspiration for Scrapy.
We believe that, if something is already done well, there’s no need to reinvent it. This concept, besides being one of the foundations for open source and free software, not only applies to software but also to documentation, procedures, policies, etc. So, instead of going through each problem ourselves, we choose to copy ideas from those projects that have already solved them properly, and focus on the real problems we need to solve.
We’d be proud if Scrapy serves as an inspiration for other projects. Feel free to steal from us!
Yes. Support for HTTP proxies is provided (since Scrapy 0.8) through the HTTP Proxy downloader middleware. See HttpProxyMiddleware.
You need to install pywin32 because of this Twisted bug.
See Using FormRequest.from_response() to simulate a user login.
Yes, there’s a setting for that: SCHEDULER_ORDER.
Yes, see HttpAuthMiddleware.
Try changing the default Accept-Language request header by overriding the DEFAULT_REQUEST_HEADERS setting.
Scrapy comes with a built-in, fully functional project to scrape the Google Directory. You can find it in the examples/googledir directory of the Scrapy distribution.
Also, there is a public repository of spiders called Community Spiders.
Finally, you can find some example code for performing not-so-trivial tasks in the Scrapy Recipes page.
Yes. You can use the runspider command. For example, if you have a spider written in a my_spider.py file you can run it with:
scrapy-ctl.py runspider my_spider.py
Those messages (logged with DEBUG level) don’t necessarily mean there is a problem, so you may not need to fix them.
Those message are thrown by the Offsite Spider Middleware, which is a spider middleware (enabled by default) whose purpose is to filter out requests to domains outside the ones covered by the spider.
For more info see: OffsiteMiddleware.
There’s a whole documentation section about this subject, please see: Debugging memory leaks.
Also, Python has a builtin memory leak issue which is described in Leaks without leaks.