Logging
Scrapy provides a logging facility which can be used through the
scrapy.log module. The current underling implementation uses Twisted
logging but this may change in the future.
Logging service must be explicitly started through the scrapy.log.start() function.
Log levels
Scrapy provides 5 logging levels:
- CRITICAL - for critical errors
- ERROR - for regular errors
- WARNING - for warning messages
- INFO - for informational messages
- DEBUG - for debugging messages
How to set the log level
You can set the log level using the –loglevel/-L command line option, or
using the LOG_LEVEL setting.
How to log messages
Here’s a quick example of how to log a message using the WARNING level:
from scrapy import log
log.msg("This is a warning", level=log.WARNING)
Logging from Spiders
The recommended way to log from spiders is by using the Spider
log() method, which already populates the
spider argument of the scrapy.log.msg() function. The other arguments
are passed directly to the msg() function.
scrapy.log module
-
scrapy.log.log_level
- The current log level being used
-
scrapy.log.started
- A boolean which is True is logging has been started or False otherwise.
-
scrapy.log.start(logfile=None, loglevel=None, logstdout=None)
Start the logging facility. This must be called before actually logging any
messages. Otherwise, messages logged before this call will get lost.
Parameters: |
- logfile (str) – the file path to use for logging output. If omitted, the
LOG_FILE setting will be used. If both are None, the log
will be sent to standard error.
- loglevel – the minimum logging level to log. Availables values are:
CRITICAL, ERROR, WARNING, INFO and
DEBUG.
- logstdout (boolean) – if True, all standard output (and error) of your
application will be logged instead. For example if you “print ‘hello’”
it will appear in the Scrapy log. If ommited, the LOG_STDOUT
setting will be used.
|
-
scrapy.log.msg(message, level=INFO, component=BOT_NAME, spider=None)
Log a message
Parameters: |
- message (str) – the message to log
- level – the log level for this message. See
Log levels.
- component (str) – the component to use for logging, it defaults to
BOT_NAME
- spider (BaseSpider object) – the spider to use for logging this message. This parameter
should always be used when logging things related to a particular
spider.
|
-
scrapy.log.exc(message, level=ERROR, component=BOT_NAME, spider=None)
Log an exception. Similar to msg() but it also appends a stack trace
report using traceback.format_exc.
It accepts the same parameters as the msg() function.
-
scrapy.log.CRITICAL
- Log level for critical errors
-
scrapy.log.ERROR
- Log level for errors
-
scrapy.log.WARNING
- Log level for warnings
-
scrapy.log.INFO
- Log level for informational messages (recommended level for production)
-
scrapy.log.DEBUG
- Log level for debugging messages (recommended level for development)