Python Logging Tutorial

Submitted by nanodano on Sat, 09/07/2019 - 17:14

Introduction

This is a simple guide to Python core logging package basics. The Python logging package is very powerful and widely used. For example, Django uses Python's built-in logging package.

For the most in-depth and up-to-date information, always refer to the official Python logging documentation. This guide will walk through a summary of the things I think are most important and useful in my regular work.

Simplest example

Here is a simple example of using the Python logging module.

# hello_logging.py
import logging

print('Printing to STDOUT')
logging.error('Logging goes to STDERR')

By default, the logs to go STDERR not STDOUT. This means text will typically go to your terminal output and look like normal STDOUT print statements, unless you specifically pipe the STDERR stream somewher else. For example, to redirect the STDERR output to a file, you can run the application like this:

# Both outputs will go to terminal by default
python hello_logging.py

# Or you can redirect STDERR (file id 2 to the OS)
# to a file like `debug.log`
# This will work in Windows, Mac, and Linux
python hello_logging.py 2>debug.log

You could also redirect one stream to another, for example, redirecting STDERR to STDOUT before redirecting STDOUT to a file. This way both outputs will go to a single source. Learn more in my STDOUT, STDERR, Piping, and Redirecting Tutorial.

Different log levels

Let's say you want to log some information, but it's not an error. Then you wouldn't want to call logger.error() because it sets the log level too high.

To demonstrate the concept of log levels, try a similar example to the first one, but this time, instead of calling logger.error() also call logger.info(), logger.warn(), logger.critical() and logger.debug(). They are called in order of their log level, with debug being the lowest and critical being the highest.

import logging

# Default log level is WARNING
logging.debug('This is a debug log')          # Won't show
logging.info('This is an informational log')  # Won't show
logging.warning('This is a warn log')
logging.error('This is a error log')
logging.critical('This is a critical log')

In order to get the debug level messages to print, you need to configure the logger to have a different log level. To configure the global logger, use basicConfig() like this:

import logging

# Configure global logger to output DEBUG and higher level
logging.basicConfig(level=logging.DEBUG)

logging.debug('This is a debug log')
logging.info('This is an informational log')
logging.warning('This is a warn log')
logging.error('This is a error log')
logging.critical('This is a critical log')

Logger configuration

In the previous example we looked at how to set the log level using basicConfig(). The basicConfig() method takes many other arguments to configure the logger. For example, you can configure:

Log level
Text and date formatting
File name (optional) and mode (append vs create)
Output stream
Handlers

Note that you can only choose one of the output methods: file, output stream, or handlers. You can't have file and stream output at the same time, and you can't use handlers with any others. So the real important settings are:

Log level
Formatting
Where you output the message (file, stream, or handler)

Log levels

As demonstrated in the previous example, log level can be set using the basicConfig() method on the logger. Specify one of the levels:

DEBUG
INFO
WARNING
ERROR
CRITICAL

import logging

logging.basicConfig(level=logging.DEBUG)

Customize log formatting

There are two strings you can format with the logger, the output message and the date. The default format does not include the time, but outputs the log level, the logger name, and the log message itself. You might want to exclude the logger name if you only have one global logger, and you may want to include the time or other attributes.

Using basicConfig() you can specify a format string for both the message format and the date format.

You can review the fill list of log formatting variables, but there are some important ones that are worth noting:

asctime - a string with the log message time
levelname - a string with log level name
message - the actual log message string
module - the string module name where the log message originated
filename - the string file name where the log message originated
lineno - the integer line number where log message originated

A format string would look something like this:

%(asctime)s %(lineno)d %(levelname)s %(message)s

You can use it in the configuration like this:

import logging

logging.basicConfig(format='%(asctime)s %(levelname)s %(message)s')

logging.error('Test error message')

To alter the date formatting, also pass in a datefmt option like this:

import logging

logging.basicConfig(format='%(asctime)s %(message)s',
                    datefmt='%Y-%m-%d %H:%M:%S')

logging.error('Test error message')

Refer to the strftime() documentation for a list of the available variables for date formatting.

Output to file

To output to a file, you can use basicConfig() to specify a file like this:

import logging

logging.basicConfig(filename='debug.log')

By default, it will open the file in append mode (a). You can specify the open mode, for example to create a new file each time:

import logging

logging.basicConfig(filename='debug.log', filemode='w')

Note that you cannot use file and stream or handler output at the same time. You must pick one or the other.

Output to stream

You can output all log messages to an in-memory file stream if desired. For more information on using the streams, check out my tutorial Python Use StringIO to Capture STDOUT and STDERR.

This example will output the log information to a text stream that you can read from later.

import logging
from io import StringIO

text_stream = StringIO()
logging.basicConfig(stream=text_stream, level=logging.DEBUG)

logging.debug('Testing output')
logging.debug('to stream')

print("Log contents:")
print(text_stream.getvalue())

Note that you cannot use file and stream or handler output at the same time. You must pick one.

Use handlers

The previous examples used either a stream or a file output, and you had to pick only one. Starting in Python 3.3 they introduces a new option to use handlers. You can't use handlers with file or stream output so when you want to output to multiple things, handlers are the only way to go.

There are over 15 built-in handlers that you can use. Check out the full list of useful handlers. Some noteworthy handlers are:

logging.StreamHandler
logging.FileHandler
logging.handlers.RotatingFileHandler
logging.handlers.SocketHandler
logging.handlers.SysLogHandler
logging.handlers.SMTPHandler
logging.handlers.HTTPHandler

As you can see, you have the basic stream and file handlers like were used in the previous examples, but you also have more advanced handlers like a file handler that includes log rotation, a syslog handler to send syslog events, and SMTP and HTTP handlers to send log events over email or HTTP. You can also create custom handlers.

You pass an iteratable of handler objects to the configuration.

import logging
from logging import FileHandler, StreamHandler
from logging.handlers import RotatingFileHandler

handlers = [
    FileHandler('log.txt'),  # Default mode='a', encoding=None
    RotatingFileHandler('log-rotating.txt', maxBytes=10000, backupCount=3),
    StreamHandler(),  # Default stream=sys.stderr
]

logging.basicConfig(handlers=handlers)

logging.error('This goes to stderr, log.txt, and the rotating log')

You can also add handlers by calling .addHandler() on the logger object.

Create multiple loggers

In all of the previous examples we directly called and configured the global logger logging. You may want to configure separate or multiple loggers.

You can get the root (global) logger by calling logging.getLogger() with no arguments. You can create or fetch unique loggers by providing the name of the logger you want. A common convention is to use __name__, which is the name of the Python file, as the logger name. You can also choose to use a string like "global" or any other arbitrary name.

import logging

logger = logging.getLogger(__name__)
# Configure or alter the logger if neeeded
# logger.basicConfig()
# logger.addHandler()

logger.error('This is a simple error log')

Conclusion

After reading this guide, you should know how to use Python's default logging package to do simple logging tasks like outputting to STDERR, controlling log levels, outputting to a file, and customizing the log formatting. You should also know how to use handlers to log via email, syslog, http, and rotate log files.