Tutorial on Python Logging
Python logging and code snippets for you to use right away
Logging is a very important functionality for a programmer. For both debugging and displaying run-time information, logging is equally useful. In this article, I will present why and how you could use the python’s logging module in your programs.
Why Logging and not print()
There is a key difference between a print statement and logging output. Usually, print statements write to stdout (standard output) which is expected to be the useful information or the output of the program. However, logs are written into stderr (standard error). We can demonstrate this scenario as follows.
import logginglogging.basicConfig(level=logging.INFO) #We'll talk about this soon!logging.warning('Something bad could happen!')
logging.info('You are running the program')
logging.error('Aw snap! Everything failed.')print("This is the program output")
Now if I run this program, I will see the following in the command line.
$ python log_test.py
WARNING:root:Something bad could happen!
INFO:root:You are running the program
ERROR:root:Aw snap! Everything failed.
This is the program outputHowever, for the usual user, the information is too much. Though this is actually displayed all together in the command line the data is written into two separate streams. So a typical user should do the following.
$ python log_test.py > program_output.txt
WARNING:root:Something bad could happen!
INFO:root:You are running the program
ERROR:root:Aw snap! Everything failed.$ cat program_output.txt
This is the program output
Here the useful program output is written to a file, by redirection >. So we can see what's happening on the terminal and get the output conveniently on a file. Now let’s try to understand the log levels!
Logging and Log Levels
Logging can happen for different reasons. These reasons are separated into levels of severity as following.
- DEBUG: Debug information for developers such as computed values, estimated parameters, URLs, API calls, etc.
- INFO: Information, nothing serious.
- WARNING: Warnings to users about inputs, parameters, etc.
- ERROR: Reports an error caused by something that the user did or occurred within the program.
- CRITICAL: The highest priority log output. Used for critical concerns (Depends on the use-case).
The most common types of logs are DEBUG, INFO, and ERROR. However, you can easily end up with scenarios where python throws warnings for version mismatches.
Configuring the Logger and Log Handlers
The loggers can be configured under different parameters. The logger can be configured to follow a particular log level, a file name, file mode, and a format to print the log output.
Configuring the Logger Parameters
The logger can be configured as follows.
import logging
logging.basicConfig(filename='program.log', filemode='w', level=logging.DEBUG)
logging.warning('You are given a warning!')The above setting asks the logger to output the log into a file named program.log. The filemode=’w’ defines the nature of writing to file. For example, 'w' open a new file overwriting whatever that was there. By default, this parameter is 'a' which will open the log file in append mode. Sometimes it is useful to have a log history. The level parameter defines the lowest severity for logging. For example, if you set this to INFO, DEBUG logs will not be printed. You may have seen programs needs to be run inverbose=debug mode to see some parameters. By default the level is INFO.
Creating a Log Handler
Although the above approach is straightforward for a simple application we need a comprehensive logging process for a production-ready software or a service. This is because it might be quite difficult to find for a particular ERROR log amidst millions of DEBUG logs. Furthermore, we need to use a single logger throughout the program and modules. This way we’ll correctly append the logs to the same file. For this, we can use handlers with different configurations for this task.
import logginglogger = logging.getLogger("My Logger")
logger.setLevel(logging.DEBUG)console_handler = logging.StreamHandler()
file_handler = logging.FileHandler('file.log', mode='w')
console_handler.setLevel(logging.INFO)
file_handler.setLevel(logging.DEBUG)logger.addHandler(console_handler)
logger.addHandler(file_handler)
You can see that we first get a logger passing a name. This enables us to reuse the same logger everywhere else in the program. We set the global logging level to be DEBUG. This is the lowest log level, hence enables us to use any log level in other handlers.
Next, we create two handlers for console and file writing. For each handler, we provide a log level. This can help reduce the overhead on console output and transfer them to the file handler. Makes it easy to deal with debugs later.
Formatting the Log Output
Logging is not merely print our own message. Sometimes we need to print other information such as time, log level, and process ids. For this task, we can use log formatting. Let’s see the following code.
console_format = logging.Formatter('%(name)s - %(levelname)s - %(message)s')
file_format = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')console_handler.setFormatter(console_format)
file_handler.setFormatter(file_format)
Before we add handlers to the logger we can format the log outputs as above. There are many more parameters that you can use for this. You can find them all here.
A Code for Reuse
The following is a code snippet for logging that I continue to use in many of my applications. Thought it might be useful for you as the reader.
Loggin with Multithreading
The logging module is made with thread safety in mind. Hence no special action is required when you’re logging from different threads except very few exceptions (out of the main scope of this article).
I hope this is a simple but useful article for many budding programmers and engineers. The following are a few more python tutorials that you might like having a look at. Happy reading. Cheers! :-)
