Examples of Logging with Python Logging

by Alex
Examples of Logging with Python Logging

While running a project in python, you may get to a place where even a debugger can’t find a bug. At that point, you will realize that creating a log file with a record of the program’s actions is really useful. Once you have a working version of the program, you need to understand what happens when you run it next.

The easiest (note: not the best) way to do this is to use lots of output statements(print) in your code. This is not a good idea because you will get a lot of output in the console, and you will probably delete the instructions for output as soon as you fix the bug.

How do you track errors correctly? Logging!

Python has a built-in logging library, a great tool for recording program actions to a file. This article for “logging” looks at a lot of examples from basic and advanced uses of the library.

Example #1 – A single file application

The first example is a simple program that helps illustrate the basic features of the logging module. This program consists of a single file named app.py that contains a single class:

class FirstClass:
   def __init__(self):
        self.current_number = 0
        
   def increment_number(self):
        self.current_number += 1
        
   def decrement_number(self):
        self.current_number -= 1
        
   def clear_number(self):
        self.current_number = 0


number = FirstClass()
number.increment_number()
number.increment_number()
print("Current value: %s" % str(number.current_number))
number.clear_number()
print("Current value: %s" % str(number.current_number))

You can run this program by running:

$ python app.py
Current value: 2
Current value: 0

This program uses two statements of output to the console. This is normal for the program to work, and switching to message logging is the best long-term approach.

Configuring the logging module

Configuring the logging module can seem complicated as you start to specify more and more information about how to log. The following sequence provides a good configuration for:

  • set severity level of logging
  • specify the file for logging messages
  • message format settings

Modify the FirstClass class constructor to configure the logging module:

def __init__(self):
    self.current_number = 0
 
   # Create Logger
   self.logger = logging.getLogger(__name__)
    self.logger.setLevel(logging.WARNING)
 
   # Create a handler to write the data into the
   logger_handler = logging.FileHandler('python_logging.log')
    logger_handler.setLevel(logging.WARNING)
 
   # Create a Formatter to format log messages
   logger_formatter = logging.Formatter('%(name)s - %(levelname)s - %(message)s')
 
   # Add Formatter to the
   logger_handler.setFormatter(logger_formatter)
 
   # Add the handler to the logger
   self.logger.addHandler(logger_handler)
    self.logger.info('Logging setup is over!')

Don’t forget to add import logging

The constructor configures the use of the logging module and ends with the entry that the configuration is complete.

Adding logging messages

To add log messages, you can use one of the methods of the python logging module to output logs:

def increment_number(self):
    self.current_number += 1
    self.logger.warning('increment number!')
    self.logger.info('The number is still increasing!!!')

def decrement_number(self):
    self.current_number -= 1

def clear_number(self):
    self.current_number = 0
    self.logger.warning('value clear!')
    self.self-logger.info('Value not cleared yet!!!')

If you run the program again, you will still see the same console output.

$ python app.py
Current value: 2
Current value: 0

The severity level of the logging

To find out what the logging module is doing, check the log file that was created:

$ cat python_logging.log
__main__ - WARNING - Number increasing!  
__main__ - WARNING - The number is increasing!  
__main__ - WARNING - Clearing value!

I wonder if this is what you were expecting? There should have been two more entries, “The number is still increasing!!!”, but they didn’t show up. Why? Well, the thing is that the logging module has 5 levels of severity:

  • DEBUG (lowest)
  • INFO
  • WARNING
  • ERROR
  • CRITICAL (highest)

Our program uses the default setting(WARNING) for logging severity, which means that any log message with less than WARNING severity will not be displayed. Consequently, INFO messages are also not displayed. Change the following lines in __init__():

# Create Logger
self.logger = logging.getLogger(__name__)
self.logger.setLevel(logging.INFO)
# Create a handler to write the data into the
logger_handler = logging.FileHandler('python_logging.log')
logger_handler.setLevel(logging.INFO)

Now check the log file, INFO level messages are displayed:

$ cat python_logging.log
__main__ - WARNING - Number increasing!  
__main__ - WARNING - Number is incrementing!  
__main__ - WARNING - Clearing value!  
__main__ - INFO - Logging setup is over!  
__main__ - WARNING - The number is increasing!  
__main__ - INFO - The number is still increasing!!!  
__main__ - WARNING - The number is going up!  
__main__ - INFO - The number is still growing!!!  
__main__ - WARNING - Clearing value!  
__main__ - INFO - The value has not yet been cleared!!!

Example #2 – Logging in a module

The second example is a bit more complicated because it updates the program structure to include a single module package:

python_logging
    python_logging
        __init__.py
        first_class.py
    app.

The first_class.py file contains the FirstClass class that was created in the first example:

import logging


class FirstClass 
   def __init__(self) 
        self.current_number = 0 
  
 # Create a Logger  
  self.logger = logging.getLogger(__name__) 
        self.logger.setLevel(logging.INFO) 
  
       # Create a handler to write the data into the  
  logger_handler = logging.FileHandler('python_logging.log') 
        logger_handler.setLevel(logging.INFO) 
  
       # Create a Formatter to format messages in the log  
  logger_formatter = logging.Formatter('%(name)s - %(levelname)s - %(message)s') 
  
       # Add Formatter to the  
  logger_handler.setFormatter(logger_formatter) 
  
       # Add the handler to the logger  
  self.logger.addHandler(logger_handler) 
        self.logger.info('Logging setup is over!') 
  
   def increment_number(self) 
        self.current_number += 1 
  self.logger.warning('increment number!') 
        self.logger.info('The number is still increasing!!!') 
  
   def decrement_number(self) 
        self.current_number -= 1 
  
 def clear_number(self) 
        self.current_number = 0 
  self.logger.warning('value clear!') 
        self.self-logger.info('Value not cleared yet!!!')

To use this module, update the app.py file in the top-level directory and import the FirstClass class to use it:

from python_logging.first_class import FirstClass


number = FirstClass()
number.increment_number()
number.increment_number()
print("Current value: %s" % str(number.current_number)) 
number.clear_number() 
print("Current value: %s" % str(number.current_number))

Run app.py and check the log file:

$ python app.py
Current value: 2
Current value: 0
$ cat python_logging.log
...
__main__ - WARNING - Clearing value!  
__main__ - INFO - The value has not been cleared yet!!!
python_logging - INFO - Logging setup is over!  
python_logging.first_class - WARNING - The number is increasing!  
python_logging.first_class - INFO - The number is still increasing!!!  
python_logging.first_class - WARNING - The number is increasing!  
python_logging.first_class - INFO - Number still growing!!!  
python_logging.first_class - WARNING - Clearing value!  
python_logging.first_class - INFO - Value not yet cleared!!!

Example #3: Logging in packages

The third example adds a second class to the python_logging package to show how to configure the logging module for the entire package. Here is the structure of this example:

python_logging
    python_logging
        __init__.py
        first_class.py
        second_class.py
    app.py

Here is the base version of the second_class.py file:

class SecondClass:
   def __init__(self):
        self.enabled = False
 
   def enable_system(self):
        self.enabled = True
 
   def disable_system(self):
        self.enabled = False

You can duplicate the logger settings in the constructor of this class (a copy from first_class.py). This would lead to a lot of unnecessary repetitive code. It is best to move the logger setting to the __init__.py file:

from os import path, remove  
import logging  
import logging.config  
  
from .first_class import FirstClass  
from .second_class import SecondClass  
  
  
# Delete the existing log file, if there is one, to create a new one at each execution  
 if path.isfile("python_logging.log") 
    remove("python_logging.log") 
  
# Create a Logger  
 logger = logging.getLogger(__name__) 
logger.setLevel(logging.INFO) 
  
# Create a handler to write data into the  
 logger_handler = logging.FileHandler('python_logging.log') 
logger_handler.setLevel(logging.INFO) 
  
# Create a Formatter to format messages in the log  
 logger_formatter = logging.Formatter('%(name)s - %(levelname)s - %(message)s') 
  
# Add Formatter to the  
 logger_handler.setFormatter(logger_formatter) 
  
# Add a handler to the logger  
 logger.addHandler(logger_handler) 
logger.info('Logging setup is over!')

Depending on the type of program you are creating, you may want to remove any existing log files before writing any new messages while that program is running. One option, if you want to keep the current application log, is to use the RotatingFileHandler from the Logging module. Now that the logging module is set up in __init__.py, the second_class.py file can be greatly simplified and you don’t have to worry about setting it up:

import logging


class SecondClass(object) 
   def __init__(self) 
        self.enabled = False 
        self.logger = logging.getLogger(__name__) 
  
   def enable_system(self) 
        self.enabled = True 
        self.logger.warning('enable system!') 
        self.logger.info('The system is still enabled!!') 
  
   def disable_system(self) 
        self.enabled = False 
        self.logger.warning('The system is down!') 
        self.logger.info('The system is still shutting down!!')

In the same way, we need to simplify def __init__() in first_class.py:

def __init__(self) 
    self.current_number = 0 
    self.logger = logging.getLogger(__name__)

Finally, updates to __init__.py lead to the need for the following updates in app.py:

from python_logging import FirstClass, SecondClass  
  
  
number = FirstClass() 
number.increment_number() 
number.increment_number() 
print("Current value: %s" % str(number.current_number)) 
number.clear_number() 
print("Current value: %s" % str(number.current_number)) 
  
system = SecondClass() 
system.enable_system() 
system.disable_system() 
print("Current state of the system: %s" % str(system.enabled))

Try running the program again and look at the log file:

$ cat python_logging.log
python_logging - INFO - Logging setup is over!  
python_logging.first_class - WARNING - The number is increasing!  
python_logging.first_class - INFO - The number is still increasing!!!  
python_logging.first_class - WARNING - The number is increasing!  
python_logging.first_class - INFO - Number still increasing!!!  
python_logging.first_class - WARNING - Clearing value!  
python_logging.first_class - INFO - Value not yet cleared!!!  
python_logging.second_class - WARNING - System on!  
python_logging.second_class - INFO - System is still on!!!  
python_logging.second_class - WARNING - System shutdown!  
python_logging.second_class - INFO - System is still turning off!!!

Notice how the module names are displayed! This is a very handy feature that allows you to quickly find out where specific operations are taking place.

Example #4: Logging in JSON packages

The fourth (and final) example expands on logging by adding an input file(JSON) to customize the logger. The first change for this example is in __init__.py to change the logger settings to use the JSON input file:

from os import path, remove  
import logging  
import logging.config  
import json  
  
from .first_class import FirstClass  
from .second_class import SecondClass  
  
  
# Delete the existing log file, if there is one, to create a new one at each execution  
 if path.isfile("python_logging.log") 
    remove("python_logging.log") 
  
with open("python_logging_configuration.json", 'r') as logging_configuration_file  
    config_dict = json.load(logging_configuration_file) 
  
logging.config.dictConfig(config_dict) 
  
# Record that the logger is configured  
 logger = logging.getLogger(__name__) 
logger.info('Logging setup is over!')

Now that we have the code to handle the input file, let’s define the input file (python_logging_configuration.json). Be sure to add this file to the top-level folder so that it can be easily identified by the python interpreter. You must run this program from the top-level folder to make the JSON input file available to the python interpreter because it is in the current / working directory. Here is the configuration file:

{ 
  "version": 1,  
  "disable_existing_loggers": false,  
  { "formatters": { 
    { "simple": { 
      { "format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s"  
  } 
  },  
  { "handlers": { 
    { "file_handler": { 
      { "class": "logging.FileHandler",  
  { "level": { "DEBUG",  
  { "formatter": { "simple",  
  "filename": "python_logging.log",  
  "encoding": "utf8"  
  } 
  },  
  { "root": { 
    { "level": { "DEBUG",  
  { "handlers": { "file_handler"]  
  } 
}

Run the program again and look at the log file:

$ cat python_logging.log
2019-01-05 20:13:40,636 - python_logging - INFO - Logging setup is over!  
2019-01-05 20:13:40,636 - python_logging.first_class - WARNING - The number is increasing!  
2019-01-05 20:13:40,636 - python_logging.first_class - INFO - The number is still increasing!!!  
2019-01-05 20:13:40,636 - python_logging.first_class - WARNING - The number is increasing!  
2019-01-05 20:13:40,636 - python_logging.first_class - INFO - The number is still increasing!!!  
2019-01-05 20:13:40,636 - python_logging.first_class - WARNING - Clearing value!  
2019-01-05 20:13:40,636 - python_logging.first_class - INFO - Value not yet cleared!!!  
2019-01-05 20:13:40,636 - python_logging.second_class - WARNING - System on!  
2019-01-05 20:13:40,636 - python_logging.second_class - INFO - System is still on!!!  
2019-01-05 20:13:40,636 - python_logging.second_class - WARNING - System shutdown!  
2019-01-05 20:13:40,636 - python_logging.second_class - INFO - System is still turning off!!!

The log file is almost the same, but a date and time line has been added to each message. This logging message format is very convenient: date / time – package.module – log message If you don’t like working with JSON, the input file can also be defined as a YAML file. Here is an example: Good Logging Practice in Python

The idea of going from output statements to actually logging messages is made simple by the logging module, which is built into python. The logging module requires some configuration, but it is a small price to pay for such an easy-to-use module. Having a log file for a program, especially command line applications, gives a great way to understand what the program is doing.

Related Posts

LEAVE A COMMENT