Python Logging in Jupyter Notebooks

Introduction

Logging is an essential part of software development, especially when building complex data pipelines, dashboards, or scientific notebooks. In a Jupyter Notebook environment, logging can be slightly different than in a standard Python script. This blog post aims to guide you through setting up and using Python’s built-in logging module in Jupyter Notebooks.

Why Logging in Jupyter Notebooks?

1. Debugging: Easier to debug errors and exceptions.
2. Monitoring: Keep track of variable values, data transformations, and function calls.
3. Audit Trail: Maintain a record of actions for compliance and review.

Setting Up Logging

Firstly, let’s import the logging module and configure it. The basic setup involves setting the logging level and format.

When you set up logging using logging.basicConfig(), the format parameter allows you to specify the layout of log messages. This is done through a format string that can contain various placeholders, encapsulated in percentage signs %. These placeholders get substituted with actual log record attributes when a log message is emitted.

Here are some commonly used placeholders:

• %(asctime)s: The time when the log record was created.
• %(levelname)s: The level of the log (DEBUG, INFO, etc.).
• %(message)s: The log message itself.
• %(name)s: The name of the logger.
• %(filename)s: The filename where the log call was made.
• %(lineno)d: The line number in the file where the log call was made.

Basic Logging

The most straightforward way to log is to use the logging levels provided by the Python logging module: DEBUG, INFO, WARNING, ERROR, and CRITICAL.

Certainly. Debug levels in Python’s logging module help you control the granularity of log output. Here’s a brief explanation of each:

• DEBUG: Provides detailed information for diagnostic purposes. Use this level to output everything, including data that might help diagnose issues or understand the flow of the application.
• INFO: Confirms that things are working as expected. Useful for general runtime confirmations and tracking the state of the application.
• WARNING: Indicates something unexpected happened or may happen soon, but the software is still functioning. Use this level to log events that might cause problems but are not necessarily errors.
• ERROR: Records errors that have occurred, affecting some functionality but not causing the program to terminate. Use this level to log severe issues that prevent certain operations from being carried out.
• CRITICAL: Logs severe errors that cause the program to terminate. Use this level for unrecoverable errors that stop the application from running.

Each level has a numeric value (DEBUG=10, INFO=20, WARNING=30, ERROR=40, CRITICAL=50). Setting the logging level to a particular value will capture all logs at that level and above. For example, setting the level to WARNING will capture WARNING, ERROR, and CRITICAL logs, but ignore DEBUG and INFO.

Tricks to Use Logging in Jupyter Notebooks

1. Logging in Functions and Classes

Logging can be particularly useful when encapsulated within functions or classes. This approach helps you track the execution flow and debug issues more effectively. When you create loggers within functions, you can include the function name in the logger to identify where messages originate from.

import logging

def process_data(data):
    logger = logging.getLogger(__name__)
    logger.info(f"Processing {len(data)} records")
    
    try:
        # Your processing logic here
        result = [x * 2 for x in data]
        logger.info("Data processing completed successfully")
        return result
    except Exception as e:
        logger.error(f"Error processing data: {str(e)}")
        raise

2. Custom Handlers and Formatters

In Jupyter Notebooks, you might want to display logs in the notebook itself rather than the console. You can achieve this by adding a custom handler. This is particularly useful when you want to see logs inline with your notebook output or when working in environments where console output isn’t easily accessible.

import logging
import sys

# Create a custom formatter
formatter = logging.Formatter(
    '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)

# Create a stream handler for notebook output
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(formatter)

# Configure the logger
logger = logging.getLogger('notebook_logger')
logger.setLevel(logging.INFO)
logger.addHandler(handler)

logger.info("This message will appear in the notebook cell output")

3. Logging Variable Data

To log variable data, you can use string formatting. This is essential for debugging as it allows you to track the state of variables throughout your notebook execution. Use f-strings, .format(), or % formatting to include dynamic data in your log messages.

import logging

logger = logging.getLogger(__name__)

# Example with different data types
user_id = 12345
processing_time = 2.34
status = "completed"

# Using f-strings (recommended)
logger.info(f"User {user_id} processed in {processing_time:.2f}s with status: {status}")

# Using .format() method
batch_id = "batch_001"
items = [1, 2, 3, 4, 5]
logger.info("Processing batch {} with {} items".format(batch_id, len(items)))

# Logging dictionary data
config = {"model": "RandomForest", "n_estimators": 100}
logger.info(f"Training model with config: {config}")

4. Avoid Duplicated Handlers

Running a cell multiple times can add duplicate handlers, leading to repeated log messages. This is a common issue in Jupyter Notebooks due to their interactive nature. Make sure to remove existing handlers before adding new ones to prevent message duplication.

import logging

# Get the logger
logger = logging.getLogger('my_logger')

# Remove all existing handlers
for handler in logger.handlers[:]:
    logger.removeHandler(handler)

# Now add your handler
handler = logging.StreamHandler()
formatter = logging.Formatter('%(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(logging.INFO)

logger.info("This message will only appear once, even if you run the cell multiple times")

5. Use Rich Output

You can use libraries like rich to make the log output more readable and colorful. Rich provides beautiful formatting, syntax highlighting, and structured output that makes logs much easier to read in Jupyter Notebooks.

# First install rich: !pip install rich
from rich.logging import RichHandler
import logging

# Configure logging with Rich
logging.basicConfig(
    level="INFO",
    format="%(message)s",
    datefmt="[%X]",
    handlers=[RichHandler(rich_tracebacks=True)]
)

logger = logging.getLogger("rich")
logger.info("Hello, [bold magenta]World[/bold magenta]!", extra={"markup": True})
logger.error("Something went wrong!")
logger.warning("This is a warning with [yellow]highlighted text[/yellow]", extra={"markup": True})

6. Dynamic Log Level Switching

You can dynamically change the log level without resetting the entire logger. This is useful for debugging specific cells or when you want to see more detailed output temporarily without modifying your entire logging configuration.

import logging

logger = logging.getLogger(__name__)
handler = logging.StreamHandler()
logger.addHandler(handler)

# Start with INFO level
logger.setLevel(logging.INFO)
logger.info("This INFO message will be shown")
logger.debug("This DEBUG message will NOT be shown")

# Switch to DEBUG for detailed troubleshooting
print("Switching to DEBUG level...")
logger.setLevel(logging.DEBUG)
logger.info("This INFO message will be shown")
logger.debug("This DEBUG message will NOW be shown")

# Switch back to WARNING to reduce noise
logger.setLevel(logging.WARNING)
logger.info("This INFO message will NOT be shown anymore")
logger.warning("This WARNING message will be shown")

7. Use Context Managers for Temporary Logging Levels

For temporary logging level changes, you can use a context manager to ensure the level reverts back after a specific block of code. This is perfect when you need detailed logging for a specific operation but don’t want to change the global logging level permanently.

import logging
from contextlib import contextmanager

@contextmanager
def log_level(logger, new_level):
    """Context manager to temporarily change logging level"""
    old_level = logger.level
    logger.setLevel(new_level)
    try:
        yield
    finally:
        logger.setLevel(old_level)

# Setup logger
logger = logging.getLogger(__name__)
handler = logging.StreamHandler()
logger.addHandler(handler)
logger.setLevel(logging.WARNING)  # Default level

logger.info("This won't be shown (below WARNING level)")

# Temporarily change to DEBUG level
with log_level(logger, logging.DEBUG):
    logger.debug("This DEBUG message will be shown")
    logger.info("This INFO message will be shown")
    # Perform detailed debugging here

# Back to WARNING level
logger.info("This won't be shown again (back to WARNING level)")
logger.warning("This WARNING will be shown")

Best Practices to Write Great Log Messages

Be Descriptive but Concise

Log messages should provide enough context to understand what’s happening but be concise enough to not overwhelm the reader. Use clear language that describes the action, state, or condition.

• 👍 Good: logging.info("Connection to database established.")
• 👎 Bad: logging.info("DB OK.")

Include Relevant Variables or Identifiers

When logging events, include any relevant variables, identifiers, or parameters that could be useful for debugging or auditing. Use string formatting to include these in the log message.

• 👍 Good: logging.info("User %s successfully authenticated.", user_id)
• 👎 Bad: logging.info("Authentication successful.")

Choose the Appropriate Log Level

Use the correct log level to indicate the severity or importance of the log message. This helps in filtering logs and understanding the system state quickly.

• DEBUG for detailed diagnostic information.
• INFO for confirmation of successful operations.
• WARNING for unexpected situations that don’t cause errors.
• ERROR for issues that disrupt normal functionality.
• CRITICAL for severe problems that cause program termination.

Use Consistent Formatting

Maintain a consistent format for your log messages. This makes it easier to search, filter, and analyze logs. Consistency should apply to the structure of the message, the terminology used, and even the tense.

• 👍 Good: logging.info("File uploaded: filename={}, size={}KB".format(file_name, file_size))
• 👎 Bad: logging.info("Uploaded file. Name of file is {}. Size is {} kilobytes.".format(file_name, file_size))

Avoid Logging Sensitive Information

Be cautious about the data you log. Never log sensitive information like passwords, API keys, or personally identifiable information (PII). This is crucial for security and compliance reasons.

• 👍 Good: logging.info("User {} requested password reset.".format(user_id))
• 👎 Bad: logging.info("User {} requested password reset. New password is {}.".format(user_id, new_password))

Caveats and Solutions in Jupyter Notebooks

1. Statefulness: Jupyter Notebooks are stateful, which means logging configurations persist across cells. Reset the kernel to clear configurations.
2. Multiple Handlers: Running a cell multiple times can add duplicate handlers. Make sure to remove existing handlers before adding new ones.
3. Kernel Restart Required for Global Changes: If you make global changes to the logging configuration and want them to take effect, you may need to restart the Jupyter Notebook kernel, which will also clear all your variables and imports.
4. Asynchronous Output: Jupyter Notebooks can sometimes produce asynchronous output, making logs appear out of order. This can be confusing when you’re trying to debug the sequence of events.

Conclusion

Logging in Jupyter Notebooks is a straightforward yet powerful way to monitor, debug, and audit your data applications. It becomes even more potent when used in a comprehensive platform like MINEO, where Python notebooks serve as the backbone for various data-centric tasks.

By incorporating logging into your notebooks, you can build more robust, maintainable, and transparent data apps.

Happy coding!