Ready to get going? Start your 14 days free trial today

Start free trial

Have an account? Sign in

Send data via Python to your Logstash instance provided by Logit.io

Logstash Python Configuration Example

Ship logs from Python applications to logstash

This guide has been setup and tested using both the Python 2.7 (64-bit) and Python 3.6 (64-bit) environments. Code samples included can be used inside a basic Python script, or web framework such as Django or Flask. To send logs to logstash through Python you will need the python-logstash-async (version 1.4.1) Python plugin.

Step 1 - InstallingCopy

Execute the following command using pip:

pip install python-logstash-async
If you receive errors suggesting pip is an unknown command, navigate to its installed location with your chosen terminal program and try again (or add pip to your environmental variables if using Windows).

Step 2 - Sending LogsCopy

Add your imports to the top of your Python file (the file you wish to include logging on):

import logging
from logstash_async.handler import AsynchronousLogstashHandler
from logstash_async.handler import LogstashFormatter

Add this code somewhere inside the same file (for example, this could be before Django or Flask renders a view):

# Create the logger and set it's logging level
logger = logging.getLogger("logstash")
logger.setLevel(logging.ERROR)        

# Create the handler
handler = AsynchronousLogstashHandler(
    host='your-logstash-host', 
    port=your-ssl-port, 
    ssl_enable=True, 
    ssl_verify=False,
    database_path='')
# Here you can specify additional formatting on your log record/message
formatter = LogstashFormatter()
handler.setFormatter(formatter)

# Assign handler to the logger
logger.addHandler(handler)

# Send log records to Logstash 
logger.error('python-logstash-async: test error message.')
logger.info('python-logstash-async: test info message.')
logger.warning('python-logstash-async: test warning message.')
logger.debug('python-logstash-async: test debug message.')

Step 3 - Python Log OverviewCopy

Python’s logging module is used by most third-party libraries and allows you to integrate your log messages with application-specific logs.

The five standard levels of logs aggregated by the Python module include; CRITICAL, ERROR, WARNING, INFO, DEBUG.

  • Critical logs let you know that the application is shutting down and can’t recover without a reboot.
  • Error logs capture unexpected bugs that can be handled without a reboot, for example, a 503 or 404 error.
  • Warnings can show that resources are running low or that suspicious web requests have been identified. Warning logs can be monitored to protect the security of your applications.
  • Info is log data that is sent to your ELK Stack that has captured information on your application’s behaviour metrics.

The creation of a single Python script can produce thousands of logs which only increase once migrated to a production environment. Each logging level adds complexity to being able to efficiently report on the development process, as well as increasing the risk of breaking points across the module.

Using the Logit platform allows you to centralise all five types of Python logs for the streamlining of monitoring, troubleshooting and debugging in the cloud. Our onboarding has your ELK Stacks ready for analysis of Python logging events in minutes by using our highly available hosted Logstash pipeline for collecting log files.

Using Logit you can view all of the five log types for easy analysis with our out of the box Kibana dashboards. Set alerts on errors, warnings and mission-critical Python issues from one centralised dashboard. Logit integrates with Slack, Webhooks and PagerDuty to send notification messages to your Dev teams.

If you need any further assistance with migrating your Python application logs to Logstash we're here to help you get started. Just get in contact with our support team by visiting our dedicated Help Centre or via live chat & we'll be happy to assist.

Toggle View

Expand View

Return to Search