Azure Eventhub Diagnostic Logs

Pull diagnostic logs from Azure Event Hub to Logstash

Step 1 - Prerequisites

Before you begin you will need to ensure you have an available Azure Event Hub in your Azure Portal.

Confirm you have the following:

An Eventhub you wish to get diagnostic Logs from.
A separate Eventhub you wish to stream the diagnostic logs to.

The Azure event hub Logstash plugin is only available for stacks running Logstash 6.4 onwards

Step 2 - Configure Diagnostic Logs

Once you have confirmed you have everything required it's time to configure diagnostic logging.

In the Eventhub Namespace you wish to get diagnostic logs from you need to browse to the Diagnostic Settings from the left hand menu.

Diagnostic settings

Go ahead and choose add diagnostic setting, enter a suitable name and then select the diagnostic logs you need.

Select stream to an Eventhub and enter the details of the second Eventhub.

Once you're happy with the settings select save.

Save changes

Step 3 - Configure permissions

Once you have data streaming to your Azure event hub, it is recommended to create a Consumer Group specifically for Logstash and not to reuse any default or existing groups.

The Logstash input supports multiple event hubs - the connection string for each hub can be found in the Azure Portal -> Event Hub -> Shared access policies.

example connection string
Endpoint=sb://<youreventhubnamespace>.servicebus.windows.net/;SharedAccessKeyName=<yoursharedaccesspolicyname>;SharedAccessKey=<yoursharedaccesskey>;EntityPath=<youreventhubname>

A blob storage account is used to preserve state across logstash reboots. The Storage account connection string can be found in the Access Keys section under the Storage Account Settings menu in the Azure Portal

example connection string

DefaultEndpointsProtocol=https;AccountName=<storage-account-name>;
AccountKey=<storage-account-key>;
EndpointSuffix=core.windows.net

Step 4 - Start Sending Logs to a Stack

To start pulling logs and metrics from the Azure Event Hub to your Stack you need to configure an Azure Logstash Input on your Logit.io Stack.

Go to Dashboard

Logit.io will verify your input before it is applied, we will contact you to confirm when this has been completed.

Step 5 - Check Logit.io for your logs

Now you should view your logs:

Launch Dashboard

If you don't see logs take a look at How to diagnose no data in Stack below for how to diagnose common issues.

Step 6 - How to diagnose no data in Stack

If you don't see data appearing in your Stack after following the steps, visit the Help Centre guide for steps to diagnose no data appearing in your Stack or Chat to support now.

Step 7 - Azure Log Monitoring Overview

As one of the most popular cloud computing platforms, Microsoft Azure provides businesses with the capabilities to run databases, servers & many other services virtually.

Azure Event Hub is a platform built for streaming & ingesting big data. The Microsoft platform is able to ingest millions of events per second, so having a log management service for transforming and tailing events is necessary for a thorough analysis of log messages.

As a disaster recovery tool, Azure is known for its advanced site recovery which is operational across all operating systems, locations, & languages. The platform also allows users to customise the frequency of backup schedules & works alongside the Logit.io platform as our hosted ELK Stacks support version control & rollbacks for added peace of mind.

Our platform is built to simplify Azure log management as we provide your hybrid cloud & multi-cloud Azure environments with the ability to parse & monitor your activity across hybrid environments & audit logs to bring data-driven insights that are valuable for a wide variety of use cases.

Logit.io’s hosted Elastic Stack allows you to aggregate and analyze all of your logs centrally, saving your InfoSec & DevOps teams from unnecessary maintenance overheads by centralising all of your Azure monitor logs.

Tail large volumes of logs & gain near real-time insights on the performance and security issues your infrastructure faces using our cloud-based platform. Logit.io also offers automated parsing through highly available Logstash pipelines as a solution for migrating your log data fast.

If you need any further assistance with migrating your Azure data to Logstash we're here to help you discover the insights locked in data hosted on Azure, GCP, AWS or any of the integrations covered in our data sources. Feel free to reach out by contacting our support team through our Help Centre or live chat & we'll be happy to assist.

Compact View