Send data via Puppet Metrics to your Logstash instance provided by Logit.io

Puppet Metrics

Ship Puppet Server Metrics using Metricbeat to Logstash

Metricbeat is a lightweight shipper that helps you monitor Puppet performance metrics by collecting metrics running on the Puppet Agents. Configure Metricbeat using the pre-defined examples below to collect and ship Puppet metrics to Logstash or Elasticsearch.

Step 1 - Requirements

In order for puppet metrics to be shipped correctly you have to have the following server pre-requisites.

  • Puppet Server 5.0 or above, tested on version 6
  • Jolokia agent installed
  • Tested on Ubuntu server

Step 2 - Install Jolokia Agent

Download Tomcat 7 from using the link below.

https://tomcat.apache.org/download-70.cgi

Then extract it.

tar zxvf apache-tomcat-7.*.tar.gz 
cd apache-tomcat-7.*
TC=`pwd`

Download the Jolokia WAR-agent and copy it into Tomcat's webapp directory.

cd $TC/webapps
wget https://repo1.maven.org/maven2/org/jolokia/jolokia-war/1.2.3/jolokia-war-1.2.3.war
mv jolokia-war-1.2.3.war jolokia.war

Step 3 - Verify Jolokia Installation

Now the container can be started with.

$TC/bin/catalina.sh start

In order to verify the installation was successful, point your browser to

http://localhost:8080/jolokia/version

Your response should have status 200 in the json data.

Step 4 - Configure Jolokia

To change the security access policy for Jolokia.

Create the /etc/puppetlabs/puppetserver/jolokia-access.xml file with contents below that follow the Jolokia access policy. Uncomment the metrics.metrics-webservice.jolokia.servlet-init-params.policyLocation parameter the restart the puppetserver.

<?xml version="1.0" encoding="utf-8"?>
<restrict>
  <remote>
    <host>${PUPPET-HOST-SERVER}</host>
  </remote>
  <http>
    <method>post</method>
  </http>
  <commands>
    <command>read</command>
    <command>list</command>
  </commands>
  <allow>
    <mbean>
      <name>java.lang:type=Memory</name>
      <operation>gc</operation>
    </mbean>
  </allow>
  <deny>
    <mbean>
      <name>com.mchange.v2.c3p0:type=PooledDataSource,*</name>
      <attribute>properties</attribute>
    </mbean>
  </deny>
</restrict>

Step 5 - Install Metricbeat

deb (Debian/Ubuntu/Mint)

curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-oss-7.6.2-amd64.deb
sudo dpkg -i metricbeat-oss-7.6.2-amd64.deb

rpm (CentOS/RHEL/Fedora)

curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-oss-7.6.2-x86_64.rpm
sudo rpm -vi metricbeat-oss-7.6.2-x86_64.rpm

Step 6 - Locate configuration file

/etc/metricbeat/metricbeat.yml

Step 7 - Configure output

We'll be shipping to your hosted Logstash so that we have the option to apply filters before the data is indexed into Elasticsearch.
Comment out the elasticsearch output block.

## Comment out elasticsearch output
#output.elasticsearch:
#  hosts: ["localhost:9200"]

Uncomment and change the Logstash output to match below.

output.logstash:
    hosts: ["your-logstash-host:your-port"]
    loadbalance: true
    ssl.enabled: true

Step 8 - Enable module

There are several built in Metricbeat modules that you can use. To enable the Jolokia module, run the following:

metricbeat modules list
metricbeat modules enable jolokia

Step 9 - Configure module

Each module has its own configuration file where different metricsets can be enabled / disabled. Locate the configuration file for the jolokia module.

/etc/metricbeat/modules.d/jolokia.yml

By default the following metricsets are disabled. To enable or disable a metric simply comment or uncomment the line out.

- module: jolokia
  metricsets: ["jmx"]
  enabled: true
  period: 10s
  hosts: ["localhost:8080"]
  namespace: "jolokia_metrics"
  path: "/jolokia"
  http_method: 'GET'
  jmx.mappings:
    - mbean: 'java.lang:type=Memory'
      attributes:
      - attr: HeapMemoryUsage
        field: memory.heap_usage
      - attr: NonHeapMemoryUsage
        field: memory.non_heap_usage
    - mbean: 'Catalina:name=*,type=ThreadPool'
      attributes:
      - attr: port
        field: catalina.port
      - attr: maxConnections
        field: catalina.maxConnections
    - mbean: 'java.lang:type=Runtime'
      attributes:
      - attr: Uptime
        field: uptime

Step 10 - Start Metricbeat

Ok, time to start gathering metrics!

sudo systemctl enable metricbeat
sudo systemctl start metricbeat

Step 11 - Puppet Logs Overview

Puppet is a leading configuration management (CM) system that makes the process of automating common tasks for DevOps & Sysadmins manageable. The platform can also be used to eliminate the need for manual configuration in order to deploy servers automatically.

This IT automation software can manage many of the common operations for maintaining system infrastructure including patching, provisioning, and management of operating systems & servers across on-premise and cloud environments.

Puppet are technology partners with many of the biggest leaders in the cloud computing space, including Microsoft Azure & Amazon Web Services. They work with over 30,000 companies globally helping them to secure their infrastructure, including Red Hat, Cisco, & Salesforce.

Puppet has been deployed across especially large infrastructures managing in excess of 5000+ machines. As Puppet is built for dynamically scaling and centralising machine activity, their platform works hand in hand with Logit to scale your infrastructure and centralise your data. Logit provides you with a scalable alternative for inspecting Puppet metrics data fast. Logit also enables you to visualise key performance metrics for Puppet using our hosted Kibana integration.

Puppet metrics include key insights on the number of changes the application has made, resources managed, and the type of events that have occurred. Each of the metric’s log messages has a name, label, and values that make it easy to parse when using Logstash to transform your files.

If you need any additional assistance with migrating your Puppet metric logs we're here to help you. Feel free to reach out by contacting our support team through our Microsoft Azure & Amazon Web Services. They work with over 30,000 companies globally helping them to secure their infrastructure, including Help Centre or live chat & we'll be happy to assist.

expand view

Expand View

compact view

Compact View

Return to Search
Sign Up