Step 1 - Confirm your logs are in google Stackdriver

Locate the logs you wish to export in google stackdriver

Step 2 - Export your logs to google Pub/Sub

Having located the logs you wish to export.

Create Export:

  • Sink service : Cloud Pub/Sub
  • Sink destination: Create new Cloud Pub/Sub topic
Step 3 - Create the google Pub/Sub subscription

Locate your newly created Pub/Sub topic.

Click the icon on the right and select New subscription

Give your subscription and leave the delivery type as pull

Step 4 - Create a service account for pubsubbeat to use

If you're planning on running pubsubbeat from google compute instance you can skip this step.

Visit service accounts and create a service account for pubsubbeat to use, download the service key as json when prompted.

Step 5 - Download and configure pubsubbeat

Download and extract the appropriate release from Github.

linux

wget https://github.com/GoogleCloudPlatform/pubsubbeat/releases/download/v1.0.1/pubsubbeat-linux-amd64.tar.gz
tar -xzvf pubsubbeat-linux-amd64.tar.gz
cd pubsubbeat-linux-amd64

Copy your credentials file from step 4 into the pubsubbeat folder

Edit your pubsubbbeat.yml config file

pubsubbeat:
  credentials_file: [if running on GCE leave blank, if running elsewhere this is the path to the json credentials file from step 4]
  project_id: [the GCP project ID]
  topic: [the name of your pub/sub topic]
  subscription.name: [the subscription name of your pub/sub topic]
  subscription.retain_acked_messages: false
  subscription.retention_duration: 24h
  json.enabed: true
  json.add_error_key: true

fields:
  type: googlesubpub
fields_under_root: true
encoding: utf-8
ignore_older: 3h

Comment out anything in the output.elasticsearch section

Uncomment and change the logstash output to match below.

output.logstash:
    hosts: ["your-logstash-host:your-port"]
    loadbalance: true
    ssl.enabled: true
Step 6 - Starting Pubsubbeat

Running pubsubbeat with debug enabled

./pubsubbeat -c pubsubbeat.yml -e -d "*"

If you get a permission error check the your project_id/topic/subscription name and the permissions of your service account.

Ctrl-C to quit once you're happy it's working

Step 7 - Running Pubsubbeat from systemd (optional)

Pubsubbeat doesn't have a service associated by default

Once you've had it working copy your pubsubbeat folder and json key to /opt/pubsubbeat

We need to create a systemd unit file for pubsubbeat

sudo vim /etc/systemd/system/pubsubbeat.service

And add the following

[Unit]
Description=PubSubBeat
After=network.target

[Service]
WorkingDirectory=/opt/pubsubbeat/
ExecStart=/opt/pubsubbeat/pubsubbeat -c pubsubbeat.yml
Type=simple
Restart=always

[Install]
WantedBy=default.target

We need to reload systemd and start the unit

sudo systemctl daemon-reload
sudo systemctl start pubsubbeat.service
sudo systemctl status pubsubbeat.service

Don't forget to enable the service if you wish it to start at boot

sudo systemctl enable pubsubbeat.service

Ready to get going?

Try our 14 day free trial

No commitment and no catches

Create Free Trial