Confirm your logs are in google Stackdriver

Locate the logs you wish to export in google stackdriver

Export your logs to google Pub/Sub

Having located the logs you wish to export.

Create Export:

  • Sink service : Cloud Pub/Sub
  • Sink destination: Create new Cloud Pub/Sub topic

Create the google Pub/Sub subscription

Locate your newly created Pub/Sub topic.

Click the icon on the right and select New subscription

Give your subscription and leave the delivery type as pull

Create a service account for pubsubbeat to use

If you're planning on running pubsubbeat from google compute instance you can skip this step.

Visit service accounts and create a service account for pubsubbeat to use, download the service key as json when prompted.

Download and configure pubsubbeat

Download and extract the appropriate release from Github.

linux

wget https://github.com/GoogleCloudPlatform/pubsubbeat/releases/download/v1.0.1/pubsubbeat-linux-amd64.tar.gz
tar -xzvf pubsubbeat-linux-amd64.tar.gz
cd pubsubbeat-linux-amd64

Copy your credentials file from step 4 into the pubsubbeat folder

Edit your pubsubbbeat.yml config file

pubsubbeat:
  credentials_file: [if running on GCE leave blank, if running elsewhere this is the path to the json credentials file from step 4]
  project_id: [the GCP project ID]
  topic: [the name of your pub/sub topic]
  subscription.name: [the subscription name of your pub/sub topic]
  subscription.retain_acked_messages: false
  subscription.retention_duration: 24h
  json.enabed: true
  json.add_error_key: true

fields:
  type: googlesubpub
fields_under_root: true
encoding: utf-8
ignore_older: 3h

Comment out anything in the output.elasticsearch section

Uncomment and change the logstash output to match below.

output.logstash:
    hosts: ["your-logstash-host:your-port"]
    loadbalance: true
    ssl.enabled: true

Starting Pubsubbeat

Running pubsubbeat with debug enabled

./pubsubbeat -c pubsubbeat.yml -e -d "*"

If you get a permission error check the your project_id/topic/subscription name and the permissions of your service account.

Ctrl-C to quit once you're happy it's working

Running Pubsubbeat from systemd (optional)

Pubsubbeat doesn't have a service associated by default

Once you've had it working copy your pubsubbeat folder and json key to /opt/pubsubbeat

We need to create a systemd unit file for pubsubbeat

sudo vim /etc/systemd/system/pubsubbeat.service

And add the following

[Unit]
Description=PubSubBeat
After=network.target

[Service]
WorkingDirectory=/opt/pubsubbeat/
ExecStart=/opt/pubsubbeat/pubsubbeat -c pubsubbeat.yml
Type=simple
Restart=always

[Install]
WantedBy=default.target

We need to reload systemd and start the unit

sudo systemctl daemon-reload
sudo systemctl start pubsubbeat.service
sudo systemctl status pubsubbeat.service

Don't forget to enable the service if you wish it to start at boot

sudo systemctl enable pubsubbeat.service