Send data via Google Cloud to your Logstash instance provided by Logit.io

Google Cloud

Ship logs from Google Stackdriver to logstash

Step 1 - Confirm Google Stackdriver Logs

First, confirm your logs are located in Google Stackdriver by going to Google Stackdriver.

Step 2 - Export Logs to Google Pub/Sub

Having located the logs you wish to export.

Create Export:

  • Sink service : Cloud Pub/Sub
  • Sink destination: Create new Cloud Pub/Sub topic

Step 3 - Create Google Pub/Sub

Locate your newly created Pub/Sub topic.

Click the icon on the right and select New subscription

Give your subscription and leave the delivery type as pull

Step 4 - Create Service Account

If you're planning on running pubsubbeat from Google compute instance you can skip this step.

Visit service accounts and create a service account for pubsubbeat to use, download the service key as json when prompted.

Step 5 - Configure pubsubbeat

First, download and extract the appropriate release from Github.

linux

wget https://github.com/GoogleCloudPlatform/pubsubbeat/releases/download/v1.0.1/pubsubbeat-linux-amd64.tar.gz
tar -xzvf pubsubbeat-linux-amd64.tar.gz
cd pubsubbeat-linux-amd64

Copy your credentials file from step 4 into the pubsubbeat folder

Edit your pubsubbbeat.yml config file

pubsubbeat:
  credentials_file: [if running on GCE leave blank, if running elsewhere this is the path to the json credentials file from step 4]
  project_id: [the GCP project ID]
  topic: [the name of your pub/sub topic]
  subscription.name: [the subscription name of your pub/sub topic]
  subscription.retain_acked_messages: false
  subscription.retention_duration: 24h
  json.enabed: true
  json.add_error_key: true
    
fields:
  type: googlesubpub
fields_under_root: true
encoding: utf-8
ignore_older: 3h

Comment out anything in the output.elasticsearch section

Uncomment and change the logstash output to match below.

output.logstash:
    hosts: ["your-logstash-host:your-port"]
    loadbalance: true
    ssl.enabled: true

Step 6 - Starting Pubsubbeat

Running pubsubbeat with debug enabled:

./pubsubbeat -c pubsubbeat.yml -e -d "*"

If you get a permission error check the your project_id/topic/subscription name and the permissions of your service account.

Ctrl-C to quit once you're happy it's working

Step 7 - (Optional) Running Pubsubbeat

Pubsubbeat doesn't have a service associated by default.

Once you've had it working copy your pubsubbeat folder and json key to /opt/pubsubbeat.

We need to create a systemd unit file for pubsubbeat.

sudo vim /etc/systemd/system/pubsubbeat.service

And add the following

[Unit]
Description=PubSubBeat
After=network.target

[Service]
WorkingDirectory=/opt/pubsubbeat/
ExecStart=/opt/pubsubbeat/pubsubbeat -c pubsubbeat.yml
Type=simple
Restart=always

[Install]
WantedBy=default.target

We need to reload systemd and start the unit

sudo systemctl daemon-reload
sudo systemctl start pubsubbeat.service
sudo systemctl status pubsubbeat.service

Don't forget to enable the service if you wish it to start at boot

sudo systemctl enable pubsubbeat.service
expand view

Expand View

compact view

Compact View

Return to Search
Sign up to get started Return to Search