Ship logs from Google Stackdriver to logstash
Locate the logs you wish to export in google stackdriver
Having located the logs you wish to export.
Locate your newly created Pub/Sub topic.
Click the icon on the right and select New subscription
Give your subscription and leave the delivery type as pull
If you're planning on running pubsubbeat from google compute instance you can skip this step.
Visit service accounts and create a service account for pubsubbeat to use, download the service key as json when prompted.
Download and extract the appropriate release from Github.
wget https://github.com/GoogleCloudPlatform/pubsubbeat/releases/download/v1.0.1/pubsubbeat-linux-amd64.tar.gz tar -xzvf pubsubbeat-linux-amd64.tar.gz cd pubsubbeat-linux-amd64
Copy your credentials file from step 4 into the pubsubbeat folder
Edit your pubsubbbeat.yml config file
pubsubbeat: credentials_file: [if running on GCE leave blank, if running elsewhere this is the path to the json credentials file from step 4] project_id: [the GCP project ID] topic: [the name of your pub/sub topic] subscription.name: [the subscription name of your pub/sub topic] subscription.retain_acked_messages: false subscription.retention_duration: 24h json.enabed: true json.add_error_key: true fields: type: googlesubpub fields_under_root: true encoding: utf-8 ignore_older: 3h
Comment out anything in the output.elasticsearch section
Uncomment and change the logstash output to match below.
output.logstash: hosts: ["your-logstash-host:your-port"] loadbalance: true ssl.enabled: true
Running pubsubbeat with debug enabled
./pubsubbeat -c pubsubbeat.yml -e -d "*"
If you get a permission error check the your project_id/topic/subscription name and the permissions of your service account.
Ctrl-C to quit once you're happy it's working
Pubsubbeat doesn't have a service associated by default
Once you've had it working copy your pubsubbeat folder and json key to /opt/pubsubbeat
We need to create a systemd unit file for pubsubbeat
sudo vim /etc/systemd/system/pubsubbeat.service
And add the following
[Unit] Description=PubSubBeat After=network.target [Service] WorkingDirectory=/opt/pubsubbeat/ ExecStart=/opt/pubsubbeat/pubsubbeat -c pubsubbeat.yml Type=simple Restart=always [Install] WantedBy=default.target
We need to reload systemd and start the unit
sudo systemctl daemon-reload sudo systemctl start pubsubbeat.service sudo systemctl status pubsubbeat.service
Don't forget to enable the service if you wish it to start at boot
sudo systemctl enable pubsubbeat.service