Step 1 - Confirm your logs are in google Stackdriver
Locate the logs you wish to export in google stackdriver
Step 2 - Export your logs to google Pub/Sub
Having located the logs you wish to export.
- Sink service : Cloud Pub/Sub
- Sink destination: Create new Cloud Pub/Sub topic
Step 3 - Create the google Pub/Sub subscription
Locate your newly created Pub/Sub topic.
Click the icon on the right and select New subscription
Give your subscription and leave the delivery type as pull
Step 4 - Create a service account for pubsubbeat to use
If you're planning on running pubsubbeat from google compute instance you can skip this step.
Visit service accounts and create a service account for pubsubbeat to use, download the service key as json when prompted.
Step 5 - Download and configure pubsubbeat
Download and extract the appropriate release from Github.
wget https://github.com/GoogleCloudPlatform/pubsubbeat/releases/download/v1.0.1/pubsubbeat-linux-amd64.tar.gz tar -xzvf pubsubbeat-linux-amd64.tar.gz cd pubsubbeat-linux-amd64
Copy your credentials file from step 4 into the pubsubbeat folder
Edit your pubsubbbeat.yml config file
pubsubbeat: credentials_file: [if running on GCE leave blank, if running elsewhere this is the path to the json credentials file from step 4] project_id: [the GCP project ID] topic: [the name of your pub/sub topic] subscription.name: [the subscription name of your pub/sub topic] subscription.retain_acked_messages: false subscription.retention_duration: 24h json.enabed: true json.add_error_key: true fields: type: googlesubpub fields_under_root: true encoding: utf-8 ignore_older: 3h
Comment out anything in the output.elasticsearch section
Uncomment and change the logstash output to match below.
output.logstash: hosts: ["your-logstash-host:your-port"] loadbalance: true ssl.enabled: true
Step 6 - Starting Pubsubbeat
Running pubsubbeat with debug enabled
./pubsubbeat -c pubsubbeat.yml -e -d "*"
If you get a permission error check the your project_id/topic/subscription name and the permissions of your service account.
Ctrl-C to quit once you're happy it's working
Step 7 - Running Pubsubbeat from systemd (optional)
Pubsubbeat doesn't have a service associated by default
Once you've had it working copy your pubsubbeat folder and json key to /opt/pubsubbeat
We need to create a systemd unit file for pubsubbeat
sudo vim /etc/systemd/system/pubsubbeat.service
And add the following
[Unit] Description=PubSubBeat After=network.target [Service] WorkingDirectory=/opt/pubsubbeat/ ExecStart=/opt/pubsubbeat/pubsubbeat -c pubsubbeat.yml Type=simple Restart=always [Install] WantedBy=default.target
We need to reload systemd and start the unit
sudo systemctl daemon-reload sudo systemctl start pubsubbeat.service sudo systemctl status pubsubbeat.service
Don't forget to enable the service if you wish it to start at boot
sudo systemctl enable pubsubbeat.service