Skip to content

Latest commit

 

History

History
76 lines (60 loc) · 2.16 KB

README.md

File metadata and controls

76 lines (60 loc) · 2.16 KB
Raw

fluent-plugin-gcloud-pubsub

Build Status Gem Version

Overview

Cloud Pub/Sub Input/Output plugin for Fluentd with gcloud gem

  • Publish BufferedOutput chunk into Cloud Pub/Sub with batch publishing
  • Pull messages from Cloud Pub/Sub

Preparation

  • Create a project on Google Developer Console
  • Add a topic of Cloud Pub/Sub to the project
  • Add a pull style subscription to the topic
  • Download your credential (json)

Publish messages

Use out_gcloud_pubsub.

Configuration

publish dummy json data like {"message": "dummy", "value": 0}\n{"message": "dummy", "value": 1}\n ....

<source>
  type dummy
  tag example.publish
  rate 100 
  auto_increment_key value
</source>

<match example.publish>
  type gcloud_pubsub
  project <YOUR PROJECT>
  topic <YOUR TOPIC>
  key <YOUR KEY>
  flush_interval 10
  autocreate_topic false
</match>
  • autocreate_topic (optional, default: false)
    • If set to true, specified topic will be created when it doesn't exist.

Pull messages

Use in_gcloud_pubsub.

Configuration

Pull json data from Cloud Pub/Sub

<source>
  type gcloud_pubsub
  tag example.pull
  project <YOUR PROJECT>
  topic <YOUR TOPIC>
  subscription <YOUR SUBSCRIPTION>
  key <YOUR KEY>
  max_messages 1000
  return_immediately true
  pull_interval 2
  format json
</source>

<match example.pull>
  type stdout
</match>