Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parser type filter not working with systemd logs #106

Open
vipulbehalpande08 opened this issue Aug 22, 2023 · 0 comments
Open

Parser type filter not working with systemd logs #106

vipulbehalpande08 opened this issue Aug 22, 2023 · 0 comments

Comments

@vipulbehalpande08
Copy link

Hi Team,

I have an inline JSON string in message field of systemd logs that I want to parse but when I add filter @type parser it stops capturing the logs in cloudwatch. It works with other log sources. Also, there are no error logs in td-agent.log

{
"syslog_identifier": "vault-audit",
"message": "{\"field_1\": \"value1\", \"field_2\": \"value_2\"}"
}

I am trying to parse above message in JSON format like below

{
   "syslog_identifier":"vault-audit",
   "message":{
      "field_1":"value1",
      "field_2":"value_2"
   }
}

Config:

<source>
  @type systemd
  tag journal
  path /var/log/journal
  read_from_head true

  <storage>
    @type local
    path /var/log/td-agent/fluentd-journald-cursor.json
  </storage>

  <entry>
    fields_strip_underscores true
    fields_lowercase true
  </entry>
</source>

<filter vault-audit>
  @type parser
  
  key_name message
  reserve_data true
  hash_value_field app_message
  emit_invalid_record_to_error false
  <parse>
    @type json
  </parse>
</filter>

<match journal>
  @type rewrite_tag_filter

  <rule>
    key syslog_identifier
    pattern /^vault-audit$/
    tag vault-audit
  </rule>
 </match>

<match vault-audit>
  @type cloudwatch_logs
  region {{region}}
  log_group_name /ec2/vault/audit
  log_stream_name {{instance_id}}
  auto_create_stream true
  retention_in_days 7
</match> 
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant