Windows sensu purge issue #879

danb223 · 2018-01-30T16:15:14Z

When trying to put a json file in the conf.d directory, sensu removes the file and then puppet puts it back, on every puppet run (plus the sensu service restarting). The same does not happen in linux. The workaround for this is to disable purging for conf for windows, but not an ideal workaround. I have tried with both File resource and using the sensu::write_json, the same happens for both, but run fine on a Linux client, example below:

Info: Caching catalog for 89asuydasd1
Info: Applying configuration version '1517237731'
Info: Computing checksum on file C:/opt/sensu/conf.d/notification.json
Info: FileBucket got a duplicate file {md5}fe1ead95b9e306de4b898f21f22261e9
Info: /Stage[main]/Sensu::Package/File[C:/opt/sensu/conf.d/notification.json]: Filebucketed C:/opt/sensu/conf.d/notification.json to puppet with sum fe1ead95b9e306de4b898f21f22261e9
Notice: /Stage[main]/Sensu::Package/File[C:/opt/sensu/conf.d/notification.json]/ensure: removed
Info: Class[Sensu::Package]: Scheduling refresh of Service[sensu-client]
Notice: /Stage[main]/Sensu::Client/Service[sensu-client]: Triggered 'refresh' from 1 events
Notice: Applied catalog in 13.72 seconds

Platform and version information

Your OS: Windows 2012 R2
Your Ruby version: ruby 2.4.1p111
Your version of Puppet: 1.9.3 (agent)
Your version of Sensu: 1.2.0-1
Your version of this module: v2.50.0

treydock · 2018-04-07T14:53:52Z

@danb223 Could you please provide Puppet code for resources so I can try and reproduce?

ghoneycutt · 2018-04-16T00:23:04Z

Please re-open if this is still have an issue.

danb223 · 2018-04-16T07:56:20Z

An example below:
file:
'/etc/sensu/conf.d/notification.json':
content: |
{
"pagerduty": { "api_key": "%{hiera('sensu_apikey_operations')}" },
"windows": { "api_key": "%{hiera('sensu_apikey_windows')}" }
}
rt_notify:
service: 'sensu-client'

I also tried to do the same with sensu::write_json instead of using file, but the same happens, the file get changed on every puppet run and therefore sensu reloads every time. When sensu reloads this also causes an issue in the windows plugin, but providing it isn't reloading all the time then it isn't too much of an issue (sensu-plugins/sensu-plugins-windows#33). This causes no problem with a Linux client and the workaround we have in Windows is to tell sensu purge config to false, but is not ideal.

@ghoneycutt can you please re-open?

ghoneycutt · 2018-04-16T15:39:35Z

The above is not valid puppet code that we can use to reproduce the issue. Could you please provide valid Puppet code.

danb223 · 2018-04-16T15:56:59Z

This is just in a Hiera format. If you just make a file in puppet or use the sensu::write_json along with any content, you should be able to recreate this with Windows.

treydock · 2018-04-17T22:07:29Z

@danb223 What parameters are being passed to sensu class? I modified tests/sensu-client-windows.pp to try and reproduce and was unable. The code I used is upgraded to 1.2.0-1 and with empty notification.json

class { '::sensu':
  version           => '1.2.0-1',
  rabbitmq_password => 'correct-horse-battery-staple',
  rabbitmq_host     => '192.168.56.10',
  rabbitmq_vhost    => '/sensu',
  subscriptions     => 'all',
  client_address    => $facts['networking']['ip'],
  purge             => true,
}

# Test for #820
::sensu::subscription { 'roundrobin:foo':
  ensure => present,
}

file { 'C:/opt/sensu/conf.d/notification.json':
  ensure => 'file',
  content => '{}',
  notify  => Service['sensu-client'],
}

danb223 · 2018-04-18T14:56:11Z

Have you tried with content, does the actual file get created on the Windows client? here is the extract from our Windows in Hiera

file:
  'c:\\opt\\sensu\\conf.d\\notification.json':
    content: |
              {
                "windows": { "api_key": "%{hiera('sensu_apikey_windows')}" }
              }
    rt_notify:
      service: 'sensu-client'

We are using the sensu::check: along with the multiple checks from sensu-plugin-windows

sensu::client_name: "%{::environment}-%{::role}-%{::stack}-%{::hostname}-%{::ipaddress}"
sensu::subscriptions: ['windows', 'system']
sensu::version: '1.3.1-2'
sensu::client_keepalive:
handlers: ['windows', 'slack']
sensu::purge:
config: false < note that this is set to resolve the problem

sensu::server: false
sensu::api_host: '127.0.0.1'
sensu::api: false
sensu::api_user: ''
sensu::api_password: ''
sensu::subscriptions: 'system'
sensu::client_name: "%{::environment}-%{::role}-%{::stack}-%{::ipaddress}"
sensu::rabbitmq_host: "127.0.0.1"
sensu::rabbitmq_vhost: '/'
sensu::rabbitmq_password: 'guest'
sensu::rabbitmq_user: 'guest'
sensu::redis_host: '127.0.0.1'
sensu::log_level: 'warn'
sensu_apikey_operations: ''
slack_sensu_reporting_kalive: 'slack'

treydock · 2018-04-18T22:30:45Z

Added sensu-plugins-windows and gave notifications.json content, which does exist, and still not getting removed. I manually restarted the sensu-client service to see if that changed anything and still no issues.

In your original output from Puppet, I don't see Puppet putting the file back. I only see Puppet removing the file. If I comment out the file resource for notifications.json I get output similar to what was originally provided. If I run Puppet again the file is not removed again.

C:\Users\vagrant>puppet apply -v C:/vagrant/tests/sensu-client-windows.pp
Info: Loading facts
Info: Loading facts
Info: Loading facts
Notice: Scope(Class[Sensu::Package]): Managing a local windows user is not imple
mented on windows
Notice: Compiled catalog for win2012r2-client in environment production in 2.17
seconds
Info: Applying configuration version '1524090454'
Info: Computing checksum on file C:/opt/sensu/conf.d/notification.json
Info: /Stage[main]/Sensu::Package/File[C:/opt/sensu/conf.d/notification.json]: F
ilebucketed C:/opt/sensu/conf.d/notification.json to puppet with sum c1465101dfb
a021d9738aae94d833789
Notice: /Stage[main]/Sensu::Package/File[C:/opt/sensu/conf.d/notification.json]/
ensure: removed
Info: Class[Sensu::Package]: Scheduling refresh of Service[sensu-client]
Notice: /Stage[main]/Sensu::Client/Service[sensu-client]: Triggered 'refresh' fr
om 1 events
Notice: Applied catalog in 14.81 seconds

C:\Users\vagrant>
C:\Users\vagrant>puppet apply -v C:/vagrant/tests/sensu-client-windows.pp
Info: Loading facts
Info: Loading facts
Info: Loading facts
Notice: Scope(Class[Sensu::Package]): Managing a local windows user is not imple
mented on windows
Notice: Compiled catalog for win2012r2-client in environment production in 1.91
seconds
Info: Applying configuration version '1524090514'
Notice: Applied catalog in 8.02 seconds

C:\Users\vagrant>

Test case below, run first withfile resource not commented out then commented out to produce output above.

class { '::sensu':
  version           => '1.3.1-2',
  rabbitmq_password => 'correct-horse-battery-staple',
  rabbitmq_host     => '192.168.56.10',
  rabbitmq_vhost    => '/sensu',
  subscriptions     => 'all',
  client_address    => $facts['networking']['ip'],
  purge             => true,
}

# Test for #820
::sensu::subscription { 'roundrobin:foo':
  ensure => present,
}

#file { 'C:/opt/sensu/conf.d/notification.json':
#  ensure => 'file',
#  content => '{"windows" : {"api_key": "test"}}',
#  notify  => Service['sensu-client'],
#}

::sensu::plugin { 'sensu-plugins-windows':
  type => 'package',
}

danb223 · 2018-04-20T09:31:20Z

My initial output looks like it was wrong, apologises. However if I remove the sensu::purge, this is what happens everytime puppet runs

Info: Caching catalog for nfirlajxzp98mc6ukq70vdt52boshwe4
Info: Applying configuration version '1524216598'
Info: Computing checksum on file C:/opt/sensu/conf.d/notification.json
Info: FileBucket got a duplicate file {md5}fe1ead95b9e306de4b898f21f22261e9
Info: /Stage[main]/Sensu::Package/File[C:/opt/sensu/conf.d/notification.json]: F
ilebucketed C:/opt/sensu/conf.d/notification.json to puppet with sum fe1ead95b9e
306de4b898f21f22261e9
Notice: /Stage[main]/Sensu::Package/File[C:/opt/sensu/conf.d/notification.json]/
ensure: removed
Info: Class[Sensu::Package]: Scheduling refresh of Service[sensu-client]
Notice: /Stage[main]/Resource_tree/Resource_tree::Resource[file-c:\opt\sensu\
conf.d\notification.json]/File[c:\opt\sensu\conf.d\notification.json]/ensur
e: defined content as '{md5}fe1ead95b9e306de4b898f21f22261e9'
Info: /Stage[main]/Resource_tree/Resource_tree::Resource[file-c:\opt\sensu\co
nf.d\notification.json]/File[c:\opt\sensu\conf.d\notification.json]: Schedu
ling refresh of Service[sensu-client]
Notice: /Stage[main]/Sensu::Client/Service[sensu-client]: Triggered 'refresh' fr
om 2 events
Notice: Applied catalog in 24.88 seconds

treydock · 2018-04-20T13:02:32Z

Looks like the resource_tree module is adding the file to catalog. What version of the resource_tree module are you using? The file key in your hiera example doesn't appear like something provided by the resource_tree module, was part of the hiera missing? May be worth trying to reproduce the issue without using resource_tree and see if that is where the problem lies.

danb223 · 2018-04-20T13:21:23Z

https://github.com/jake-dog/puppet-resource_tree latest version (1.1.0)
The key is in hiera, but even if I just put anything in the file rather than referencing hiera the same happens. It only happens in Windows, the same is fine in Linux, and if I put the file somewhere else (for example c:\opt\notifications.json) it is fine there and doesn't get replaced

treydock · 2018-04-20T13:27:00Z

Is the file defined using a key above file or do you have some logic in a manifest somewhere that picks up file key? The resource_tree module looks to define things using resource_tree::collections so I'm curious how your resource is getting from Hiera to the catalog. I need to know exactly how resource_tree::resource is getting defined so I can add equivalent code to a manifest for a reproduction test.

danb223 · 2018-04-20T14:20:47Z

So there is a "windows" file in hiera that is applied through a fact to windows machine.

This contains the below:


  - sensu
  - resource_tree

resource_tree::apply:
classes:
  - sensu
  - resource_tree

resource_tree::apply:
  - windowsbase

sensu::client_name: "%{::environment}-%{::role}-%{::stack}-%{::hostname}-%{::ipaddress}" 
sensu::subscriptions: ['windows', 'system']
sensu::version: '1.3.1-2'
sensu::client_keepalive:
  handlers: ['windows', 'slack']
sensu::purge:
  config: false

Then in a file that contains our resrouce_tree::collections

  'windowsbase':
    package:
      'sensu-plugins-windows':
        ensure: '2.4.0'
        provider: 'sensu_gem'
    sensu::check:

Checks from https://github.com/sensu-plugins/sensu-plugins-windows here

      'c:\\opt\\sensu\\conf.d\\notification.json':
        content: |
                  {
                    "windows": { "api_key": "%{hiera('sensu_apikey_windows')}" }
                  }
        rt_notify:
          service: 'sensu-client'

We then have the "sensu_apikey_windows" stored elsewhere, but this is irrelevant as if I specify anything instead of that whole string in there, the same happens.

ghoneycutt · 2018-06-18T03:41:01Z

@treydock could you have another look at this please

treydock · 2018-06-18T22:37:14Z

I managed to reproduce:

Notice: Compiled catalog for win2012r2-client in environment production in 4.06
seconds
Notice: /Stage[main]/Sensu::Package/File[C:/opt/sensu/conf.d/notification.json]/
ensure: removed
Notice: /Stage[main]/Resource_tree/Resource_tree::Resource[file-c:\opt\sensu\con
f.d\notification.json]/File[c:\opt\sensu\conf.d\notification.json]/ensure: defin
ed content as '{md5}fdc8c494c67fbcc5c632a17f6d86c18a'
Notice: /Stage[main]/Sensu::Client/Service[sensu-client]: Triggered 'refresh' fr
om 2 events
Notice: Applied catalog in 13.42 seconds

This is reproducer used with Vagrant:

class { '::sensu':
  version           => '1.3.1-2',
  rabbitmq_password => 'correct-horse-battery-staple',
  rabbitmq_host     => '192.168.56.10',
  rabbitmq_vhost    => '/sensu',
  subscriptions     => 'all',
  client_address    => $facts['networking']['ip'],
  purge             => true,
}

class { 'resource_tree':
collections => {
  'windowsbase' => {
    'package' => {
      'sensu-plugins-windows' => { 'ensure' => '2.4.0', 'provider' => 'sensu_gem' },
    },
    'file' => {
      'c:\\opt\\sensu\\conf.d\\notification.json' => { 'content' => '{"windows":{"api_key":"test"}}', 'rt_notify' => { 'service' => 'sensu-client' }},
    }
  }
},
apply => ['windowsbase'],
}

I did manage to make resource_tree work by changing the path used by the file resource to be C:/opt/sensu/conf.d/notification.json. I also used C:\\opt\\sensu\\conf.d\\notification.json with success. My guess is using C: is needed and not the lowercase c:.

treydock · 2018-06-18T22:38:02Z

@danb223 Try using upper case C: with your Hiera paths for files and let me know if that resolves your issue.

danb223 · 2018-06-19T08:46:08Z

@treydock that did the trick. I'm not sure whether this is now a puppet issue, or whether some logic needs to be applied to this sensu module, or whether we just need to make sure we are using an upper case C:

treydock · 2018-06-19T14:53:16Z

I think this is a Puppet + Windows quirk. There is this page: https://puppet.com/docs/puppet/5.5/lang_windows_file_paths.html but it doesn't mention C: vs c:. All the examples of File resources use capital C:. My guess is that when Puppet matches the resources in catalog for purging it's doing so with case sensitive matching.

ghoneycutt · 2018-06-19T15:23:38Z

Thanks for figuring this out!

danb223 mentioned this issue Jan 30, 2018

Checks Intermintent fail sensu-plugins/sensu-plugins-windows#33

Open

ghoneycutt assigned treydock Apr 5, 2018

ghoneycutt added the bug label Apr 5, 2018

ghoneycutt added the waiting for feedback label Apr 16, 2018

ghoneycutt closed this as completed Apr 16, 2018

ghoneycutt reopened this Apr 16, 2018

ghoneycutt closed this as completed Jun 19, 2018

ghoneycutt added the sensu v1 label Sep 5, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Windows sensu purge issue #879

Windows sensu purge issue #879

danb223 commented Jan 30, 2018

treydock commented Apr 7, 2018

ghoneycutt commented Apr 16, 2018

danb223 commented Apr 16, 2018 •

edited

Loading

ghoneycutt commented Apr 16, 2018 •

edited

Loading

danb223 commented Apr 16, 2018

treydock commented Apr 17, 2018

danb223 commented Apr 18, 2018 •

edited

Loading

treydock commented Apr 18, 2018 •

edited

Loading

danb223 commented Apr 20, 2018

treydock commented Apr 20, 2018

danb223 commented Apr 20, 2018 •

edited

Loading

treydock commented Apr 20, 2018

danb223 commented Apr 20, 2018 •

edited

Loading

ghoneycutt commented Jun 18, 2018

treydock commented Jun 18, 2018

treydock commented Jun 18, 2018

danb223 commented Jun 19, 2018

treydock commented Jun 19, 2018

ghoneycutt commented Jun 19, 2018

Windows sensu purge issue #879

Windows sensu purge issue #879

Comments

danb223 commented Jan 30, 2018

Platform and version information

treydock commented Apr 7, 2018

ghoneycutt commented Apr 16, 2018

danb223 commented Apr 16, 2018 • edited Loading

ghoneycutt commented Apr 16, 2018 • edited Loading

danb223 commented Apr 16, 2018

treydock commented Apr 17, 2018

danb223 commented Apr 18, 2018 • edited Loading

treydock commented Apr 18, 2018 • edited Loading

danb223 commented Apr 20, 2018

treydock commented Apr 20, 2018

danb223 commented Apr 20, 2018 • edited Loading

treydock commented Apr 20, 2018

danb223 commented Apr 20, 2018 • edited Loading

ghoneycutt commented Jun 18, 2018

treydock commented Jun 18, 2018

treydock commented Jun 18, 2018

danb223 commented Jun 19, 2018

treydock commented Jun 19, 2018

ghoneycutt commented Jun 19, 2018

danb223 commented Apr 16, 2018 •

edited

Loading

ghoneycutt commented Apr 16, 2018 •

edited

Loading

danb223 commented Apr 18, 2018 •

edited

Loading

treydock commented Apr 18, 2018 •

edited

Loading

danb223 commented Apr 20, 2018 •

edited

Loading

danb223 commented Apr 20, 2018 •

edited

Loading