Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mattermost-team-edition does not startup on some clusters #411

Open
mustdiechik opened this issue Jul 18, 2023 · 1 comment
Open

Mattermost-team-edition does not startup on some clusters #411

mustdiechik opened this issue Jul 18, 2023 · 1 comment

Comments

@mustdiechik
Copy link
Contributor

mustdiechik commented Jul 18, 2023 •
edited
Loading

test environment:

  • image mattermost/mattermost-team-edition
  • tag: 7.10.3,7.10.4
  • remote s3 (no PVC used)
  • remote PostgreSQL database (bouncer session mode)
  • chart ver: 6.6.31

I patched the chart and add additional loadBalancer for calls:

...
  loadBalancerIP: x.x.x.x
  externalTrafficPolicy: Local
  ports:
  - name: rtc
    port: 8443
    protocol: UDP
  type: LoadBalancer
...

The same configuration with only different loadBalancerIP tested on 4 k8s clusters. One of them - minikube, 3 other - managed by cloud provider (test, stage, prod).

I have strange problem with prod cluster - in 3 other all works as expected. Pod didn`t switch in Running state.
7.10.3 logs:

mattermost@mattermost-5ddcdf4665-tzkcw:~$ cat /mattermost/logs/mattermost.log 
{"timestamp":"2023-07-17 22:56:20.669 Z","level":"info","msg":"Server is initializing...","caller":"platform/service.go:165","go_version":"go1.19.5"}
{"timestamp":"2023-07-17 22:56:20.669 Z","level":"info","msg":"Pinging SQL","caller":"sqlstore/store.go:242","database":"master","dataSource":"postgres:__REDACTED__?connect_timeout=3&sslmode=disable"}
{"timestamp":"2023-07-17 22:56:20.993 Z","level":"debug","msg":"Deleting any unused pre-release features","caller":"sqlstore/preference_store.go:25"}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"info","msg":"Starting websocket hubs","caller":"platform/web_hub.go:95","number_of_hubs":16}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":1}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":3}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":2}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":0}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":6}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":7}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":9}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":4}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":8}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":10}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":11}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":12}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":13}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":5}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":14}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":15}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"info","msg":"Loaded system translations","caller":"i18n/i18n.go:132","for locale":"en","from locale":"/mattermost/i18n/en.json"}
{"timestamp":"2023-07-17 22:56:21.072 Z","level":"info","msg":"Ensuring the telemetry ID","caller":"telemetry/telemetry.go:142","id":"nqpe5ooaujypxqmjh56ucxsmbe"}
{"timestamp":"2023-07-17 22:56:21.132 Z","level":"debug","msg":"Email batching job starting. Checking for pending emails periodically.","caller":"email/email_batching.go:90","interval_in_seconds":30}
{"timestamp":"2023-07-17 22:56:21.251 Z","level":"info","msg":"Current version is 7.10.3 (7.10.3/Wed Jun 14 12:09:40 UTC 2023/74af1505a69a560085fd609620e3f92123e84536/none)","caller":"app/server.go:412","current_version":"7.10.3","build_number":"7.10.3","build_date":"Wed Jun 14 12:09:40 UTC 2023","build_hash":"74af1505a69a560085fd609620e3f92123e84536","build_hash_enterprise":"none"}
{"timestamp":"2023-07-17 22:56:21.251 Z","level":"info","msg":"Team Edition Build","caller":"app/server.go:423","enterprise_build":false}
{"timestamp":"2023-07-17 22:56:21.251 Z","level":"info","msg":"Printing current working","caller":"app/server.go:427","directory":"/mattermost"}
{"timestamp":"2023-07-17 22:56:21.251 Z","level":"info","msg":"Loaded config","caller":"app/server.go:428","source":"postgres://%2A%2A%2A%2A:%2A%2A%2A%[email protected]:6432/mattermost?connect_timeout=3&sslmode=disable"}
{"timestamp":"2023-07-17 22:56:21.251 Z","level":"debug","msg":"Will fetch notices from","caller":"app/product_notices.go:344","url":"https://notices.mattermost.com/","skip_cache":false}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"info","msg":"Starting workers","caller":"jobs/workers.go:48"}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"debug","msg":"Cleaning up command webhook store.","caller":"sqlstore/command_webhook_store.go:93"}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"debug","msg":"Worker started","caller":"scheduler/worker.go:39","worker":"Plugins"}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"debug","msg":"Watcher Started","caller":"jobs/jobs_watcher.go:38"}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"debug","msg":"Worker Started","caller":"indexer/indexing_job.go:100","workername":"BleveIndexer"}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ExportDelete"}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"debug","msg":"Worker started","caller":"resend_invitation_email/worker.go:54","worker":"resend_invitation_email"}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ImportProcess"}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ExtractContent"}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ProductNotices"}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"HostedPurchaseScreening"}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ExportProcess"}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"debug","msg":"Worker started","caller":"migrations/worker.go:50","worker":"Migrations"}
{"timestamp":"2023-07-17 22:56:21.322 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"InstallNotifyAdmin"}
{"timestamp":"2023-07-17 22:56:21.322 Z","level":"debug","msg":"Cleaning up session store.","caller":"app/server.go:1277"}
{"timestamp":"2023-07-17 22:56:21.322 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ImportDelete"}
{"timestamp":"2023-07-17 22:56:21.322 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ExpiryNotify"}
{"timestamp":"2023-07-17 22:56:21.322 Z","level":"info","msg":"Starting schedulers.","caller":"jobs/schedulers.go:47"}
{"timestamp":"2023-07-17 22:56:21.322 Z","level":"debug","msg":"Cleaning up token store.","caller":"app/server.go:1262"}
{"timestamp":"2023-07-17 22:56:21.437 Z","level":"debug","msg":"We could not find the license key in the database or on disk at","caller":"utils/license.go:104","filename":"/mattermost/config/mattermost.mattermost-license"}
{"timestamp":"2023-07-17 22:56:21.446 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"plugins","next_runtime":"2023-07-18 22:56:22.336425601 +0000 UTC m=+86402.403838454"}
{"timestamp":"2023-07-17 22:56:21.472 Z","level":"error","msg":"License key from https://mattermost.com required to unlock enterprise features.","caller":"platform/license.go:101","error":"resource: License id: "}
{"timestamp":"2023-07-17 22:56:21.473 Z","level":"debug","msg":"Skipping license expiration check because no license is expected on Team Edition","caller":"app/server.go:1379"}
{"timestamp":"2023-07-17 22:56:21.525 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"expiry_notify","next_runtime":"2023-07-17 23:06:22.754042035 +0000 UTC m=+602.821454887"}
{"timestamp":"2023-07-17 22:56:21.590 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"product_notices","next_runtime":"2023-07-17 23:56:21.590547182 +0000 UTC m=+3601.657960032"}
{"timestamp":"2023-07-17 22:56:21.653 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"export_delete","next_runtime":"2023-07-18 22:56:22.184000011 +0000 UTC m=+86402.251412877"}
{"timestamp":"2023-07-17 22:56:21.653 Z","level":"debug","msg":"Scheduler: isEnabled: false","caller":"last_accessible_post/scheduler.go:20","scheduler":"last_accessible_post"}
{"timestamp":"2023-07-17 22:56:21.653 Z","level":"debug","msg":"Scheduler: isEnabled: true","caller":"notify_admin/install_plugin_scheduler.go:20","scheduler":"install_plugin_notify_admin"}
{"timestamp":"2023-07-17 22:56:21.715 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"install_plugin_notify_admin","next_runtime":"2023-07-18 22:56:22.559660692 +0000 UTC m=+86402.627073549"}
{"timestamp":"2023-07-17 22:56:21.775 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"migrations","next_runtime":"2023-07-17 22:57:21.775861481 +0000 UTC m=+61.843274332"}
{"timestamp":"2023-07-17 22:56:21.775 Z","level":"debug","msg":"Scheduler: isEnabled: false","caller":"last_accessible_file/scheduler.go:20","scheduler":"last_accessible_file"}
{"timestamp":"2023-07-17 22:56:21.775 Z","level":"debug","msg":"Scheduler: isEnabled: false","caller":"notify_admin/scheduler.go:20","scheduler":"upgrade_notify_admin"}
{"timestamp":"2023-07-17 22:56:21.775 Z","level":"debug","msg":"Scheduler: isEnabled: false","caller":"notify_admin/scheduler.go:20","scheduler":"trial_notify_admin"}
{"timestamp":"2023-07-17 22:56:21.837 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"import_delete","next_runtime":"2023-07-18 22:56:23.515543916 +0000 UTC m=+86403.582956768"}
{"timestamp":"2023-07-17 22:56:22.736 Z","level":"info","msg":"Skipping triggering Elasticsearch channel index fix job as build is not Enterprise ready","caller":"app/migrations.go:570"}
{"timestamp":"2023-07-17 22:56:22.752 Z","level":"debug","msg":"Initializing web routes","caller":"web/web.go:26"}
{"timestamp":"2023-07-17 22:56:22.753 Z","level":"debug","msg":"Rewriting static assets","caller":"utils/subpath.go:83","from_subpath":"/","to_subpath":"/"}
{"timestamp":"2023-07-17 22:56:22.789 Z","level":"debug","msg":"Using client directory","caller":"web/static.go:34","clientDir":"/mattermost/client"}
{"timestamp":"2023-07-17 22:56:22.789 Z","level":"info","msg":"Starting up plugins","caller":"app/plugin.go:218"}
{"timestamp":"2023-07-17 22:56:22.789 Z","level":"debug","msg":"Enabling plugin health check job","caller":"plugin/environment.go:613","interval_s":"30s"}
{"timestamp":"2023-07-17 22:56:22.789 Z","level":"info","msg":"Syncing plugins from the file store","caller":"app/plugin.go:299"}
{"timestamp":"2023-07-17 22:56:22.789 Z","level":"debug","msg":"Plugin health check job starting.","caller":"plugin/health_check.go:31"}
{"timestamp":"2023-07-17 22:56:22.874 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/com.mattermost.calls.tar.gz"}
{"timestamp":"2023-07-17 22:56:22.874 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/com.github.manland.mattermost-plugin-gitlab.tar.gz"}
{"timestamp":"2023-07-17 22:56:22.874 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/com.mattermost.custom-attributes.tar.gz"}
{"timestamp":"2023-07-17 22:56:22.874 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/com.mattermost.apps.tar.gz"}
{"timestamp":"2023-07-17 22:56:22.874 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/com.github.moussetc.mattermost.plugin.giphy.tar.gz"}
{"timestamp":"2023-07-17 22:56:22.874 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/com.github.matterpoll.matterpoll.tar.gz"}
{"timestamp":"2023-07-17 22:56:22.874 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/github.tar.gz"}
{"timestamp":"2023-07-17 22:56:22.874 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/jitsi.tar.gz"}
{"timestamp":"2023-07-17 22:56:22.874 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/jenkins.tar.gz"}
{"timestamp":"2023-07-17 22:56:22.874 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/mattermost-autolink.tar.gz"}
{"timestamp":"2023-07-17 22:56:25.195 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"jenkins","version":"1.1.0"}
{"timestamp":"2023-07-17 22:56:25.337 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.custom-attributes","wrapped_extras":"pathplugins/com.mattermost.custom-attributes/server/dist/plugin-linux-amd64args[plugins/com.mattermost.custom-attributes/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 22:56:25.337 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.custom-attributes","wrapped_extras":"pathplugins/com.mattermost.custom-attributes/server/dist/plugin-linux-amd64pid33"}
{"timestamp":"2023-07-17 22:56:25.337 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.custom-attributes","wrapped_extras":"pathplugins/com.mattermost.custom-attributes/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 22:56:25.345 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.custom-attributes","wrapped_extras":"address/tmp/plugin554633014networkunixtimestamp2023-07-17T22:56:25.345Z"}
{"timestamp":"2023-07-17 22:56:25.345 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.custom-attributes","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 22:56:25.352 Z","level":"debug","msg":"2023/07/17 22:56:25 LoadPluginConfiguration API failed to unmarshal: json: cannot unmarshal string into Go struct field CustomAttribute.CustomAttributes.GroupIDs of type []string","caller":"plugin/hclog_adapter.go:54","plugin_id":"com.mattermost.custom-attributes"}
{"timestamp":"2023-07-17 22:56:25.352 Z","level":"debug","msg":"Plugin activated","caller":"plugin/environment.go:333","plugin_id":"com.mattermost.custom-attributes","version":"1.3.1"}
{"timestamp":"2023-07-17 22:56:25.352 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"com.mattermost.custom-attributes","version":"1.3.1"}
{"timestamp":"2023-07-17 22:56:25.380 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"mattermost-autolink","wrapped_extras":"pathplugins/mattermost-autolink/server/dist/plugin-linux-amd64args[plugins/mattermost-autolink/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 22:56:25.381 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"mattermost-autolink","wrapped_extras":"pathplugins/mattermost-autolink/server/dist/plugin-linux-amd64pid41"}
{"timestamp":"2023-07-17 22:56:25.381 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"mattermost-autolink","wrapped_extras":"pathplugins/mattermost-autolink/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 22:56:25.400 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"mattermost-autolink","wrapped_extras":"networkunixaddress/tmp/plugin3266541804timestamp2023-07-17T22:56:25.400Z"}
{"timestamp":"2023-07-17 22:56:25.400 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"mattermost-autolink","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 22:56:25.404 Z","level":"debug","msg":"Plugin activated","caller":"plugin/environment.go:333","plugin_id":"mattermost-autolink","version":"1.4.0"}
{"timestamp":"2023-07-17 22:56:25.404 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"mattermost-autolink","version":"1.4.0"}
{"timestamp":"2023-07-17 22:56:31.101 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"github","version":"2.1.6"}
{"timestamp":"2023-07-17 22:56:31.122 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"com.github.moussetc.mattermost.plugin.giphy","version":"2.1.0"}
{"timestamp":"2023-07-17 22:56:31.387 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.manland.mattermost-plugin-gitlab","wrapped_extras":"pathplugins/com.github.manland.mattermost-plugin-gitlab/server/dist/plugin-linux-amd64args[plugins/com.github.manland.mattermost-plugin-gitlab/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 22:56:31.388 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.manland.mattermost-plugin-gitlab","wrapped_extras":"pathplugins/com.github.manland.mattermost-plugin-gitlab/server/dist/plugin-linux-amd64pid53"}
{"timestamp":"2023-07-17 22:56:31.388 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.manland.mattermost-plugin-gitlab","wrapped_extras":"pathplugins/com.github.manland.mattermost-plugin-gitlab/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 22:56:31.409 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.manland.mattermost-plugin-gitlab","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 22:56:31.409 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.manland.mattermost-plugin-gitlab","wrapped_extras":"address/tmp/plugin1920211718networkunixtimestamp2023-07-17T22:56:31.408Z"}
{"timestamp":"2023-07-17 22:56:31.463 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"pathplugins/com.mattermost.apps/server/dist/plugin-linux-amd64args[plugins/com.mattermost.apps/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 22:56:31.463 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"pathplugins/com.mattermost.apps/server/dist/plugin-linux-amd64pid65"}
{"timestamp":"2023-07-17 22:56:31.463 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"pathplugins/com.mattermost.apps/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 22:56:31.488 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 22:56:31.488 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"address/tmp/plugin194508154networkunixtimestamp2023-07-17T22:56:31.488Z"}
{"timestamp":"2023-07-17 22:56:31.792 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"jitsi","wrapped_extras":"pathplugins/jitsi/server/dist/plugin-linux-amd64args[plugins/jitsi/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 22:56:31.792 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"jitsi","wrapped_extras":"pathplugins/jitsi/server/dist/plugin-linux-amd64pid77"}
{"timestamp":"2023-07-17 22:56:31.792 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"jitsi","wrapped_extras":"pathplugins/jitsi/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 22:56:31.796 Z","level":"warn","msg":"ignored duplicate step name","caller":"app/plugin_api.go:979","plugin_id":"com.github.manland.mattermost-plugin-gitlab","name":"oauth-info","flow":"oauth"}
{"timestamp":"2023-07-17 22:56:31.796 Z","level":"debug","msg":"Plugin activated","caller":"plugin/environment.go:333","plugin_id":"com.github.manland.mattermost-plugin-gitlab","version":"1.6.0"}
{"timestamp":"2023-07-17 22:56:31.796 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"com.github.manland.mattermost-plugin-gitlab","version":"1.6.0"}
{"timestamp":"2023-07-17 22:56:31.800 Z","level":"debug","msg":"ensured bot","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","id":"rx9bx5d3ntrhumudpw3qf387ce","username":"appsbot","plugin_caller":"server/plugin.go:68"}
{"timestamp":"2023-07-17 22:56:31.814 Z","level":"debug","msg":"failed to fetch license twice. May incorrectly default to on-prem mode","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","plugin_caller":"server/config/service.go:226"}
{"timestamp":"2023-07-17 22:56:31.815 Z","level":"debug","msg":"configured the plugin.","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","developer_mode":"false","allow_http_apps":"true","version":"1.2.0","commit":"44db542","plugin_caller":"server/plugin.go:92","build_date":"Fri 06 Jan 2023 09:59:54 PM UTC","cloud_mode":"false"}
{"timestamp":"2023-07-17 22:56:31.815 Z","level":"debug","msg":"created an AWS client","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","access":"<nil>","secret":"<nil>","plugin_caller":"upstream/upaws/client.go:121","purpose":"Manifest store","region":"us-east-1"}
{"timestamp":"2023-07-17 22:56:31.815 Z","level":"debug","msg":"initialized API and persistent store","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","plugin_caller":"server/plugin.go:106"}
{"timestamp":"2023-07-17 22:56:31.816 Z","level":"debug","msg":"available upstream","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","type":"HTTP","plugin_caller":"server/proxy/service.go:232"}
{"timestamp":"2023-07-17 22:56:31.816 Z","level":"debug","msg":"Skipped upstream: AWS Lambda: not configured.","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","error":"AWS credentials are not set: not found","plugin_caller":"server/proxy/service.go:227"}
{"timestamp":"2023-07-17 22:56:31.816 Z","level":"debug","msg":"available upstream","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","type":"Mattermost Plugin","plugin_caller":"server/proxy/service.go:232"}
{"timestamp":"2023-07-17 22:56:31.816 Z","level":"debug","msg":"Skipped upstream: OpenFaaS: not configured.","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","plugin_caller":"server/proxy/service.go:227","error":"OPENFAAS_URL environment variable must be defined: not found"}
{"timestamp":"2023-07-17 22:56:31.817 Z","level":"debug","msg":"initialized the app proxy","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","plugin_caller":"server/plugin.go:122"}
{"timestamp":"2023-07-17 22:56:31.818 Z","level":"debug","msg":"initialized incoming HTTP","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","plugin_caller":"server/plugin.go:125"}
{"timestamp":"2023-07-17 22:56:31.818 Z","level":"info","msg":"activated","caller":"app/plugin_api.go:973","plugin_id":"com.mattermost.apps","plugin_caller":"server/plugin.go:135"}
{"timestamp":"2023-07-17 22:56:31.819 Z","level":"debug","msg":"Plugin activated","caller":"plugin/environment.go:333","plugin_id":"com.mattermost.apps","version":"1.2.0"}
{"timestamp":"2023-07-17 22:56:31.819 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"com.mattermost.apps","version":"1.2.0"}
{"timestamp":"2023-07-17 22:56:32.084 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"jitsi","wrapped_extras":"address/tmp/plugin2141876054networkunixtimestamp2023-07-17T22:56:32.083Z"}
{"timestamp":"2023-07-17 22:56:32.084 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"jitsi","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 22:56:32.207 Z","level":"debug","msg":"Plugin activated","caller":"plugin/environment.go:333","plugin_id":"jitsi","version":"2.0.1"}
{"timestamp":"2023-07-17 22:56:32.207 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"jitsi","version":"2.0.1"}
{"timestamp":"2023-07-17 22:56:32.411 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.matterpoll.matterpoll","wrapped_extras":"pathplugins/com.github.matterpoll.matterpoll/server/dist/plugin-linux-amd64args[plugins/com.github.matterpoll.matterpoll/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 22:56:32.411 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.matterpoll.matterpoll","wrapped_extras":"pathplugins/com.github.matterpoll.matterpoll/server/dist/plugin-linux-amd64pid90"}
{"timestamp":"2023-07-17 22:56:32.411 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.matterpoll.matterpoll","wrapped_extras":"pathplugins/com.github.matterpoll.matterpoll/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 22:56:32.428 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.matterpoll.matterpoll","wrapped_extras":"address/tmp/plugin3804533052networkunixtimestamp2023-07-17T22:56:32.428Z"}
{"timestamp":"2023-07-17 22:56:32.428 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.matterpoll.matterpoll","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 22:56:32.660 Z","level":"debug","msg":"Plugin activated","caller":"plugin/environment.go:333","plugin_id":"com.github.matterpoll.matterpoll","version":"1.5.0"}
{"timestamp":"2023-07-17 22:56:32.660 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"com.github.matterpoll.matterpoll","version":"1.5.0"}
{"timestamp":"2023-07-17 22:56:33.352 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.calls","wrapped_extras":"pathplugins/com.mattermost.calls/server/dist/plugin-linux-amd64args[plugins/com.mattermost.calls/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 22:56:33.352 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.calls","wrapped_extras":"pathplugins/com.mattermost.calls/server/dist/plugin-linux-amd64pid103"}
{"timestamp":"2023-07-17 22:56:33.353 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.calls","wrapped_extras":"pathplugins/com.mattermost.calls/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 22:56:33.376 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.calls","wrapped_extras":"address/tmp/plugin4227929444networkunixtimestamp2023-07-17T22:56:33.375Z"}
{"timestamp":"2023-07-17 22:56:33.376 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.calls","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 22:56:33.396 Z","level":"debug","msg":"Initializing telemetry","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.calls","origin":"main.(*Plugin).initTelemetry telemetry.go:86"}
{"timestamp":"2023-07-17 22:56:33.398 Z","level":"debug","msg":"activating","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.calls","origin":"main.(*Plugin).OnActivate activate.go:44"}
{"timestamp":"2023-07-17 22:56:33.403 Z","level":"debug","msg":"cleaning up calls state","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.calls","origin":"main.(*Plugin).cleanUpState channel_state.go:251"}
{"timestamp":"2023-07-17 22:56:51.133 Z","level":"debug","msg":"Email batching job ran. Notifications might be still pending.","caller":"email/email_batching.go:137","number_of_users":0}

After changed tag to 7.10.4 pod was up, but after delete\apply release it not swiching in Runnig state. Logs:

mattermost@mattermost-d89c98c89-qdxf4:~$ cat /mattermost/logs/mattermost.log 
{"timestamp":"2023-07-17 23:09:13.667 Z","level":"info","msg":"Server is initializing...","caller":"platform/service.go:165","go_version":"go1.19.5"}
{"timestamp":"2023-07-17 23:09:13.667 Z","level":"info","msg":"Pinging SQL","caller":"sqlstore/store.go:242","database":"master","dataSource":"__REDACTED__?connect_timeout=3&sslmode=disable"}
{"timestamp":"2023-07-17 23:09:13.991 Z","level":"debug","msg":"Deleting any unused pre-release features","caller":"sqlstore/preference_store.go:25"}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"info","msg":"Starting websocket hubs","caller":"platform/web_hub.go:95","number_of_hubs":16}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":2}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":1}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":3}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":0}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":6}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":4}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":5}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":8}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":9}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":7}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":10}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":14}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":12}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":13}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":15}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":11}
{"timestamp":"2023-07-17 23:09:14.066 Z","level":"info","msg":"Loaded system translations","caller":"i18n/i18n.go:132","for locale":"en","from locale":"/mattermost/i18n/en.json"}
{"timestamp":"2023-07-17 23:09:14.071 Z","level":"info","msg":"Ensuring the telemetry ID..","caller":"telemetry/telemetry.go:142"}
{"timestamp":"2023-07-17 23:09:14.139 Z","level":"info","msg":"telemetry ID is set","caller":"telemetry/telemetry.go:152","id":"i4j7i9w6rtnc5qiprkm7uxtucc"}
{"timestamp":"2023-07-17 23:09:14.140 Z","level":"debug","msg":"Email batching job starting. Checking for pending emails periodically.","caller":"email/email_batching.go:90","interval_in_seconds":30}
{"timestamp":"2023-07-17 23:09:14.258 Z","level":"info","msg":"Current version is 7.10.4 (7.10.4/Wed Jul  5 09:25:31 UTC 2023/f4bc16bc8d7890dad21cae3073663d6911320a96/none)","caller":"app/server.go:412","current_version":"7.10.4","build_number":"7.10.4","build_date":"Wed Jul  5 09:25:31 UTC 2023","build_hash":"f4bc16bc8d7890dad21cae3073663d6911320a96","build_hash_enterprise":"none"}
{"timestamp":"2023-07-17 23:09:14.258 Z","level":"info","msg":"Team Edition Build","caller":"app/server.go:423","enterprise_build":false}
{"timestamp":"2023-07-17 23:09:14.259 Z","level":"info","msg":"Printing current working","caller":"app/server.go:427","directory":"/mattermost"}
{"timestamp":"2023-07-17 23:09:14.259 Z","level":"info","msg":"Loaded config","caller":"app/server.go:428","source":"postgres://%2A%2A%2A%2A:%2A%2A%2A%[email protected]:6432/mattermost?connect_timeout=3&sslmode=disable"}
{"timestamp":"2023-07-17 23:09:14.259 Z","level":"debug","msg":"Will fetch notices from","caller":"app/product_notices.go:344","url":"https://notices.mattermost.com/","skip_cache":false}
{"timestamp":"2023-07-17 23:09:14.324 Z","level":"info","msg":"Starting workers","caller":"jobs/workers.go:48"}
{"timestamp":"2023-07-17 23:09:14.324 Z","level":"debug","msg":"Cleaning up token store.","caller":"app/server.go:1262"}
{"timestamp":"2023-07-17 23:09:14.324 Z","level":"debug","msg":"Cleaning up command webhook store.","caller":"sqlstore/command_webhook_store.go:93"}
{"timestamp":"2023-07-17 23:09:14.324 Z","level":"debug","msg":"Worker Started","caller":"indexer/indexing_job.go:100","workername":"BleveIndexer"}
{"timestamp":"2023-07-17 23:09:14.324 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ProductNotices"}
{"timestamp":"2023-07-17 23:09:14.324 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"InstallNotifyAdmin"}
{"timestamp":"2023-07-17 23:09:14.324 Z","level":"debug","msg":"Worker started","caller":"resend_invitation_email/worker.go:54","worker":"resend_invitation_email"}
{"timestamp":"2023-07-17 23:09:14.324 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"HostedPurchaseScreening"}
{"timestamp":"2023-07-17 23:09:14.324 Z","level":"debug","msg":"Worker started","caller":"migrations/worker.go:50","worker":"Migrations"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"debug","msg":"Worker started","caller":"scheduler/worker.go:39","worker":"Plugins"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ExportDelete"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ExportProcess"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ExpiryNotify"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ImportDelete"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ImportProcess"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ExtractContent"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"debug","msg":"Watcher Started","caller":"jobs/jobs_watcher.go:38"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"info","msg":"Starting schedulers.","caller":"jobs/schedulers.go:47"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"debug","msg":"Scheduler: isEnabled: false","caller":"last_accessible_post/scheduler.go:20","scheduler":"last_accessible_post"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"debug","msg":"Scheduler: isEnabled: false","caller":"last_accessible_file/scheduler.go:20","scheduler":"last_accessible_file"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"debug","msg":"Scheduler: isEnabled: true","caller":"notify_admin/install_plugin_scheduler.go:20","scheduler":"install_plugin_notify_admin"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"debug","msg":"Cleaning up session store.","caller":"app/server.go:1277"}
{"timestamp":"2023-07-17 23:09:14.507 Z","level":"debug","msg":"We could not find the license key in the database or on disk at","caller":"utils/license.go:104","filename":"/mattermost/config/mattermost.mattermost-license"}
{"timestamp":"2023-07-17 23:09:14.536 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"install_plugin_notify_admin","next_runtime":"2023-07-18 23:09:15.470255335 +0000 UTC m=+86402.530400593"}
{"timestamp":"2023-07-17 23:09:14.538 Z","level":"error","msg":"License key from https://mattermost.com required to unlock enterprise features.","caller":"platform/license.go:101","error":"resource: License id: "}
{"timestamp":"2023-07-17 23:09:14.539 Z","level":"debug","msg":"Skipping license expiration check because no license is expected on Team Edition","caller":"app/server.go:1379"}
{"timestamp":"2023-07-17 23:09:14.598 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"plugins","next_runtime":"2023-07-18 23:09:15.861378681 +0000 UTC m=+86402.921523936"}
{"timestamp":"2023-07-17 23:09:14.692 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"expiry_notify","next_runtime":"2023-07-17 23:19:15.390785134 +0000 UTC m=+602.450930389"}
{"timestamp":"2023-07-17 23:09:14.772 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"product_notices","next_runtime":"2023-07-18 00:09:14.772598702 +0000 UTC m=+3601.832743959"}
{"timestamp":"2023-07-17 23:09:14.834 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"import_delete","next_runtime":"2023-07-18 23:09:15.729347941 +0000 UTC m=+86402.789493200"}
{"timestamp":"2023-07-17 23:09:14.896 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"export_delete","next_runtime":"2023-07-18 23:09:15.164408525 +0000 UTC m=+86402.224553782"}
{"timestamp":"2023-07-17 23:09:14.896 Z","level":"debug","msg":"Scheduler: isEnabled: false","caller":"notify_admin/scheduler.go:20","scheduler":"upgrade_notify_admin"}
{"timestamp":"2023-07-17 23:09:14.896 Z","level":"debug","msg":"Scheduler: isEnabled: false","caller":"notify_admin/scheduler.go:20","scheduler":"trial_notify_admin"}
{"timestamp":"2023-07-17 23:09:14.958 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"migrations","next_runtime":"2023-07-17 23:10:14.958225723 +0000 UTC m=+62.018370980"}
{"timestamp":"2023-07-17 23:09:15.735 Z","level":"info","msg":"Skipping triggering Elasticsearch channel index fix job as build is not Enterprise ready","caller":"app/migrations.go:570"}
{"timestamp":"2023-07-17 23:09:15.752 Z","level":"debug","msg":"Initializing web routes","caller":"web/web.go:26"}
{"timestamp":"2023-07-17 23:09:15.753 Z","level":"debug","msg":"Rewriting static assets","caller":"utils/subpath.go:83","from_subpath":"/","to_subpath":"/"}
{"timestamp":"2023-07-17 23:09:15.791 Z","level":"debug","msg":"Using client directory","caller":"web/static.go:34","clientDir":"/mattermost/client"}
{"timestamp":"2023-07-17 23:09:15.791 Z","level":"info","msg":"Starting up plugins","caller":"app/plugin.go:218"}
{"timestamp":"2023-07-17 23:09:15.791 Z","level":"debug","msg":"Enabling plugin health check job","caller":"plugin/environment.go:613","interval_s":"30s"}
{"timestamp":"2023-07-17 23:09:15.791 Z","level":"info","msg":"Syncing plugins from the file store","caller":"app/plugin.go:299"}
{"timestamp":"2023-07-17 23:09:15.791 Z","level":"debug","msg":"Plugin health check job starting.","caller":"plugin/health_check.go:31"}
{"timestamp":"2023-07-17 23:09:15.875 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/jitsi.tar.gz"}
{"timestamp":"2023-07-17 23:09:15.875 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/com.github.manland.mattermost-plugin-gitlab.tar.gz"}
{"timestamp":"2023-07-17 23:09:15.875 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/com.github.moussetc.mattermost.plugin.giphy.tar.gz"}
{"timestamp":"2023-07-17 23:09:15.875 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/com.github.matterpoll.matterpoll.tar.gz"}
{"timestamp":"2023-07-17 23:09:15.875 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/github.tar.gz"}
{"timestamp":"2023-07-17 23:09:15.875 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/com.mattermost.custom-attributes.tar.gz"}
{"timestamp":"2023-07-17 23:09:15.875 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/com.mattermost.calls.tar.gz"}
{"timestamp":"2023-07-17 23:09:15.875 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/jenkins.tar.gz"}
{"timestamp":"2023-07-17 23:09:15.875 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/mattermost-autolink.tar.gz"}
{"timestamp":"2023-07-17 23:09:15.875 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/com.mattermost.apps.tar.gz"}
{"timestamp":"2023-07-17 23:09:17.802 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"com.github.moussetc.mattermost.plugin.giphy","version":"2.1.0"}
{"timestamp":"2023-07-17 23:09:21.220 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"mattermost-autolink","wrapped_extras":"pathplugins/mattermost-autolink/server/dist/plugin-linux-amd64args[plugins/mattermost-autolink/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 23:09:21.220 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"mattermost-autolink","wrapped_extras":"pathplugins/mattermost-autolink/server/dist/plugin-linux-amd64pid31"}
{"timestamp":"2023-07-17 23:09:21.220 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"mattermost-autolink","wrapped_extras":"pathplugins/mattermost-autolink/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 23:09:21.379 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"mattermost-autolink","wrapped_extras":"address/tmp/plugin2339194495networkunixtimestamp2023-07-17T23:09:21.378Z"}
{"timestamp":"2023-07-17 23:09:21.379 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"mattermost-autolink","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 23:09:21.384 Z","level":"debug","msg":"Plugin activated","caller":"plugin/environment.go:333","plugin_id":"mattermost-autolink","version":"1.4.0"}
{"timestamp":"2023-07-17 23:09:21.384 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"mattermost-autolink","version":"1.4.0"}
{"timestamp":"2023-07-17 23:09:21.477 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.custom-attributes","wrapped_extras":"pathplugins/com.mattermost.custom-attributes/server/dist/plugin-linux-amd64args[plugins/com.mattermost.custom-attributes/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 23:09:21.477 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.custom-attributes","wrapped_extras":"pathplugins/com.mattermost.custom-attributes/server/dist/plugin-linux-amd64pid43"}
{"timestamp":"2023-07-17 23:09:21.477 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.custom-attributes","wrapped_extras":"pathplugins/com.mattermost.custom-attributes/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 23:09:21.843 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.custom-attributes","wrapped_extras":"address/tmp/plugin3892690795networkunixtimestamp2023-07-17T23:09:21.843Z"}
{"timestamp":"2023-07-17 23:09:21.843 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.custom-attributes","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 23:09:21.849 Z","level":"debug","msg":"2023/07/17 23:09:21 LoadPluginConfiguration API failed to unmarshal: json: cannot unmarshal string into Go struct field CustomAttribute.CustomAttributes.GroupIDs of type []string","caller":"plugin/hclog_adapter.go:54","plugin_id":"com.mattermost.custom-attributes"}
{"timestamp":"2023-07-17 23:09:21.849 Z","level":"debug","msg":"Plugin activated","caller":"plugin/environment.go:333","plugin_id":"com.mattermost.custom-attributes","version":"1.3.1"}
{"timestamp":"2023-07-17 23:09:21.849 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"com.mattermost.custom-attributes","version":"1.3.1"}
{"timestamp":"2023-07-17 23:09:25.373 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"jenkins","version":"1.1.0"}
{"timestamp":"2023-07-17 23:09:26.590 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"pathplugins/com.mattermost.apps/server/dist/plugin-linux-amd64args[plugins/com.mattermost.apps/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 23:09:26.590 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"pathplugins/com.mattermost.apps/server/dist/plugin-linux-amd64pid51"}
{"timestamp":"2023-07-17 23:09:26.590 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"pathplugins/com.mattermost.apps/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 23:09:26.617 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"address/tmp/plugin1700576250networkunixtimestamp2023-07-17T23:09:26.617Z"}
{"timestamp":"2023-07-17 23:09:26.617 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 23:09:26.983 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"github","version":"2.1.6"}
{"timestamp":"2023-07-17 23:09:27.058 Z","level":"debug","msg":"ensured bot","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","id":"rx9bx5d3ntrhumudpw3qf387ce","username":"appsbot","plugin_caller":"server/plugin.go:68"}
{"timestamp":"2023-07-17 23:09:27.071 Z","level":"debug","msg":"failed to fetch license twice. May incorrectly default to on-prem mode","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","plugin_caller":"server/config/service.go:226"}
{"timestamp":"2023-07-17 23:09:27.071 Z","level":"debug","msg":"configured the plugin.","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","commit":"44db542","build_date":"Fri 06 Jan 2023 09:59:54 PM UTC","cloud_mode":"false","developer_mode":"false","allow_http_apps":"true","version":"1.2.0","plugin_caller":"server/plugin.go:92"}
{"timestamp":"2023-07-17 23:09:27.072 Z","level":"debug","msg":"created an AWS client","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","secret":"<nil>","plugin_caller":"upstream/upaws/client.go:121","purpose":"Manifest store","region":"us-east-1","access":"<nil>"}
{"timestamp":"2023-07-17 23:09:27.072 Z","level":"debug","msg":"initialized API and persistent store","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","plugin_caller":"server/plugin.go:106"}
{"timestamp":"2023-07-17 23:09:27.072 Z","level":"debug","msg":"available upstream","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","type":"HTTP","plugin_caller":"server/proxy/service.go:232"}
{"timestamp":"2023-07-17 23:09:27.072 Z","level":"debug","msg":"Skipped upstream: AWS Lambda: not configured.","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","error":"AWS credentials are not set: not found","plugin_caller":"server/proxy/service.go:227"}
{"timestamp":"2023-07-17 23:09:27.073 Z","level":"debug","msg":"available upstream","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","type":"Mattermost Plugin","plugin_caller":"server/proxy/service.go:232"}
{"timestamp":"2023-07-17 23:09:27.073 Z","level":"debug","msg":"Skipped upstream: OpenFaaS: not configured.","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","error":"OPENFAAS_URL environment variable must be defined: not found","plugin_caller":"server/proxy/service.go:227"}
{"timestamp":"2023-07-17 23:09:27.073 Z","level":"debug","msg":"initialized the app proxy","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","plugin_caller":"server/plugin.go:122"}
{"timestamp":"2023-07-17 23:09:27.074 Z","level":"debug","msg":"initialized incoming HTTP","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","plugin_caller":"server/plugin.go:125"}
{"timestamp":"2023-07-17 23:09:27.074 Z","level":"info","msg":"activated","caller":"app/plugin_api.go:973","plugin_id":"com.mattermost.apps","plugin_caller":"server/plugin.go:135"}
{"timestamp":"2023-07-17 23:09:27.075 Z","level":"debug","msg":"Plugin activated","caller":"plugin/environment.go:333","plugin_id":"com.mattermost.apps","version":"1.2.0"}
{"timestamp":"2023-07-17 23:09:27.075 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"com.mattermost.apps","version":"1.2.0"}
{"timestamp":"2023-07-17 23:09:27.291 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"jitsi","wrapped_extras":"pathplugins/jitsi/server/dist/plugin-linux-amd64args[plugins/jitsi/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 23:09:27.317 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"jitsi","wrapped_extras":"pathplugins/jitsi/server/dist/plugin-linux-amd64pid64"}
{"timestamp":"2023-07-17 23:09:27.317 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"jitsi","wrapped_extras":"pathplugins/jitsi/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 23:09:27.329 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"jitsi","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 23:09:27.329 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"jitsi","wrapped_extras":"address/tmp/plugin1683166186networkunixtimestamp2023-07-17T23:09:27.328Z"}
{"timestamp":"2023-07-17 23:09:27.373 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.manland.mattermost-plugin-gitlab","wrapped_extras":"pathplugins/com.github.manland.mattermost-plugin-gitlab/server/dist/plugin-linux-amd64args[plugins/com.github.manland.mattermost-plugin-gitlab/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 23:09:27.374 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.manland.mattermost-plugin-gitlab","wrapped_extras":"pathplugins/com.github.manland.mattermost-plugin-gitlab/server/dist/plugin-linux-amd64pid75"}
{"timestamp":"2023-07-17 23:09:27.374 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.manland.mattermost-plugin-gitlab","wrapped_extras":"pathplugins/com.github.manland.mattermost-plugin-gitlab/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 23:09:27.391 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.manland.mattermost-plugin-gitlab","wrapped_extras":"address/tmp/plugin3158156457networkunixtimestamp2023-07-17T23:09:27.391Z"}
{"timestamp":"2023-07-17 23:09:27.391 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.manland.mattermost-plugin-gitlab","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 23:09:27.449 Z","level":"debug","msg":"Plugin activated","caller":"plugin/environment.go:333","plugin_id":"jitsi","version":"2.0.1"}
{"timestamp":"2023-07-17 23:09:27.449 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"jitsi","version":"2.0.1"}
{"timestamp":"2023-07-17 23:09:27.850 Z","level":"warn","msg":"ignored duplicate step name","caller":"app/plugin_api.go:979","plugin_id":"com.github.manland.mattermost-plugin-gitlab","name":"oauth-info","flow":"oauth"}
{"timestamp":"2023-07-17 23:09:27.850 Z","level":"debug","msg":"Plugin activated","caller":"plugin/environment.go:333","plugin_id":"com.github.manland.mattermost-plugin-gitlab","version":"1.6.0"}
{"timestamp":"2023-07-17 23:09:27.850 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"com.github.manland.mattermost-plugin-gitlab","version":"1.6.0"}
{"timestamp":"2023-07-17 23:09:30.530 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.matterpoll.matterpoll","wrapped_extras":"pathplugins/com.github.matterpoll.matterpoll/server/dist/plugin-linux-amd64args[plugins/com.github.matterpoll.matterpoll/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 23:09:30.530 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.matterpoll.matterpoll","wrapped_extras":"pathplugins/com.github.matterpoll.matterpoll/server/dist/plugin-linux-amd64pid88"}
{"timestamp":"2023-07-17 23:09:30.530 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.matterpoll.matterpoll","wrapped_extras":"pathplugins/com.github.matterpoll.matterpoll/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 23:09:30.868 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.matterpoll.matterpoll","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 23:09:30.868 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.matterpoll.matterpoll","wrapped_extras":"address/tmp/plugin2969576176networkunixtimestamp2023-07-17T23:09:30.867Z"}
{"timestamp":"2023-07-17 23:09:32.362 Z","level":"debug","msg":"Plugin activated","caller":"plugin/environment.go:333","plugin_id":"com.github.matterpoll.matterpoll","version":"1.5.0"}
{"timestamp":"2023-07-17 23:09:32.362 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"com.github.matterpoll.matterpoll","version":"1.5.0"}
{"timestamp":"2023-07-17 23:09:33.394 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.calls","wrapped_extras":"pathplugins/com.mattermost.calls/server/dist/plugin-linux-amd64args[plugins/com.mattermost.calls/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 23:09:33.394 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.calls","wrapped_extras":"pathplugins/com.mattermost.calls/server/dist/plugin-linux-amd64pid101"}
{"timestamp":"2023-07-17 23:09:33.394 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.calls","wrapped_extras":"pathplugins/com.mattermost.calls/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 23:09:33.417 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.calls","wrapped_extras":"address/tmp/plugin543213083networkunixtimestamp2023-07-17T23:09:33.416Z"}
{"timestamp":"2023-07-17 23:09:33.417 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.calls","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 23:09:33.424 Z","level":"debug","msg":"Initializing telemetry","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.calls","origin":"main.(*Plugin).initTelemetry telemetry.go:86"}
{"timestamp":"2023-07-17 23:09:33.426 Z","level":"debug","msg":"activating","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.calls","origin":"main.(*Plugin).OnActivate activate.go:44"}
{"timestamp":"2023-07-17 23:09:33.432 Z","level":"debug","msg":"cleaning up calls state","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.calls","origin":"main.(*Plugin).cleanUpState channel_state.go:251"}
{"timestamp":"2023-07-17 23:09:44.141 Z","level":"debug","msg":"Email batching job ran. Notifications might be still pending.","caller":"email/email_batching.go:137","number_of_users":0}
{"timestamp":"2023-07-17 23:10:14.140 Z","level":"debug","msg":"Email batching job ran. Notifications might be still pending.","caller":"email/email_batching.go:137","number_of_users":0}
{"timestamp":"2023-07-17 23:10:15.023 Z","level":"debug","msg":"Scheduling Job","caller":"migrations/scheduler.go:45","scheduler":"migrations"}
{"timestamp":"2023-07-17 23:10:15.057 Z","level":"debug","msg":"All migrations are complete.","caller":"migrations/scheduler.go:84","scheduler":"migrations"}
{"timestamp":"2023-07-17 23:10:15.089 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"migrations","next_runtime":"<nil>"}
{"timestamp":"2023-07-17 23:10:44.141 Z","level":"debug","msg":"Email batching job ran. Notifications might be still pending.","caller":"email/email_batching.go:137","number_of_users":0}
{"timestamp":"2023-07-17 23:11:14.141 Z","level":"debug","msg":"Email batching job ran. Notifications might be still pending.","caller":"email/email_batching.go:137","number_of_users":0}

If i disable calls plugin - it woks fine.
@mustdiechik
Copy link
Contributor Author

An interesting behavior: on different clusters, it takes different times for the transition to the Running state. From 40 second to up to 4 minutes.
I had similar issue with rabbitMQ release in minikube. It looks like the application can not give the load on the processor. The problem was resolved after I changed cpu governor to "performance" mode.
Maybe there is a similar problem here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mustdiechik