7.1.0 doesn't run as non-root while 7.0.0 does #34840

jeanfabrice · 2024-12-27T16:31:58Z

Description:

After upgrading to 7.1.0, Rocketchat doesn't start anymore on Kubernetes when running as non root with the following securityContext:

          allowPrivilegeEscalation: false
          runAsUser: 1000
          runAsGroup: 1000
          capabilities:
            drop:
            - ALL
          privileged: false
          readOnlyRootFilesystem: true

Error thrown:

error on boot.js Error: EACCES: permission denied, open '/app/bundle/programs/server/npm/node_modules/meteor/webapp/node_modules/useragent-ng/lib/regexps.js'
    at Object.readFileSync (node:fs:448:20)
    at getMaybeCachedSource (node:internal/modules/cjs/loader:1492:18)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1504:19)
    at Module.load (node:internal/modules/cjs/loader:1288:32)
    at Module.Mp.load (/tools/static-assets/server/runtime.js:35:31)
    at Function.Module._load (node:internal/modules/cjs/loader:1104:12)
    at Module.require (node:internal/modules/cjs/loader:1311:19)
    at Module.mod.require (/app/bundle/programs/server/node_modules/@meteorjs/reify/lib/runtime/index.js:30:33)
    at require (node:internal/modules/helpers:179:18)
    at Object.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/webapp/node_modules/useragent-ng/index.js:6:15)
    at Module._compile (node:internal/modules/cjs/loader:1469:14)
    at Module.Mp._compile (/tools/static-assets/server/runtime.js:78:21)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1548:10)
    at Module.load (node:internal/modules/cjs/loader:1288:32)
    at Module.Mp.load (/tools/static-assets/server/runtime.js:35:31)
    at Function.Module._load (node:internal/modules/cjs/loader:1104:12) {
  errno: -13,
  code: 'EACCES',
  syscall: 'open',
  path: '/app/bundle/programs/server/npm/node_modules/meteor/webapp/node_modules/useragent-ng/lib/regexps.js'
}

The root cause seems to be Meteor upgrade to 3.0.4 where the Meteor webapp component switched a dependency from [email protected] to [email protected]

For an unknown reason, useragent-ng's /lib/regexps.js is shipped with unix 0600 access mode:

Steps to reproduce:

$ kubectl run test  --image rocketchat/rocket.chat:7.1.0 --command -- sh -c "ls -l /app/bundle/programs/server/npm/node_modules/meteor/webapp/node_modules/useragent-ng/lib/regexps.js"
pod/test created
$ kubectl logs test
-rw-------    1 root     root         99939 Dec  4 01:11 /app/bundle/programs/server/npm/node_modules/meteor/webapp/node_modules/useragent-ng/lib/regexps.js

Expected behavior:

/app/bundle/programs/server/npm/node_modules/meteor/webapp/node_modules/useragent-ng/lib/regexps.js to be 0644 mode like other files in /app/bundle/programs/server/npm/node_modules/meteor/webapp/node_modules/useragent-ng/lib

Actual behavior:

/app/bundle/programs/server/npm/node_modules/meteor/webapp/node_modules/useragent-ng/lib/regexps.js is 0600 and make the container crash at start time when the container is run as non root user

Server Setup Information:

Version of Rocket.Chat Server: 7.1.0
License Type: non paying
Number of Users: 10
Operating System: Kubernetes
Deployment Method: container image
Number of Running Instances: 1
DB Replicaset Oplog:
NodeJS Version: embedded in container
MongoDB Version: 6.0.19

Additional context

I understand this is not directly under your scope, useragent-ng being a transitive dependency of meteor/webapp, which is itself a dependency of rocketchat.
However, since it directly affects rocketchat and prevent further upgrade in the current state, I believe you would like to know. Not sure how you can ask Meteor's developers to ask useragent-ng's developers to fix the situation.

FWIT, II created schmod/useragent-ng#6

Relevant logs:

$ k get pod
NAME                          READY   STATUS             RESTARTS         AGE
mongodb-6854f76f4b-qpxv8      1/1     Running            39 (3d22h ago)   45d
rocketchat-64c87bfccb-5g77r   0/1     CrashLoopBackOff   1 (17s ago)      34s
$ klf rocketchat-64c87bfccb-5g77r                                                                                                                                    
error on boot.js Error: EACCES: permission denied, open '/app/bundle/programs/server/npm/node_modules/meteor/webapp/node_modules/useragent-ng/lib/regexps.js'
    at Object.readFileSync (node:fs:448:20)
    at getMaybeCachedSource (node:internal/modules/cjs/loader:1492:18)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1504:19)
    at Module.load (node:internal/modules/cjs/loader:1288:32)
    at Module.Mp.load (/tools/static-assets/server/runtime.js:35:31)
    at Function.Module._load (node:internal/modules/cjs/loader:1104:12)
    at Module.require (node:internal/modules/cjs/loader:1311:19)
    at Module.mod.require (/app/bundle/programs/server/node_modules/@meteorjs/reify/lib/runtime/index.js:30:33)
    at require (node:internal/modules/helpers:179:18)
    at Object.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/webapp/node_modules/useragent-ng/index.js:6:15)
    at Module._compile (node:internal/modules/cjs/loader:1469:14)
    at Module.Mp._compile (/tools/static-assets/server/runtime.js:78:21)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1548:10)
    at Module.load (node:internal/modules/cjs/loader:1288:32)
    at Module.Mp.load (/tools/static-assets/server/runtime.js:35:31)
    at Function.Module._load (node:internal/modules/cjs/loader:1104:12) {
  errno: -13,
  code: 'EACCES',
  syscall: 'open',
  path: '/app/bundle/programs/server/npm/node_modules/meteor/webapp/node_modules/useragent-ng/lib/regexps.js'
}
Error: EACCES: permission denied, open '/app/bundle/programs/server/npm/node_modules/meteor/webapp/node_modules/useragent-ng/lib/regexps.js'
    at Object.readFileSync (node:fs:448:20)
    at getMaybeCachedSource (node:internal/modules/cjs/loader:1492:18)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1504:19)
    at Module.load (node:internal/modules/cjs/loader:1288:32)
    at Module.Mp.load (/tools/static-assets/server/runtime.js:35:31)
    at Function.Module._load (node:internal/modules/cjs/loader:1104:12)
    at Module.require (node:internal/modules/cjs/loader:1311:19)
    at Module.mod.require (/app/bundle/programs/server/node_modules/@meteorjs/reify/lib/runtime/index.js:30:33)
    at require (node:internal/modules/helpers:179:18)
    at Object.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/webapp/node_modules/useragent-ng/index.js:6:15)
    at Module._compile (node:internal/modules/cjs/loader:1469:14)
    at Module.Mp._compile (/tools/static-assets/server/runtime.js:78:21)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1548:10)
    at Module.load (node:internal/modules/cjs/loader:1288:32)
    at Module.Mp.load (/tools/static-assets/server/runtime.js:35:31)
    at Function.Module._load (node:internal/modules/cjs/loader:1104:12)

The text was updated successfully, but these errors were encountered:

reetp · 2024-12-27T21:32:24Z

Thanks.

I'll refer this to the devs.

tcoupin · 2025-01-07T05:22:10Z

Hi,
same behavior on the new 7.2.0

reetp · 2025-01-07T15:44:28Z

Hit the same thing in docker upgrading 7.0.3 to 7.1.0

reetp · 2025-01-07T15:54:17Z

Anyone seen this as well on 7.0.4 ?

Subprocess stderr Download https://jsr.io/@std/cli/1.0.9_meta.json

Subprocess stderr error: Permission denied (os error 13) (for '/app/bundle/programs/server/npm/node_modules/@rocket.chat/apps-engine/.deno-cache/deps/https/jsr.io/c29b640a904c999165944cb8fa91773b04e6aababb971a93737582c5ecdb16a3')
at file:///app/bundle/programs/server/npm/node_modules/@rocket.chat/apps-engine/deno-runtime/lib/parseArgs.ts:1:41

reetp · 2025-01-07T16:05:15Z

Hmmm. I am wondering if this is OS related?

Mine is with docker on CentOS 7 (which is due to get upgrade soon, but not just yet). Rocket is the only thing it runs.

reetp · 2025-01-07T16:11:57Z

Back to 7.0.3 and all quiet......

jeanfabrice · 2025-01-07T16:46:27Z

Hey @reetp , I'm not entirely sure your last comments are directly related to the initial issue where, imo, the root cause is pretty clear and pinpoints to a transitive dependency Rocketchat < Meteor < useragent-ng, what do you think?

reetp · 2025-01-07T17:46:42Z

You don't need to '@' me thanks. I read most stuff.

Can you confirm your OS that you are using?

the root cause is pretty clear and pinpoints to a transitive dependency Rocketchat < Meteor < useragent-ng

Possibly, but it may equally be a symptom of something else rather than the cause. Note I am no code guru, but I have triaged plenty of stuff here.

So the first error I got with 7.1.0 was the same as yours.

  path: '/app/bundle/programs/server/npm/node_modules/meteor/webapp/node_modules/useragent-ng/lib/regexps.js'
}
Error: EACCES: permission denied, open '/app/bundle/programs/server/npm/node_modules/meteor/webapp/node_modules/useragent-ng/lib/regexps.js'

I then backtracked fast to 7.0.4 as I didn't have time to mess about - I remember looking at perms/deno related stuff having insufficient perms in the container and hit the second issue which seems related - again, permissions issues in the container.

I can also see various issues logged for containerd and suspect that is probably related. As soon as I went back to 7.0.3 they disappeared again.

Something has changed/backported to 7.0.4 to match whatever happened in 7.1.0

Likely one of the 'fixes' here that were also in 7.1.0

https://github.com/RocketChat/Rocket.Chat/releases/tag/7.0.4

I vaguely remember seeing similar container permissions issues elsewhere some years back - when they updated cgroups or something and then the containers couldn't give the right perms. Feels very similar.

I have asked the team about it.

Let us know about your OS please.

jeanfabrice · 2025-01-07T18:05:36Z

7.0.4 is running fine, same securityContext as usual:

+-------------------------------------------------------------------+
|                           SERVER RUNNING                          |
+-------------------------------------------------------------------+
|                                                                   |
|  Rocket.Chat Version: 7.0.4                                       |
|       NodeJS Version: 20.17.0 - x64                               |
|      MongoDB Version: 6.0.19                                      |
|       MongoDB Engine: wiredTiger                                  |
|             Platform: linux                                       |
|         Process Port: 3000                                        |
|             Site URL: <redacted>                                  |
|     ReplicaSet OpLog: Enabled                                     |
|          Commit Hash: 725c414372                                  |
|        Commit Branch: HEAD                                        |
|                                                                   |
+-------------------------------------------------------------------+

It only breaks with 7.1.0+, which is concomitant with 3.0.4 Meteor upgrade as stated earlier.

Can you confirm your OS that you are using?

Kubernetes version is v1.31.4 running on Ubuntu Ubuntu 22.04.4 LTS.
Not sure how it matters as I'm using the official 7.0.4 docker image

$ kubectl get pod -l app.kubernetes.io/name=rocketchat -o jsonpath="{.items[*].spec.containers[*].image}"
rocketchat/rocket.chat:7.0.4

Thanks!

debdutdeb · 2025-01-09T09:14:11Z

meteor/meteor@d3d1b92 likely the source.

debdutdeb · 2025-01-09T09:23:09Z

Discussing what can be done.

For now though, we do have #34162 this going, which will enable non-root runtime, but any uid setting other than 65533 will still be broken.

jeanfabrice · 2025-01-09T09:30:20Z

ideally, we should be able to run Rocketchat with whatever non root uid we choose, especially in a Kubernetes/containerized environment. Moreover, the problem seems to be more on file permissions, than on effective user-id.
I agree and I understand this is out of your control and should be addressed either on the meteor side or on the useragent-ng maintainer side. Thanks much!

reetp added the Tasked Added to the internal issue tracking label Dec 27, 2024

debdutdeb mentioned this issue Jan 9, 2025

Switch to useragent-ng on meteor 3.0.4 causes permission issues at runtime meteor/meteor#13539

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

7.1.0 doesn't run as non-root while 7.0.0 does #34840

7.1.0 doesn't run as non-root while 7.0.0 does #34840

jeanfabrice commented Dec 27, 2024

reetp commented Dec 27, 2024

tcoupin commented Jan 7, 2025

reetp commented Jan 7, 2025

reetp commented Jan 7, 2025

reetp commented Jan 7, 2025

reetp commented Jan 7, 2025

jeanfabrice commented Jan 7, 2025

reetp commented Jan 7, 2025

jeanfabrice commented Jan 7, 2025

debdutdeb commented Jan 9, 2025

debdutdeb commented Jan 9, 2025

jeanfabrice commented Jan 9, 2025

7.1.0 doesn't run as non-root while 7.0.0 does #34840

7.1.0 doesn't run as non-root while 7.0.0 does #34840

Comments

jeanfabrice commented Dec 27, 2024

Description:

Steps to reproduce:

Expected behavior:

Actual behavior:

Server Setup Information:

Additional context

Relevant logs:

reetp commented Dec 27, 2024

tcoupin commented Jan 7, 2025

reetp commented Jan 7, 2025

reetp commented Jan 7, 2025

reetp commented Jan 7, 2025

reetp commented Jan 7, 2025

jeanfabrice commented Jan 7, 2025

reetp commented Jan 7, 2025

jeanfabrice commented Jan 7, 2025

debdutdeb commented Jan 9, 2025

debdutdeb commented Jan 9, 2025

jeanfabrice commented Jan 9, 2025