Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[🐛 Bug]: Standalone Chrome does not work with readOnlyRootFilesystem = true on Kubernetes #2648

Open
majusmisiak opened this issue Feb 7, 2025 · 2 comments

Comments

@majusmisiak
Copy link

What happened?

I am attempting to run Selenium inside security hardened Kubernetes cluster. Current policy does now allow read-write access to the root filesystem of the container.

After deploying Selenium with read-only filesystem, the container will run, but it is not possible to connect to it using external client.

The only breaking flag in below configuration is

readOnlyRootFilesystem: true

If I deploy exact same configuration as attached below, just changing the flag to false, everything will work fine.

It seems that Selenium container under the hood writes (unrestricted) to all kind of various filesystem paths. So this issue could be solved by allowing specify separate list of emptyDir volumes mounted in all locations that selenium container writes / caches data to. Searching through the documentation and open issues on github, I did not found a list of folders that Selenium container is writing to, and the purpose of each writable volume. Please assist :)

Reproduction - deploy selenium using attached manifest, then run following script:

Client:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import uuid

SELENIUM_GRID_URL = "http://localhost:4444/wd/hub"
TEMP_PROFILE_DIR = f"/tmp/{uuid.uuid4()}"

chrome_options = Options()
chrome_options.add_argument("--disable-infobars")
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-popup-blocking")
chrome_options.add_argument('--headless')
chrome_options.add_argument(f"--user-data-dir={TEMP_PROFILE_DIR}")

browser = webdriver.Remote(
    command_executor=SELENIUM_GRID_URL,
    options=chrome_options
)
browser.set_page_load_timeout(10)
browser.get("https://bing.com")

Result:

Traceback (most recent call last):
  File "/home/use/demo/repro_sel_minimal.py", line 15, in <module>
    browser = webdriver.Remote(
              ^^^^^^^^^^^^^^^^^
  File "/home/use/demo/.venv/lib/python3.12/site-packages/selenium/webdriver/remote/webdriver.py", line 250, in __init__
    self.start_session(capabilities)
  File "/home/use/demo/.venv/lib/python3.12/site-packages/selenium/webdriver/remote/webdriver.py", line 342, in start_session
    response = self.execute(Command.NEW_SESSION, caps)["value"]
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/use/demo/.venv/lib/python3.12/site-packages/selenium/webdriver/remote/webdriver.py", line 429, in execute
    self.error_handler.check_response(response)
  File "/home/use/demo/.venv/lib/python3.12/site-packages/selenium/webdriver/remote/errorhandler.py", line 232, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: Could not start a new session. No nodes support the capabilities in the request 
Host info: host: 'chromedriver-6f448dcfbb-7vbhk', ip: '172.20.8.192'
Build info: version: '4.28.1', revision: '73f5ad48a2'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-112-generic', java.version: '17.0.13'
Driver info: driver.version: unknown
Stacktrace:
    at org.openqa.selenium.grid.distributor.local.LocalDistributor$NewSessionRunnable.checkMatchingSlot (LocalDistributor.java:841)
    at org.openqa.selenium.grid.distributor.local.LocalDistributor$NewSessionRunnable.run (LocalDistributor.java:823)
    at org.openqa.selenium.concurrent.GuardedRunnable.lambda$guard$0 (GuardedRunnable.java:35)
    at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
    at java.util.concurrent.FutureTask.runAndReset (FutureTask.java:305)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run (ScheduledThreadPoolExecutor.java:305)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
    at java.lang.Thread.run (Thread.java:840)

Command used to start Selenium Grid with Docker (or Kubernetes)

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: run
    app.kubernetes.io/name: chromedriver
  name: chromedriver
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: run
      app.kubernetes.io/name: chromedriver
  strategy:
    rollingUpdate: null
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/component: run
        app.kubernetes.io/name: chromedriver
    spec:
      containers:
        - env:
            - name: "SCREEN_WIDTH"
              value: "1920"
            - name: "SCREEN_HEIGHT"
              value: "1080"
            - name: "SCREEN_DEPTH"
              value: "24"
            - name: "SE_NODE_GRID_URL"
              value: "http://localhost:4444"
            - name: "SE_NODE_SESSION_TIMEOUT"
              value: "600"
            - name: "SE_ENABLE_TRACING"
              value: "false"
            - name: "SE_NEW_SESSION_THREAD_POOL_SIZE"
              value: "32"
            - name: "SE_NODE_OVERRIDE_MAX_SESSIONS"
              value: "true"
            - name: "SE_NODE_MAX_SESSIONS"
              value: "4"
            - name: "JAVA_OPTS"
              value: "-XX:ActiveProcessorCount=4"
            - name: "SEL_UID"
              value: "1200"
            - name: "SEL_GID"
              value: "1201"
          image: selenium/standalone-chrome:4.28.1-20250202
          imagePullPolicy: "Always"
          name: "selenium"
          ports:
            - containerPort: 4444
              protocol: "TCP"
          resources:
            limits:
              memory: "4Gi"
              cpu: "2000m"
              ephemeral-storage: "2Gi"
            requests:
              memory: "4Gi"
              cpu: "500m"
              ephemeral-storage: "2Gi"
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - "ALL"
            readOnlyRootFilesystem: true
            runAsGroup: 1201
            runAsNonRoot: true
            runAsUser: 1200
            seccompProfile:
              type: "RuntimeDefault"
          volumeMounts:
            - mountPath: "/dev/shm"
              name: "dshm"
            - mountPath: "/tmp"
              name: "tmp"
      nodeSelector:
        kubernetes.io/os: "linux"
      restartPolicy: "Always"
      securityContext:
        fsGroup: 1201
        runAsGroup: 1201
        runAsNonRoot: true
        runAsUser: 1200
      volumes:
        - emptyDir: {}
          name: "tmp"
        - emptyDir:
            medium: "Memory"
            sizeLimit: "2Gi"
          name: "dshm"

Relevant log output

2025-02-07 12:18:23,274 INFO Included extra file "/etc/supervisor/conf.d/chrome-cleanup.conf" during parsing
2025-02-07 12:18:23,275 INFO Included extra file "/etc/supervisor/conf.d/recorder.conf" during parsing
2025-02-07 12:18:23,275 INFO Included extra file "/etc/supervisor/conf.d/selenium.conf" during parsing
2025-02-07 12:18:23,275 INFO Included extra file "/etc/supervisor/conf.d/uploader.conf" during parsing
2025-02-07 12:18:23,278 INFO RPC interface 'supervisor' initialized
2025-02-07 12:18:23,278 INFO supervisord started with pid 8
2025-02-07 12:18:24,281 INFO spawnerr: unknown error making dispatchers for 'xvfb': EROFS
2025-02-07 12:18:24,282 INFO spawnerr: unknown error making dispatchers for 'vnc': EROFS
2025-02-07 12:18:24,282 INFO spawnerr: unknown error making dispatchers for 'novnc': EROFS
2025-02-07 12:18:24,283 INFO spawned: 'selenium-standalone' with pid 9
E: [pulseaudio] main.c: Daemon startup failed.
2025-02-07 12:18:24,299 INFO gave up: xvfb entered FATAL state, too many start retries too quickly
2025-02-07 12:18:24,299 INFO gave up: vnc entered FATAL state, too many start retries too quickly
2025-02-07 12:18:24,299 INFO gave up: novnc entered FATAL state, too many start retries too quickly
2025-02-07 12:18:24,299 INFO success: selenium-standalone entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
Failed to create secure directory (/home/seluser/.config/pulse): Read-only file system
No PulseAudio daemon running, or not running as session daemon.
Failed to create secure directory (/home/seluser/.config/pulse): Read-only file system
No PulseAudio daemon running, or not running as session daemon.
Failed to create secure directory (/home/seluser/.config/pulse): Read-only file system
No PulseAudio daemon running, or not running as session daemon.
Appending Selenium option: --heartbeat-period 30
Appending Selenium option: --log-level INFO
Appending Selenium option: --http-logs false
Appending Selenium option: --structured-logs false
Appending Selenium option: --reject-unsupported-caps true
Appending Selenium option: --newsession-threadpool-size 32
/opt/bin/generate_config: line 19: /opt/selenium/config.toml: Read-only file system
/opt/bin/generate_config: line 21: /opt/selenium/config.toml: Read-only file system
/opt/bin/generate_config: line 26: /opt/selenium/config.toml: Read-only file system
/opt/bin/generate_config: line 28: /opt/selenium/config.toml: Read-only file system
/opt/bin/generate_config: line 29: /opt/selenium/config.toml: Read-only file system
/opt/bin/generate_config: line 30: /opt/selenium/config.toml: Read-only file system
/opt/bin/generate_config: line 31: /opt/selenium/config.toml: Read-only file system
/opt/bin/generate_config: line 33: /opt/selenium/config.toml: Read-only file system
/opt/bin/generate_config: line 61: /opt/selenium/config.toml: Read-only file system
/opt/bin/generate_config: line 62: /opt/selenium/config.toml: Read-only file system
/opt/bin/generate_config: line 63: /opt/selenium/config.toml: Read-only file system
/opt/bin/generate_config: line 65: /opt/selenium/config.toml: Read-only file system
Selenium Grid Standalone configuration: 
Starting Selenium Grid Standalone...
Appending Selenium option: --tracing false
Tracing is disabled
Using JAVA_OPTS: -XX:ActiveProcessorCount=4
Feb 07, 2025 12:18:24 PM org.openqa.selenium.grid.log.LoggingOptions getTracer
INFO: Using null tracer
Feb 07, 2025 12:18:25 PM org.openqa.selenium.grid.log.LoggingOptions getTracer
INFO: Using null tracer
Feb 07, 2025 12:18:25 PM org.openqa.selenium.grid.node.config.NodeOptions getSessionFactories
INFO: Detected 4 available processors
Feb 07, 2025 12:18:25 PM org.openqa.selenium.grid.node.Node <init>
INFO: Binding additional locator mechanisms: relative
Feb 07, 2025 12:18:25 PM org.openqa.selenium.grid.commands.Standalone execute
INFO: Started Selenium Standalone 4.28.1 (revision 73f5ad48a2): http://172.20.8.192:4444
Feb 07, 2025 12:18:28 PM org.openqa.selenium.grid.distributor.local.LocalDistributor$NewSessionRunnable checkMatchingSlot
INFO: No nodes support the capabilities in the request: [Capabilities {browserName: chrome, goog:chromeOptions: {args: [--disable-infobars, --disable-extensions, --disable-popup-blocking, --headless, --user-data-dir=/tmp/0decd9...], extensions: []}, pageLoadStrategy: normal}]
Feb 07, 2025 12:18:31 PM org.openqa.selenium.grid.distributor.local.LocalDistributor$NewSessionRunnable checkMatchingSlot
INFO: No nodes support the capabilities in the request: [Capabilities {browserName: chrome, goog:chromeOptions: {args: [--disable-infobars, --disable-extensions, --disable-popup-blocking, --headless, --user-data-dir=/tmp/947d2d...], extensions: []}, pageLoadStrategy: normal}]
Feb 07, 2025 12:18:55 PM org.openqa.selenium.concurrent.GuardedRunnable lambda$guard$0
WARNING: Unable to execute task 
java.lang.IllegalArgumentException: Make sure that a driver is available on $PATH
	at org.openqa.selenium.internal.Require.positive(Require.java:136)
	at org.openqa.selenium.grid.data.NodeStatus.<init>(NodeStatus.java:62)
	at org.openqa.selenium.grid.node.local.LocalNode.getStatus(LocalNode.java:1015)
	at org.openqa.selenium.grid.node.local.LocalNode.lambda$new$12(LocalNode.java:305)
	at org.openqa.selenium.concurrent.GuardedRunnable.lambda$guard$0(GuardedRunnable.java:35)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Operating System

Kubernetes Server Version: v1.29.4+vmware.3-fips.1

Docker Selenium version (image tag)

4.28.1-20250202

Selenium Grid chart version (chart version)

Standalone (no chart)

Copy link

github-actions bot commented Feb 7, 2025

@majusmisiak, thank you for creating this issue. We will troubleshoot it as soon as we can.


Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

@VietND96
Copy link
Member

VietND96 commented Feb 7, 2025

I think this is similar to this #2382, which is still not support fully.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants