Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(overlord): pebble hangs with a read-only file system when starting the daemon #481

Conversation

IronCore864
Copy link
Contributor

@IronCore864 IronCore864 commented Aug 12, 2024

Fix an issue where the Pebble daemon fails to start with a read-only file system.

Closes #462.

Changes:

  • Check if Pebble dir is writable before creating the overlord state backend.
  • Add some logging when backend writes fail to enhance user experience.
  • Listen for interrupt signals when state unlock retries because of backend writes failure.

@IronCore864 IronCore864 marked this pull request as ready for review August 12, 2024 09:21
internals/overlord/overlord.go Outdated Show resolved Hide resolved
internals/overlord/state/state.go Outdated Show resolved Hide resolved
internals/overlord/state/state.go Outdated Show resolved Hide resolved
Copy link
Contributor

@benhoyt benhoyt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thanks!

@benhoyt
Copy link
Contributor

benhoyt commented Aug 13, 2024

Looks good, let's go ahead and merge @IronCore864

@IronCore864 IronCore864 merged commit 51e64e4 into canonical:master Aug 13, 2024
17 checks passed
@IronCore864 IronCore864 deleted the pebble-hangs-state-dir-no-write-permission branch August 13, 2024 04:11
jujubot added a commit to juju/juju that referenced this pull request Aug 29, 2024
#17991

Per discussion with @hpidcock, update Juju 3.6 to use the just-released Pebble v1.16.0 ([changelog](https://github.com/canonical/pebble/releases/tag/v1.16.0)). Summary of main changes:

* Improvements to how services with dependencies are started and stopped (using "lanes"), so that independent services are started in parallel, and dependent services start up serially. Error handling is also improved. Fix in canonical/pebble#437.
* A fix for a tricky bug in pebble exec, so it doesn't lose output in interactive mode. Fix in canonical/pebble#466!
 prdesc Reduce the size of exec tasks by not storing the execSetup (which includes all environment variables) in state. This speeds up (and shrinks the data) when serialising state to disk during state.Unlock. Fix is in canonical/pebble#478.
 prdesc Ensure Pebble doesn't hang (with no error message) when the state file directory is read-only or otherwise inaccessible. Fix in canonical/pebble#481.
* Re-implement warnings using notices. This was always the intention since the notices feature was added (it was designed as a superset of warnings), but we'd never gotten to it. Lots of nice code deletion in canonical/pebble#486.

## QA steps

Deploy a K8s charm, like `snappass-test`, and ensure `pebble version --client` on the workload reports v1.16.0.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Pebble silently fails on a readOnlyRootFilesystem
2 participants