[PAL,LibOS,common] Add file recovery support for encrypted files #2082
Conversation
|
Jenkins, retest Jenkins-Direct-24.04-Debug please (fdatasync01 from LTP timed out, known and unrelated to the PR) |
|
|
||
| /* Whether to enable file recovery (used by `chroot_encrypted` filesystem), false if not | ||
| * applicable */ | ||
| bool enable_recovery; |
There was a problem hiding this comment.
what is the behavior if recovery is enabled, disabled and re-enabled?
do you remove the old shadow files when mounting?
There was a problem hiding this comment.
In the first "recovery enabled" run, if the app terminates abruptly, a shadow file will be generated. If recovery is "disabled" in the next run, the shadow file will remain and will not be accessed. When recovery is "re-enabled" in a subsequent run, the recovery file will not be removed upon mounting. However, it will be overwritten during flush() and removed upon closing.
There was a problem hiding this comment.
I think you can't assume that you can replay the shadow file once the flag is turned off, if new data is written to the same offset in the file when the flag is off and then you re-enable it - the file won't be consistent.
There was a problem hiding this comment.
Hmm... Good point. Do we consider this a legitimate usage? If so, alternatively, we could use enable_recovery to control whether a backup is needed on flush, but still perform the file recovery process as long as the update_flag is set (i.e., we restore to the last known good state even during a "disabled" run if a previous run was abruptly terminated).
There was a problem hiding this comment.
I think a better approach is either not allowing non-recoverable mounts after mounting once with recovery, or removing the shadow files in the above scenario since the user has (hopefully knowingly) disabled recovery.
There was a problem hiding this comment.
Thanks for the input! I go w/ the first approach -- not allowing non-recoverable mounts if a recovery is needed.
| * | ||
| * `uri` must not correspond to an existing file. | ||
| * | ||
| * The newly created `libos_encrypted_file` object will have `use_count` set to 1. | ||
| */ | ||
| int encrypted_file_create(const char* uri, mode_t perm, struct libos_encrypted_files_key* key, | ||
| struct libos_encrypted_file** out_enc); | ||
| bool enable_recovery, struct libos_encrypted_file** out_enc); |
There was a problem hiding this comment.
you may want to have some const fs_configuration struct that's passed around, assuming more flags are going to be introduced in the future
There was a problem hiding this comment.
Yeah, I'm okay with having a fs_configuration struct, but currently, we only have this enable_recovery option.
There was a problem hiding this comment.
I left it as is for now.
| } | ||
|
|
||
| for (size_t i = 0; i < nodes_count; i++) { | ||
| ret = read_all(recovery_file_fd, recovery_node, recovery_node_size); |
There was a problem hiding this comment.
Is there an upper bound for the amount of data written in a shadow file?
do you attempt to allocate memory for and read from the full shadow file here?
There was a problem hiding this comment.
Is there an upper bound for the amount of data written in a shadow file?
The upper bound for the data that can be written to a shadow file should be the same as on a typical Linux system. This is determined by e.g., the fs type, available disk space, and the maximum file size supported by the fs.
do you attempt to allocate memory for and read from the full shadow file here?
Sorry, I don't understand this concern. Pls note that this piece of code is on the untrusted side of Gramine.
There was a problem hiding this comment.
If I understand correctly, there's a malloc call which might attempt to allocate TBs if the size is not limited
There was a problem hiding this comment.
Ah, I understand your concern now. I was referring to the theoretical upper bound earlier. Actually, during each flush, only the data that is about to change in the encrypted files cache (which has a default size of 192KB as specified here) will be saved and rewritten to the recovery file.
There was a problem hiding this comment.
Reviewed 26 of 27 files at r1, all commit messages.
Reviewable status: 26 of 27 files reviewed, 8 unresolved discussions, not enough approvals from maintainers (1 more required), not enough approvals from different teams (1 more required, approved so far: Intel) (waiting on @kailun-qin and @ynonflumintel)
libos/src/fs/libos_fs_encrypted.c line 277 at r1 (raw file):
if (recovery_needed) { log_warning("file recovery needed but failed");
suggest changing the message to 'file recovery attempted but failed' for clarity.
libos/src/fs/libos_fs_encrypted.c line 328 at r1 (raw file):
if (enc->recovery_file_pal_handle) (void)PalStreamDelete(enc->recovery_file_pal_handle, PAL_DELETE_ALL);
wondering if PalStreamDelete(enc->recovery_file_pal_handle, ..) should be invoked as well when pf_close() fails above?
common/src/protected_files/protected_files.c line 452 at r1 (raw file):
assert(pf->host_recovery_file_handle); uint64_t offset = 0;
nitpicking: maybe moving the 'offset' declaration a few lines back, together with 'node'
common/src/protected_files/protected_files.c line 522 at r1 (raw file):
pf->file_status = PF_STATUS_FLUSH_ERROR; DEBUG_PF("failed to write changes to the recovery file"); return false;
maybe use "goto recoverable_error;" instead for consistency?
common/src/protected_files/protected_files.c line 528 at r1 (raw file):
pf->file_status = PF_STATUS_FLUSH_ERROR; DEBUG_PF("failed to set the update flag"); return false;
same as above
There was a problem hiding this comment.
Reviewed 8 of 27 files at r1, all commit messages.
Reviewable status: 26 of 27 files reviewed, 20 unresolved discussions, not enough approvals from maintainers (1 more required), not enough approvals from different teams (1 more required, approved so far: Intel) (waiting on @kailun-qin and @ynonflumintel)
a discussion (no related file):
We probably should benchmark this before merging (see #2013 (comment)).
a discussion (no related file):
Do I understand correctly (at least looking at #2013 (comment)) that the recovery file stacks all changes that happened while the file was open? This may be a lot for long-running enclaves, even if the file on the disk is itself small, but often modified?
common/src/protected_files/protected_files.h line 222 at r1 (raw file):
* \param create Overwrite file contents if true. * \param key Wrap key. * \param recovery_file_handle (optional)Underlying recovery file handle.
Suggestion:
(optional) Underlying recovery file handle.common/src/protected_files/protected_files.h line 313 at r1 (raw file):
* \param pf PF context. * \param[out] recovery_needed (optional)Whether recovery is needed for \p pf. * \param[out] pos_size (optional)Size of the \p pf node position.
I don't understand this term, "size of position" - is this the size in bytes of a number indicating the position in the file? But that doesn't make much sense.
Code quote:
Size of the \p pf node position.common/src/protected_files/protected_files.h line 314 at r1 (raw file):
* \param[out] recovery_needed (optional)Whether recovery is needed for \p pf. * \param[out] pos_size (optional)Size of the \p pf node position. * \param[out] node_size (optional)Size of the \p pf node data.
ditto (space)
common/src/protected_files/protected_files.h line 319 at r1 (raw file):
*/ pf_status_t pf_get_recovery_info(pf_context_t* pf, bool* recovery_needed, size_t* pos_size, size_t* node_size);
please add out_ prefix to the out arguments
common/src/protected_files/protected_files_format.h line 59 at r1 (raw file):
pf_nonce_t metadata_key_nonce; pf_mac_t metadata_mac; /* GCM mac */ uint8_t update_flag; /* for file recovery */
https://gramine.readthedocs.io/en/latest/devel/encfiles.html will need an update
common/src/protected_files/protected_files_format.h line 59 at r1 (raw file):
pf_nonce_t metadata_key_nonce; pf_mac_t metadata_mac; /* GCM mac */ uint8_t update_flag; /* for file recovery */
or something similar, update_flag sounds very unclear what it means
Suggestion:
has_pending_writepal/include/pal/pal.h line 1050 at r1 (raw file):
* * \param handle Handle to the file. * \param handle Handle to the recovery file.
Parameters don't match the signature.
pal/include/pal/pal.h line 1051 at r1 (raw file):
* \param handle Handle to the file. * \param handle Handle to the recovery file. * \param pos_size Size of the pf node position.
ditto, what's a "size of a position"?
Code quote:
Size of the pf node position.pal/include/pal/pal.h line 1056 at r1 (raw file):
* \returns 0 on success, negative error code on failure. */ int PalEncryptedFileRecovery(PAL_HANDLE file_handle, PAL_HANDLE recovery_file_handle,
This sounds like a bad function name, there's no verb in it and I was confused what it actually does before reading the more detailed documentation.
Code quote:
int PalEncryptedFileRecovery(Documentation/manifest-syntax.rst line 1158 at r1 (raw file):
The ``enable_recovery`` mount parameter determines whether file recovery is enabled for the mount. If omitted, it defaults to ``false``. This feature allows
Please be concise - instead of this just add "(default: false)" after the first mention of this option.
Code quote:
If omitted, it defaults to ``false``. 
kailun-qin
force-pushed
the
kailun-qin/add-encrypted-file-recovery
branch
2 times, most recently
from
00a90f3 to
4f28995
Compare
There was a problem hiding this comment.
Reviewable status: 7 of 34 files reviewed, 20 unresolved discussions, not enough approvals from maintainers (1 more required), not enough approvals from different teams (1 more required, approved so far: Intel) (waiting on @efu39, @mkow, and @ynonflumintel)
a discussion (no related file):
Previously, mkow (Michał Kowalczyk) wrote…
Do I understand correctly (at least looking at #2013 (comment)) that the recovery file stacks all changes that happened while the file was open? This may be a lot for long-running enclaves, even if the file on the disk is itself small, but often modified?
No, the recovery file is limited to the current write transaction. During each file flush, only the cached blocks about to change are saved to the recovery file. The recovery file is then truncated and rewritten with the latest pending changes.
a discussion (no related file):
Previously, mkow (Michał Kowalczyk) wrote…
We probably should benchmark this before merging (see #2013 (comment)).
Sure, I'll run some micro and macro benchmarks later and share the results here.
common/src/protected_files/protected_files.h line 313 at r1 (raw file):
Previously, mkow (Michał Kowalczyk) wrote…
I don't understand this term, "size of position" - is this the size in bytes of a number indicating the position in the file? But that doesn't make much sense.
Yes, it is exactly the size in bytes of a number indicating the position in the file. I made it initially for flexibility, but I have now removed it for clarity. Also added some comments in recover_encrypted_file().
common/src/protected_files/protected_files.h line 314 at r1 (raw file):
Previously, mkow (Michał Kowalczyk) wrote…
ditto (space)
Done.
common/src/protected_files/protected_files.h line 319 at r1 (raw file):
Previously, mkow (Michał Kowalczyk) wrote…
please add
out_prefix to the out arguments
Done.
common/src/protected_files/protected_files.c line 452 at r1 (raw file):
Previously, efu39 (Erica Fu) wrote…
nitpicking: maybe moving the 'offset' declaration a few lines back, together with 'node'
Done.
common/src/protected_files/protected_files.c line 522 at r1 (raw file):
Previously, efu39 (Erica Fu) wrote…
maybe use "goto recoverable_error;" instead for consistency?
Done.
common/src/protected_files/protected_files.c line 528 at r1 (raw file):
Previously, efu39 (Erica Fu) wrote…
same as above
Done.
common/src/protected_files/protected_files_format.h line 59 at r1 (raw file):
Previously, mkow (Michał Kowalczyk) wrote…
https://gramine.readthedocs.io/en/latest/devel/encfiles.html will need an update
Done.
common/src/protected_files/protected_files_format.h line 59 at r1 (raw file):
Previously, mkow (Michał Kowalczyk) wrote…
or something similar,
update_flagsounds very unclear what it means
Done.
Documentation/manifest-syntax.rst line 1158 at r1 (raw file):
Previously, mkow (Michał Kowalczyk) wrote…
Please be concise - instead of this just add "(default: false)" after the first mention of this option.
Done.
libos/src/fs/libos_fs_encrypted.c line 277 at r1 (raw file):
Previously, efu39 (Erica Fu) wrote…
suggest changing the message to 'file recovery attempted but failed' for clarity.
Done.
libos/src/fs/libos_fs_encrypted.c line 328 at r1 (raw file):
Previously, efu39 (Erica Fu) wrote…
wondering if PalStreamDelete(enc->recovery_file_pal_handle, ..) should be invoked as well when pf_close() fails above?
I intentionally made it this way; added a comment there.
pal/include/pal/pal.h line 1050 at r1 (raw file):
Previously, mkow (Michał Kowalczyk) wrote…
Parameters don't match the signature.
Done.
pal/include/pal/pal.h line 1051 at r1 (raw file):
Previously, mkow (Michał Kowalczyk) wrote…
ditto, what's a "size of a position"?
Not relevant any more.
pal/include/pal/pal.h line 1056 at r1 (raw file):
Previously, mkow (Michał Kowalczyk) wrote…
This sounds like a bad function name, there's no verb in it and I was confused what it actually does before reading the more detailed documentation.
Done.
common/src/protected_files/protected_files.h line 222 at r1 (raw file):
* \param create Overwrite file contents if true. * \param key Wrap key. * \param recovery_file_handle (optional)Underlying recovery file handle.
Done.
|
|
||
| /* Whether to enable file recovery (used by `chroot_encrypted` filesystem), false if not | ||
| * applicable */ | ||
| bool enable_recovery; |
There was a problem hiding this comment.
Thanks for the input! I go w/ the first approach -- not allowing non-recoverable mounts if a recovery is needed.
| * | ||
| * `uri` must not correspond to an existing file. | ||
| * | ||
| * The newly created `libos_encrypted_file` object will have `use_count` set to 1. | ||
| */ | ||
| int encrypted_file_create(const char* uri, mode_t perm, struct libos_encrypted_files_key* key, | ||
| struct libos_encrypted_file** out_enc); | ||
| bool enable_recovery, struct libos_encrypted_file** out_enc); |
There was a problem hiding this comment.
I left it as is for now.
kailun-qin
force-pushed
the
kailun-qin/add-encrypted-file-recovery
branch
from
4f28995 to
0eb5ec3
Compare
Previously, a fatal error during writes to encrypted files could cause file corruption due to incorrect GMACs and/or encryption keys. To address this, we introduce a file recovery mechanism using a "shadow" recovery file that stores data about to change and a `has_pending_write` flag in the metadata node indicating the start of a write transaction. During file flush, all cached blocks that are about to change are saved to the recovery file in the format of physical node numbers (offsets) plus encrypted block data. Before saving the main file contents, the `has_pending_write` flag is set in the file's metadata node and cleared only when the transaction is complete. If an encrypted file is opened and the `has_pending_write` flag is set, a recovery process starts to revert partial changes using the recovery file, returning to the last known good state. The "shadow" recovery file is cleaned up on file close. This commit adds a new mount parameter `enable_recovery = [true|false]` for encrypted files mounts to optionally enable this feature. We extend the file flush logic of protected files (pf) to include the recovery file dump and the setting/unsetting of the update flag. We make changes to the public pf APIs: the `pf_open()` API is extended to make the pf aware of the underlying recovery file managed by the LibOS, and recovery information (e.g., whether the pf needs recovery) is exposed back to the LibOS via a new `pf_get_recovery_info()` API. To facilitate the LibOS to initiate a file recovery process on file open, a new PAL API `PalRecoverEncryptedFile()` is introduced. Signed-off-by: Kailun Qin <[email protected]>
kailun-qin
force-pushed
the
kailun-qin/add-encrypted-file-recovery
branch
from
0eb5ec3 to
9d29158
Compare
There was a problem hiding this comment.
Reviewed 14 of 27 files at r2, all commit messages.
Reviewable status: 21 of 34 files reviewed, 13 unresolved discussions, not enough approvals from maintainers (1 more required), not enough approvals from different teams (1 more required, approved so far: Intel) (waiting on @efu39, @kailun-qin, and @ynonflumintel)
a discussion (no related file):
Previously, kailun-qin (Kailun Qin) wrote…
No, the recovery file is limited to the current write transaction. During each file flush, only the cached blocks about to change are saved to the recovery file. The recovery file is then truncated and rewritten with the latest pending changes.
Ah. Could you update the comment I liked to, then? I think it's missing that step.
pal/include/pal/pal.h line 1050 at r1 (raw file):
Previously, kailun-qin (Kailun Qin) wrote…
Done.
Not done?
a discussion (no related file):
Could you also update the PR description with update_flag -> has_pending_write?
Suggestion:
`has_pending_write` flag
Description of the changes
Previously, a fatal error during writes to encrypted files could cause file corruption due to incorrect GMACs and/or encryption keys.
To address this, we introduce a file recovery mechanism using a "shadow" recovery file that stores data about to change and an update flag in the metadata node indicating the start of a write transaction. During file flush, all cached blocks that are about to change are saved to the recovery file in the format of physical node numbers (offsets) plus encrypted block data. Before saving the main file contents, the
update_flagis set in the file's metadata node and cleared only when the transaction is complete. If an encrypted file is opened and theupdate_flagis set, a recovery process starts to revert partial changes using the recovery file, returning to the last known good state. The "shadow" recovery file is cleaned up on file close.This commit adds a new mount parameter
enable_recovery = [true|false]for encrypted files mounts to optionally enable this feature. We extend the file flush logic of protected files (pf) to include the recovery file dump and the setting/unsetting of the update flag. We make changes to the public pf APIs: thepf_open()API is extended to make the pf aware of the underlying recovery file managed by the LibOS, and recovery information (e.g., whether the pf needs recovery) is exposed back to the LibOS via a newpf_get_recovery_info()API. To facilitate the LibOS to initiate a file recovery process on file open, a new PAL APIPalEncryptedFileRecovery()is introduced.How to test this PR?
CI + manual testing.
This change is