This repository has been archived by the owner on Jan 3, 2024. It is now read-only.
S3gw rebase 09 2023 #211
Closed
S3gw rebase 09 2023 #211
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Because otherwise we might have conflicting upload ids if there are too many multipart uploads starting in parallel. Signed-off-by: Joao Eduardo Luis <[email protected]>
Signed-off-by: Joao Eduardo Luis <[email protected]>
Signed-off-by: Joao Eduardo Luis <[email protected]>
Signed-off-by: Joao Eduardo Luis <[email protected]>
Signed-off-by: Joao Eduardo Luis <[email protected]>
After delete bucket is implemented we'd need to identify buckets and objects owned by that bucket in a unique way. This implements the changes needed in metadata and code related to identify bucket and objects in a bucket with bucket_id instead of bucket name. Fixes: https://github.com/aquarist-labs/s3gw/issues/128 Signed-off-by: Xavi Garcia <[email protected]>
Signed-off-by: Giuseppe Baccini <[email protected]>
Fixes compilation issue introduced after merging the PR: #44 Fixes: https://github.com/aquarist-labs/s3gw/issues/133 Signed-off-by: Xavi Garcia <[email protected]>
run rgw/sfs unit test suite on every pull request submission. Fixes: https://github.com/aquarist-labs/s3gw/issues/94 Signed-off-by: Giuseppe Baccini <[email protected]>
Returns only the buckets owned by the SFSUser being called. It also includes SFSUser tests. Fixes: aquarist-labs/s3gw-tools#33 Signed-off-by: Xavi Garcia <[email protected]>
Added a new table in sqlite's metadata to store the access keys related to a user. That way, when querying by access key we query first that table to obtain the user id. Could not implement in the user table itself because access keys are stored in a blob and we'd need to first decode the blob in order to query by access key. Removed the helper columsn in users table for access keys as it was only supporting the first key and it's no longer used. Fixes: aquarist-labs/s3gw-tools#83 Signed-off-by: Xavi Garcia <[email protected]>
Because we were obtaining 'obj_map_lock' (through 'Bucket::finish()') after
acquiring 'multipart_map_lock', we introduced a circular dependency with
'Bucket::abort_multiparts()'.
By ensuring we acquire the locks in the intended order ('obj_map_lock'
first, then 'multipart_part_lock'), matching the other function, we
avoid getting stuck in a deadlock situation.
Signed-off-by: Joao Eduardo Luis <[email protected]>
A few user operations were not acquiring the connection lock, either at all or at some stage of their execution. This was leading to operations failing/asserting because the database would be locked by a concurrent operation. Signed-off-by: Joao Eduardo Luis <[email protected]>
Signed-off-by: Xavi Garcia <[email protected]>
This reverts commit 4b8053d, reversing changes made to 4ead447. Signed-off-by: Xavi Garcia <[email protected]>
Signed-off-by: Xavi Garcia <[email protected]>
`mtime` parameter was logged but that's an output variable and might be `nullptr`. Avoid logging it and check if it's `nullptr` before setting the value. Fixes: aquarist-labs/s3gw-tools#191 Signed-off-by: Xavi Garcia <[email protected]>
This patch allows a client to invoke stat_bucket call. The eventual actual implementation will be addressed on a dedicated issue. Fixes: aquarist-labs/s3gw-tools#159 Signed-off-by: Giuseppe Baccini <[email protected]>
Adds a check on metadata compatibility when the `SFStore` initialises. It creates a temporary copy of the metadata database and tries to sync the schema on it. If there are issues (exeptions) or any table is dropped due to incompatibilities between the data base and the actual metadata schema it throws a `sqlite_sync_exception`. The execption is catched when the `SFStore` is instantiated and the error is logged before exiting `rgw`. Previous database is not altered as all checks are done in the temporary database so we don't have data loss. Fixes: aquarist-labs/s3gw-tools#178 Signed-off-by: Xavi Garcia <[email protected]>
Adds code for adding a delete marker when deleting an object that doesn't exist. According to the AWS documentation https://docs.aws.amazon.com/cli/latest/reference/s3api/delete-object.html when deleteing an object it should reply no errors even if the object doesn't exist. This means we create a delete-marker version for the object when versioning is enabled. This also fixes the s3-test test_versioning_multi_object_delete_with_marker_create which is exactly that test case. When undeleting delete markers (or deleting all versions of an object), if the object ends up having no version we just remove it from the in-memory objects map. The Garbage collector could then permanently remove those objects without a version which are, in fact, no longer reachable. Fixes: aquarist-labs/s3gw-tools#186 Signed-off-by: Xavi Garcia <[email protected]>
This adds an initial GC approach just starting a thread and running a basic remove deleted buckets operation. The implementation of how the thread is started and stopped are taken from rgw's code for consistency across stores. It uses 2 parameters from rgw: - `rgw_gc_processor_period` : defines the time GC waits to be triggered - `rgw_gc_max_objs` : defines the maximum number of items the GC can remove per iteration. Any possible actions done to the `sqlite` metadata is considered an item (object) when taking into account the maxium number of items to be processed in a single iteration as those actions block the metadata database. Fixes: aquarist-labs/s3gw-tools#165 Signed-off-by: Xavi Garcia <[email protected]>
Signed-off-by: Volker Theile <[email protected]>
Adds newly added unit tests so they're found when running the testcontainer. It also fixes the `unittest_rgw_sfs_gc` that was throwing an exception in certain circumstances when converting a version object. Fix is done in this same commit as the goal is to have the newly added tests and prevent other open PRs to be blocked. Signed-off-by: Xavi Garcia <[email protected]>
Signed-off-by: Giuseppe Baccini <[email protected]>
Add new 'status' RGW frontend serving status pages on 127.0.0.1:9999. Includes a beast-based HTTP server, a rudimentary status page registry and a metrics page. Activate by setting 'rgw frontends' (e.g `rgw frontends = beast port=7480, status`) Signed-off-by: Marcel Lauhoff <[email protected]>
Signed-off-by: Marcel Lauhoff <[email protected]>
Add and carry around the perf counter index. This makes the index available when iterating, for example to print / export the counters. This is useful when exporting histograms in the prometheus exposition format that includes a running sum for histograms. These are not included in the perf counter histograms and require a second counter. To print both perf counters at the same time the index is needed to match them from different collections. Signed-off-by: Marcel Lauhoff <[email protected]>
Add string conversion (function, <<operator) and a last element to make iteration possible. Signed-off-by: Marcel Lauhoff <[email protected]>
Add perf counter collection 'rgw_op' containing one counter per RGW operation. Increment counter during init_processing(). Signed-off-by: Marcel Lauhoff <[email protected]>
Add two new perf counter collections to capture operation service time as one-dimensional histograms and time counters. This combination allows conversion to the Prometheus histogram format that requires not only counter buckets, but also a running sum of all values. Signed-off-by: Marcel Lauhoff <[email protected]>
After the multipart code was merged there are a few s3tests that pass and were not enabled. This PR enables those s3tests. Fixes: aquarist-labs/s3gw-tools#185 Signed-off-by: Xavi Garcia <[email protected]>
To allow access to secrets, run PRs labeled with ci/build-s3gw-image in the target repository. Signed-off-by: Marcel Lauhoff <[email protected]>
env.s3gw_telemetry is only valid when using the sfs store. If you try to run the radosgw binary using, say, dbstore, it will segfault on exit due to a null pointer dereference. Fixes: 9b02b73 Signed-off-by: Tim Serong <[email protected]>
It's not safe for DBConn::check_metadata_is_compatible() to make a copy of the database file while it's already open via SQLite, because the close() that happens after the file is copied cancels all POSIX advisory locks, which can later lead to database corruption. For details, see: https://www.sqlite.org/howtocorrupt.html#_posix_advisory_locks_canceled_by_a_separate_thread_doing_close_ This change replaces fs::copy() with use of SQLite's Backup API (https://www.sqlite.org/backup.html) which is safe to use with open databases. Fixes: https://github.com/aquarist-labs/s3gw/issues/702 Signed-off-by: Tim Serong <[email protected]>
This fixes the copy_object operation, in which we were not returning the right mtime and etag in the response. It also adds an integration test file for testing this and other copy_oject tests that are not covered in the upstream s3tests. When s3tests is forked and our CI is configured to use it, the tests in that integration tests file should be moved to s3tests and the file should be deleted. (To avoid having repeated tests) Fixes: https://github.com/aquarist-labs/s3gw/issues/662 Signed-off-by: Xavi Garcia <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
THIS IS A TEST PR FOR TESTING CI