Feature/private datasets #1000
Draft
Feature/private datasets #1000
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Changes before rebasing
* from_catalog_n: add clippy warnings suppression
* kamu-adapter-auth-oso: add TODOs
* Migrations: re-index ReBAC properties
* test_oso: update imports
* KamuAuthOso: add TODOs
* DatasetActionAuthorizer::check_action_allowed(): add a TODO
* DatasetEntryServiceHarness: update for tests
* RebacService::{get_account_properties(),get_dataset_properties()}: return idempotency
* DatasetEntryRepository::get_dataset_entries(): implement for SQLite & Postgres
* RebacRepository::properties_count(): implement for SQLite & Postgres
* AccountRepository::get_accounts(): implement for SQLite & Postgres
* OsoResourceServiceInMem: handle DatasetLifecycleMessage's
* OsoResourceServiceInMem::initialize(): update types
* Split OsoResourceHolder to OsoResourceServiceInMem & OsoResourceServiceInitializator
* OsoResourceHolder: remove dependency to JOB_KAMU_DATASETS_DATASET_ENTRY_INDEXER
* kamu-cli: register DatasetEntryIndexer even if not in workspace
* Tests stabilization activities
* RebacIndexer: add missed #[interface(dyn InitOnStartup)]
* kamu-cli: kamu_auth_rebac_services::register_dependencies()
* kamu-cli: kamu_adapter_auth_oso::register_dependencies()
* OsoDatasetAuthorizer: integrate OsoResourceHolder
* OsoResourceHolder: introduce
* DatasetEntryIndexer::index_datasets(): increase log severity
* RebacIndexer: introduce
* kamu-adapter-auth-oso: update description
* RebacServiceImpl: dataset_id_entity -> dataset_entity
* test_multi_tenant_rebac_dataset_lifecycle_message_consumer: actualize tests
* kamu-adapter-auth-oso: add anonymous() helper
* kamu-adapter-auth-oso: use MockDatasetRepositoryWriter
* kamu-adapter-auth-oso: actualize tests
* DatasetActionAuthorizer, DatasetAction: add oso-related impls
* OsoDatasetAuthorizer::get_allowed_actions(): return <HashSet<DatasetAction>, InternalError>
* GQL, Dataset::properties(): use kamu_auth_rebac::DatasetProperties
* kamu-adapter-auth{,-rebac}: remove experimental crates
* OsoDatasetAuthorizer: initial RebacService integration
* #[allow(unused_variables)] -> #[expect(unused_variables)]
* kamu-adapter-auth: extract
* kamu-adapter-rebac: initial
* kamu-adapter-oauth, AggregatingDatasetActionAuthorizer: initial
* kamu-adapter-graphql, from_catalog_n!(): introduce
* test_multi_tenant_rebac_dataset_lifecycle_message_consumer: stabilize tests
* SmTP, AxumServerPushProtocolInstance::push_main_flow(): remove extra allocations
* Tests, test_gql_datasets: use macros for tests
* Tests, test_gql_datasets: expected first
* Fixes after rebasing
* Tests: update dataset_create_empty_*()
* RunInDatabaseTransactionLayer: remove unused
* GQL, Datasets: use pretty_assertions::assert_eq!()
* GQL, DatasetPropertyName: remove outdated scalar
* MultiTenantRebacDatasetLifecycleMessageConsumer::handle_dataset_lifecycle_created_message(): add "allows_anonymous_read" property as well
* GQL, Dataset::properties(): return flags for simplicity
* DatasetMut::set_visibility(): stabilize
* Preparations
- DependencyGraphServiceInMemory: remove extra .int_err() calls
- Dataset::rebac_properties(): introduce
- RebacService::get_dataset_properties(): use DatasetPropertyName instead of PropertyName
- kamu-auth-rebac: extract value constants
- DatasetMut::{set_publicly_available(),set_anonymous_available()}: ensure account owns dataset
- DatasetMut: move to own directory
- DatasetMut::{set_publicly_available(),set_anonymous_available()}: hide methods behind logging guards
- DatasetMut::set_property(): extract method
- DatasetMut::set_anonymous_available(): implement
- DatasetMut::set_publicly_available(): implement
- RevokeResultSuccess::message(): fix typo
* Fixes after rebasing on 0.208.*
* Tests, kamu-cli: auto-register e2e-user for the e2e mode
* OSO: replace names with IDs in schema
* Tests stabilization
* sqlx: add cached queries
* Build speed-ups: remove unused deps
* test_pull_derivative_mt: correct running
* CHANGELOG: add some entries
* DatasetEntryRepository: simplify lifetimes
* kamu-adapter-auth-oso-rebac: add "-rebac" suffix
* Remove several TODOs
* CHANGELOG.md: add several entries
* OsoDatasetAuthorizer: revisit implementation
* Review 1: GQL: remove Dataset.properties
* Review 1: OsoDatasetAuthorizer::ctor(): fix param name
* database-common, EntityStreamer: introduce
* DatasetEntryServiceImpl: use EntityStreamer
* RebacServiceImpl::get_dataset_properties_by_ids(): add
* PaginationOpts::safe_limit(): add
* Tests, EntityStreamer: add tests with input data
* RebacService::get_dataset_properties_by_ids(): update interface
* DatasetEntryServiceImpl: use EntityStreamer [2]
* OsoResourceServiceInMem: rewrite to use streamed pages
* OsoDatasetAuthorizer: use get_multiple_dataset_resources()
* OsoResourceServiceInitializator: remove
* query_handler_post(): add a comma in doc
* DatasetActionAuthorizer: add TODOs
* test_flow_event_store: fix typos
* OsoDatasetAuthorizer::user_dataset_pair(): remove
* RebacIndexer::index_dataset_entries(): iterate over a stream
* EntityStreamer: remove extra int_err() & resort declarations
* AccountRepository::get_accounts(): streamed version
* RebacIndexer::index_accounts(): use iterate over a stream
* Test fixes
* RebacRepository::get_entity_properties_by_ids(): implementations
* Remove extra as_did_str() call
* RebacRepository::get_entity_properties_by_ids(): implementations[2]
* AccountRepository::accounts_count(): implementations
* PostgresAccountRepository::get_accounts(): implementation
* sqlx: update cached queries
* RebacRepository::get_entity_properties_by_ids(): implementations[3]
* DatasetEntryServiceImpl: use tokio::sync::RwLock
* PostgresDatasetEntryRepository: tweaks
* EntityStreamer -> EntityPageStreamer
* sqlite_generate_placeholders_list: extract & use
* OsoResourceServiceInMem: add a TODO about state
* Search::query(): use from_catalog_n!()
* OsoResourceServiceInMem -> OsoResourceServiceImpl
* KamuAuthOso: impl Deref to Arc<Oso>
* OsoResourceServiceImpl: concrete error types
* kamu-adapter-auth-oso-rebac: remove extra dep
* DatasetEntryRepository: use odf namespace
* DatasetEntryServiceImpl: use odf namespace
* DatasetEntryService::list_entries_owned_by(): do not clone owner_id
* DatasetEntryRepository::get_dataset_entries(): update ORDER BY column
* EntityListing -> EntityPageListing
* Tweaks before merging
* Do not show usage error for --all flag (#960) * Do not show usage error for --all flag When --all flag is set for the `repo delete` command, and there are no repositories to delete, do not shoow usage error. * Improve args validation * Improve args validation, e2e tests * Typo corrected in feature flags (#974) * Images, kamu-base-git: fix collision of executable files (#975) * 868 api server provide feature flags for UI (#976) Separated runtime and UI configuration flags. UI config is provided by API server too. * Release v0.210.0 + minor deps * 854 persistent storage of dataset dependencies graph (#973) Dependency graph service moved to 'datasets' domain. Defined dataset dependency repository interface and created 3 implementations. No more postponed initialization, organized initial setup in the form of an indexer. Added telemetry extensions on the way. Tests for repositories, stabilized other tests. Cascading effect on delete within the dataset entry domain. * v0.211.0 + minor deps * Fixed image building (#977) Replaced cascade delete of dataset entries in graph with more explicit events to allow orphan upstream dependencies where only ID is given * Upgrade to datafusion 43 * Use thiserror v2 throughout * trust-dns-resolver => hickory-resolver + minor deps * Fix non-sequential offsets on ingest * 0.212.0 * Use KAMU_CONTAINER_RUNTIME_TYPE env var in Makefile (#991) * Use KAMU_CONTAINER_RUNTIME_TYPE env var in Makefile * Make podman default engine for e2e tests * Backporting changes from Private Datasets feature branch (#992) * Backport tweaks * Add doc strings * Remove unused deps * Remove unactual test * CHANGELOG.md: update * Tips after self-review * Delete env var on dataset delete (#993) * Delete env var on dataset delete * 984 refactoring separate planning and execution phases in key dataset manipulation services (#994) * Draft split of `CompactionService` into planner and execution parts * Compaction cleanups * Compacting more cleanups * Compacting: read old HEAD on planning phase * Reset service split on planner and execution * Extracted `MetadataQueryService` - to query polling, push sources and set transform, instead of ingest/transform planners * DataWriterMetadataState became part of polling ingest item at the planning phase * Setting watermark : separate planner and execution service * Push ingest service prepared for split * Push ingest split on planning and executing * Made some order in infra/core services * {Flow,Task,Outbox}Executor=>Agent * Unified naming of planners and executors * Revised telemetry in refactored components * Review: DataWriterDataFusionBuilder flattened * changelog * v0.123.0 + minor deps * kamu-dev-base: include short commit hash as well (#995) * v0.213.1: less agressive telemetry with `DataWriterMetadataState` --------- Co-authored-by: Andrii Demus <[email protected]> Co-authored-by: Sergei Zaychenko <[email protected]> Co-authored-by: Sergii Mikhtoniuk <[email protected]> Co-authored-by: Roman Boiko <[email protected]>
…nted via `DatasetEntryServiceImpl` (#1004) * DatasetOwnershipService: use odf namespace * DatasetEntryServiceImpl: impl DatasetOwnershipService * DatasetOwnershipService: move to kamu-datasets scope * CHANGELOG.md: update
…at are not found (#1013)
* E2E: added the ability to create an account using CLI * OutboxImmediateImpl::post_message_as_json(): return a dispatch error, if present
# Conflicts: # CHANGELOG.md # Cargo.lock # Cargo.toml # LICENSE.txt # resources/openapi-mt.json # resources/openapi.json # src/adapter/auth-oso/Cargo.toml # src/adapter/flight-sql/Cargo.toml # src/adapter/flight-sql/src/lib.rs # src/adapter/flight-sql/src/session_factory.rs # src/adapter/graphql/Cargo.toml # src/adapter/graphql/src/queries/datasets/dataset.rs # src/adapter/graphql/src/queries/datasets/dataset_metadata.rs # src/adapter/graphql/tests/tests/test_gql_account_flow_configs.rs # src/adapter/graphql/tests/tests/test_gql_dataset_flow_runs.rs # src/adapter/http/Cargo.toml # src/adapter/oauth/Cargo.toml # src/adapter/odata/Cargo.toml # src/app/cli/src/app.rs # src/domain/accounts/services/Cargo.toml # src/domain/core/src/services/mod.rs # src/domain/datasets/domain/src/repos/dataset_entry_repository.rs # src/domain/datasets/domain/src/services/dataset_entry_service.rs # src/domain/datasets/services/src/dataset_entry_service_impl.rs # src/domain/flow-system/services/tests/tests/utils/flow_harness_shared.rs # src/infra/accounts/inmem/Cargo.toml # src/infra/accounts/mysql/Cargo.toml # src/infra/accounts/postgres/Cargo.toml # src/infra/accounts/sqlite/Cargo.toml # src/infra/auth-rebac/repo-tests/Cargo.toml # src/infra/core/src/services/dataset_ownership_service_inmem.rs # src/infra/core/src/services/mod.rs # src/infra/core/tests/tests/test_dataset_ownership_service_inmem.rs # src/infra/datasets/inmem/Cargo.toml # src/infra/datasets/inmem/src/repos/inmem_dateset_entry_repository.rs # src/infra/datasets/postgres/.sqlx/query-30c92efe33072f0b9fa446ea3255ffca15f34c2af9aaeb8d31453ab364f97495.json # src/infra/datasets/postgres/Cargo.toml # src/infra/datasets/repo-tests/Cargo.toml # src/infra/datasets/sqlite/Cargo.toml # src/infra/messaging-outbox/inmem/Cargo.toml # src/infra/messaging-outbox/postgres/Cargo.toml # src/infra/messaging-outbox/sqlite/Cargo.toml # src/infra/task-system/inmem/Cargo.toml # src/infra/task-system/postgres/Cargo.toml # src/utils/database-common/Cargo.toml # src/utils/event-sourcing/Cargo.toml # src/utils/init-on-startup/Cargo.toml
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Closes: #issue
Checklist before requesting a review