Skip to content
Permalink

Comparing changes

This is a direct comparison between two commits made in this repository or its related repositories. View the default comparison for this range or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: apache/opendal
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 45ee5c639232768394c9e814d60b55f9a43c5884
Choose a base ref
..
head repository: apache/opendal
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: 791bb95dd6c4d1e18c9f6815e4cd85b4f30df3a3
Choose a head ref
Showing with 407 additions and 232 deletions.
  1. +5 −5 CHANGELOG.md
  2. +5 −2 bin/ofs/src/bin/ofs.rs
  3. +0 −2 bin/ofs/src/fuse/adapter.rs
  4. +5 −4 bin/ofs/src/fuse/mod.rs
  5. +2 −1 bin/ofs/tests/common/mod.rs
  6. +9 −7 bin/ofs/tests/file.rs
  7. +0 −1 bin/ofs/tests/path.rs
  8. +2 −1 bin/oli/src/commands/cp.rs
  9. +12 −6 bin/ovfs/src/virtiofs.rs
  10. +14 −7 bin/ovfs/src/virtiofs_utils.rs
  11. +2 −1 bindings/c/src/reader.rs
  12. +2 −1 bindings/c/src/types.rs
  13. +3 −1 bindings/cpp/src/reader.rs
  14. +3 −1 bindings/java/src/async_operator.rs
  15. +4 −2 bindings/java/src/executor.rs
  16. +0 −1 bindings/java/src/lib.rs
  17. +0 −1 bindings/java/src/operator.rs
  18. +10 −4 bindings/java/src/operator_input_stream.rs
  19. +5 −3 bindings/java/src/operator_output_stream.rs
  20. +2 −1 bindings/nodejs/src/lib.rs
  21. +4 −2 bindings/python/src/file.rs
  22. +3 −1 core/benches/types/concurrent_tasks.rs
  23. +1 −1 core/src/docs/internals/accessor.rs
  24. +3 −0 core/src/docs/mod.rs
  25. +0 −1 core/src/docs/rfcs/3017_remove_write_copy_from.md
  26. +2 −2 core/src/docs/upgrade.md
  27. +2 −1 core/src/layers/prometheus.rs
  28. +3 −2 core/src/layers/prometheus_client.rs
  29. +1 −1 core/src/layers/retry.rs
  30. +2 −1 core/src/raw/adapters/kv/api.rs
  31. +4 −2 core/src/raw/adapters/typed_kv/api.rs
  32. +2 −1 core/src/raw/atomic_util.rs
  33. +4 −2 core/src/raw/futures_util.rs
  34. +2 −1 core/src/raw/http_util/body.rs
  35. +0 −1 core/src/raw/layer.rs
  36. +2 −1 core/src/raw/oio/buf/pooled_buf.rs
  37. +2 −2 core/src/raw/oio/list/page_list.rs
  38. +1 −1 core/src/raw/oio/read/api.rs
  39. +0 −6 core/src/raw/oio/write/api.rs
  40. +2 −2 core/src/raw/oio/write/append_write.rs
  41. +3 −4 core/src/raw/oio/write/block_write.rs
  42. +4 −3 core/src/raw/oio/write/multipart_write.rs
  43. +2 −2 core/src/raw/oio/write/one_shot_write.rs
  44. +2 −2 core/src/raw/oio/write/range_write.rs
  45. +2 −1 core/src/services/aliyun_drive/backend.rs
  46. +7 −5 core/src/services/aliyun_drive/core.rs
  47. +4 −6 core/src/services/aliyun_drive/lister.rs
  48. +8 −5 core/src/services/aliyun_drive/writer.rs
  49. +5 −2 core/src/services/compfs/core.rs
  50. +3 −1 core/src/services/fs/core.rs
  51. +1 −0 core/src/services/fs/reader.rs
  52. +0 −1 core/src/services/ftp/backend.rs
  53. +2 −1 core/src/services/ftp/reader.rs
  54. +4 −3 core/src/services/hdfs/reader.rs
  55. +1 −1 core/src/services/icloud/docs.md
  56. +1 −1 core/src/services/koofr/backend.rs
  57. +1 −1 core/src/services/mysql/backend.rs
  58. +1 −1 core/src/services/pcloud/backend.rs
  59. +2 −1 core/src/services/s3/backend.rs
  60. +2 −2 core/src/services/s3/compatible_services.md
  61. +0 −1 core/src/services/sftp/reader.rs
  62. +2 −2 core/src/services/sqlite/backend.rs
  63. +2 −1 core/src/services/supabase/backend.rs
  64. +1 −1 core/src/services/tikv/docs.md
  65. +1 −1 core/src/types/blocking_read/blocking_reader.rs
  66. +3 −2 core/src/types/blocking_read/buffer_iterator.rs
  67. +2 −1 core/src/types/blocking_read/std_bytes_iterator.rs
  68. +2 −1 core/src/types/blocking_write/blocking_writer.rs
  69. +2 −1 core/src/types/blocking_write/std_writer.rs
  70. +0 −1 core/src/types/buffer.rs
  71. +7 −4 core/src/types/context.rs
  72. +7 −4 core/src/types/execute/api.rs
  73. +9 −5 core/src/types/execute/executor.rs
  74. +2 −2 core/src/types/execute/executors/mod.rs
  75. +6 −3 core/src/types/execute/executors/tokio_executor.rs
  76. +6 −5 core/src/types/operator/blocking_operator.rs
  77. +13 −9 core/src/types/operator/operator.rs
  78. +6 −4 core/src/types/read/buffer_stream.rs
  79. +65 −6 core/src/types/read/reader.rs
  80. +7 −7 core/src/types/scheme.rs
  81. +9 −5 core/src/types/write/buffer_sink.rs
  82. +2 −1 core/src/types/write/futures_async_writer.rs
  83. +9 −6 core/src/types/write/futures_bytes_sink.rs
  84. +55 −24 core/src/types/write/writer.rs
  85. +2 −1 core/tests/behavior/async_delete.rs
  86. +7 −8 integrations/object_store/src/send_wrapper.rs
10 changes: 5 additions & 5 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -342,14 +342,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/).

### Changed
* refactor: Merge compose_{read,write} into enum_utils by @Xuanwo in https://github.com/apache/opendal/pull/3871
* refactor(services/ftp): Impl parse_error instead of From<Error> by @bokket in https://github.com/apache/opendal/pull/3891
* refactor(services/ftp): Impl parse_error instead of `From<Error>` by @bokket in https://github.com/apache/opendal/pull/3891
* docs: very minor English wording fix in error message by @gabrielgrant in https://github.com/apache/opendal/pull/3900
* refactor(services/rocksdb): Impl parse_error instead of From<Error> by @suyanhanx in https://github.com/apache/opendal/pull/3903
* refactor(services/rocksdb): Impl parse_error instead of `From<Error>` by @suyanhanx in https://github.com/apache/opendal/pull/3903
* refactor: Re-organize the layout of tests by @Xuanwo in https://github.com/apache/opendal/pull/3904
* refactor(services/etcd): Impl parse_error instead of From<Error> by @suyanhanx in https://github.com/apache/opendal/pull/3910
* refactor(services/sftp): Impl parse_error instead of From<Error> by @G-XD in https://github.com/apache/opendal/pull/3914
* refactor(services/etcd): Impl parse_error instead of `From<Error>` by @suyanhanx in https://github.com/apache/opendal/pull/3910
* refactor(services/sftp): Impl parse_error instead of `From<Error>` by @G-XD in https://github.com/apache/opendal/pull/3914
* refactor!: Bump MSRV to 1.75 by @Xuanwo in https://github.com/apache/opendal/pull/3851
* refactor(services/redis): Impl parse_error instead of From<Error> by @suyanhanx in https://github.com/apache/opendal/pull/3938
* refactor(services/redis): Impl parse_error instead of `From<Error>` by @suyanhanx in https://github.com/apache/opendal/pull/3938
* refactor!: Revert the bump of MSRV to 1.75 by @Xuanwo in https://github.com/apache/opendal/pull/3952
* refactor(services/onedrive): Add OnedriveConfig to implement ConfigDeserializer by @Borber in https://github.com/apache/opendal/pull/3954
* refactor(service/dropbox): Add DropboxConfig by @howiieyu in https://github.com/apache/opendal/pull/3961
7 changes: 5 additions & 2 deletions bin/ofs/src/bin/ofs.rs
Original file line number Diff line number Diff line change
@@ -43,11 +43,14 @@ async fn main() -> Result<()> {

#[cfg(any(target_os = "linux", target_os = "freebsd"))]
async fn execute(cfg: Config) -> Result<()> {
use std::{collections::HashMap, env, str::FromStr};
use std::collections::HashMap;
use std::env;
use std::str::FromStr;

use anyhow::anyhow;
use ofs::fuse::Fuse;
use opendal::{Operator, Scheme};
use opendal::Operator;
use opendal::Scheme;

if cfg.backend.has_host() {
log::warn!("backend host will be ignored");
2 changes: 0 additions & 2 deletions bin/ofs/src/fuse/adapter.rs
Original file line number Diff line number Diff line change
@@ -24,14 +24,12 @@ use std::time::Duration;
use std::time::SystemTime;

use bytes::Bytes;

use fuse3::path::prelude::*;
use fuse3::Errno;
use fuse3::Result;
use futures_util::stream;
use futures_util::stream::BoxStream;
use futures_util::StreamExt;

use opendal::EntryMode;
use opendal::ErrorKind;
use opendal::Metadata;
9 changes: 5 additions & 4 deletions bin/ofs/src/fuse/mod.rs
Original file line number Diff line number Diff line change
@@ -17,13 +17,14 @@

mod adapter;

use adapter::FuseAdapter;
use std::io;
use std::path::Path;

use adapter::FuseAdapter;
use fuse3::path::Session;
pub use fuse3::raw::MountHandle;

use fuse3::{path::Session, MountOptions};
use fuse3::MountOptions;
use opendal::Operator;
use std::{io, path::Path};

/// Ofs fuse filesystem mounter.
#[derive(Debug, Clone)]
3 changes: 2 additions & 1 deletion bin/ofs/tests/common/mod.rs
Original file line number Diff line number Diff line change
@@ -21,7 +21,8 @@ use opendal::raw::tests;
use opendal::Capability;
use tempfile::TempDir;
use test_context::TestContext;
use tokio::runtime::{self, Runtime};
use tokio::runtime::Runtime;
use tokio::runtime::{self};

static INIT_LOGGER: OnceLock<()> = OnceLock::new();
static RUNTIME: OnceLock<Runtime> = OnceLock::new();
16 changes: 9 additions & 7 deletions bin/ofs/tests/file.rs
Original file line number Diff line number Diff line change
@@ -17,15 +17,17 @@

mod common;

use std::{
fs::{self, File, OpenOptions},
io::{Read, Seek, SeekFrom, Write},
thread,
time::Duration,
};
use std::fs::File;
use std::fs::OpenOptions;
use std::fs::{self};
use std::io::Read;
use std::io::Seek;
use std::io::SeekFrom;
use std::io::Write;
use std::thread;
use std::time::Duration;

use common::OfsTestContext;

use test_context::test_context;

static TEST_TEXT: &str = include_str!("../Cargo.toml");
1 change: 0 additions & 1 deletion bin/ofs/tests/path.rs
Original file line number Diff line number Diff line change
@@ -20,7 +20,6 @@ mod common;
use std::fs;

use common::OfsTestContext;

use test_context::test_context;
use walkdir::WalkDir;

3 changes: 2 additions & 1 deletion bin/oli/src/commands/cp.rs
Original file line number Diff line number Diff line change
@@ -24,7 +24,8 @@ use clap::Arg;
use clap::ArgAction;
use clap::ArgMatches;
use clap::Command;
use futures::{AsyncWriteExt, TryStreamExt};
use futures::AsyncWriteExt;
use futures::TryStreamExt;

use crate::config::Config;

18 changes: 12 additions & 6 deletions bin/ovfs/src/virtiofs.rs
Original file line number Diff line number Diff line change
@@ -18,14 +18,20 @@
use std::io;
use std::sync::RwLock;

use vhost::vhost_user::message::{VhostUserProtocolFeatures, VhostUserVirtioFeatures};
use vhost::vhost_user::message::VhostUserProtocolFeatures;
use vhost::vhost_user::message::VhostUserVirtioFeatures;
use vhost::vhost_user::Backend;
use vhost_user_backend::{VhostUserBackend, VringMutex, VringState, VringT};
use vhost_user_backend::VhostUserBackend;
use vhost_user_backend::VringMutex;
use vhost_user_backend::VringState;
use vhost_user_backend::VringT;
use virtio_bindings::bindings::virtio_config::VIRTIO_F_VERSION_1;
use virtio_bindings::bindings::virtio_ring::{
VIRTIO_RING_F_EVENT_IDX, VIRTIO_RING_F_INDIRECT_DESC,
};
use vm_memory::{ByteValued, GuestMemoryAtomic, GuestMemoryMmap, Le32};
use virtio_bindings::bindings::virtio_ring::VIRTIO_RING_F_EVENT_IDX;
use virtio_bindings::bindings::virtio_ring::VIRTIO_RING_F_INDIRECT_DESC;
use vm_memory::ByteValued;
use vm_memory::GuestMemoryAtomic;
use vm_memory::GuestMemoryMmap;
use vm_memory::Le32;
use vmm_sys_util::epoll::EventSet;
use vmm_sys_util::eventfd::EventFd;

21 changes: 14 additions & 7 deletions bin/ovfs/src/virtiofs_utils.rs
Original file line number Diff line number Diff line change
@@ -17,17 +17,24 @@

use std::cmp::min;
use std::collections::VecDeque;
use std::io::{self, Read, Write};
use std::mem::{size_of, MaybeUninit};
use std::io::Read;
use std::io::Write;
use std::io::{self};
use std::mem::size_of;
use std::mem::MaybeUninit;
use std::ops::Deref;
use std::ptr::copy_nonoverlapping;

use virtio_queue::DescriptorChain;
use vm_memory::bitmap::{Bitmap, BitmapSlice};
use vm_memory::{
Address, ByteValued, GuestMemory, GuestMemoryMmap, GuestMemoryRegion, VolatileMemory,
VolatileSlice,
};
use vm_memory::bitmap::Bitmap;
use vm_memory::bitmap::BitmapSlice;
use vm_memory::Address;
use vm_memory::ByteValued;
use vm_memory::GuestMemory;
use vm_memory::GuestMemoryMmap;
use vm_memory::GuestMemoryRegion;
use vm_memory::VolatileMemory;
use vm_memory::VolatileSlice;

use crate::error::*;

3 changes: 2 additions & 1 deletion bindings/c/src/reader.rs
Original file line number Diff line number Diff line change
@@ -15,9 +15,10 @@
// specific language governing permissions and limitations
// under the License.

use ::opendal as core;
use std::io::Read;

use ::opendal as core;

use super::*;

/// \brief The result type returned by opendal's reader operation.
3 changes: 2 additions & 1 deletion bindings/c/src/types.rs
Original file line number Diff line number Diff line change
@@ -15,10 +15,11 @@
// specific language governing permissions and limitations
// under the License.

use opendal::Buffer;
use std::collections::HashMap;
use std::os::raw::c_char;

use opendal::Buffer;

/// \brief opendal_bytes carries raw-bytes with its length
///
/// The opendal_bytes type is a C-compatible substitute for Vec type
4 changes: 3 additions & 1 deletion bindings/cpp/src/reader.rs
Original file line number Diff line number Diff line change
@@ -15,9 +15,11 @@
// specific language governing permissions and limitations
// under the License.

use std::io::Read;
use std::io::Seek;

use anyhow::Result;
use opendal as od;
use std::io::{Read, Seek};

use super::ffi;

4 changes: 3 additions & 1 deletion bindings/java/src/async_operator.rs
Original file line number Diff line number Diff line change
@@ -35,7 +35,9 @@ use opendal::Scheme;

use crate::convert::jmap_to_hashmap;
use crate::convert::jstring_to_string;
use crate::executor::{executor_or_default, get_current_env, Executor};
use crate::executor::executor_or_default;
use crate::executor::get_current_env;
use crate::executor::Executor;
use crate::make_entry;
use crate::make_metadata;
use crate::make_operator_info;
6 changes: 4 additions & 2 deletions bindings/java/src/executor.rs
Original file line number Diff line number Diff line change
@@ -19,9 +19,11 @@ use std::cell::RefCell;
use std::ffi::c_void;
use std::future::Future;

use jni::objects::{JClass, JObject};
use jni::objects::JClass;
use jni::objects::JObject;
use jni::sys::jlong;
use jni::{JNIEnv, JavaVM};
use jni::JNIEnv;
use jni::JavaVM;
use once_cell::sync::OnceCell;
use tokio::task::JoinHandle;

1 change: 0 additions & 1 deletion bindings/java/src/lib.rs
Original file line number Diff line number Diff line change
@@ -23,7 +23,6 @@ use jni::sys::jboolean;
use jni::sys::jint;
use jni::sys::jlong;
use jni::JNIEnv;

use opendal::raw::PresignedRequest;
use opendal::Capability;
use opendal::Entry;
1 change: 0 additions & 1 deletion bindings/java/src/operator.rs
Original file line number Diff line number Diff line change
@@ -25,7 +25,6 @@ use jni::sys::jobject;
use jni::sys::jobjectArray;
use jni::sys::jsize;
use jni::JNIEnv;

use opendal::BlockingOperator;

use crate::convert::jstring_to_string;
14 changes: 10 additions & 4 deletions bindings/java/src/operator_input_stream.rs
Original file line number Diff line number Diff line change
@@ -15,11 +15,17 @@
// specific language governing permissions and limitations
// under the License.

use crate::convert::jstring_to_string;
use jni::objects::{JByteArray, JClass, JObject, JString};
use jni::sys::{jbyteArray, jlong};
use jni::objects::JByteArray;
use jni::objects::JClass;
use jni::objects::JObject;
use jni::objects::JString;
use jni::sys::jbyteArray;
use jni::sys::jlong;
use jni::JNIEnv;
use opendal::{BlockingOperator, StdBytesIterator};
use opendal::BlockingOperator;
use opendal::StdBytesIterator;

use crate::convert::jstring_to_string;

/// # Safety
///
8 changes: 5 additions & 3 deletions bindings/java/src/operator_output_stream.rs
Original file line number Diff line number Diff line change
@@ -15,11 +15,13 @@
// specific language governing permissions and limitations
// under the License.

use jni::objects::{JByteArray, JClass, JString};
use jni::objects::JByteArray;
use jni::objects::JClass;
use jni::objects::JString;
use jni::sys::jlong;
use jni::JNIEnv;

use opendal::{BlockingOperator, BlockingWriter};
use opendal::BlockingOperator;
use opendal::BlockingWriter;

use crate::convert::jstring_to_string;

3 changes: 2 additions & 1 deletion bindings/nodejs/src/lib.rs
Original file line number Diff line number Diff line change
@@ -26,7 +26,8 @@ use std::io::Read;
use std::str::FromStr;
use std::time::Duration;

use futures::{AsyncReadExt, TryStreamExt};
use futures::AsyncReadExt;
use futures::TryStreamExt;
use napi::bindgen_prelude::*;

#[napi]
6 changes: 4 additions & 2 deletions bindings/python/src/file.rs
Original file line number Diff line number Diff line change
@@ -25,9 +25,11 @@ use std::io::Write;
use std::ops::DerefMut;
use std::sync::Arc;

use futures::{AsyncReadExt, AsyncSeekExt};
use futures::AsyncReadExt;
use futures::AsyncSeekExt;
use pyo3::buffer::PyBuffer;
use pyo3::exceptions::{PyIOError, PyValueError};
use pyo3::exceptions::PyIOError;
use pyo3::exceptions::PyValueError;
use pyo3::prelude::*;
use pyo3_asyncio::tokio::future_into_py;
use tokio::sync::Mutex;
4 changes: 3 additions & 1 deletion core/benches/types/concurrent_tasks.rs
Original file line number Diff line number Diff line change
@@ -15,7 +15,9 @@
// specific language governing permissions and limitations
// under the License.

use criterion::{black_box, BatchSize, Criterion};
use criterion::black_box;
use criterion::BatchSize;
use criterion::Criterion;
use once_cell::sync::Lazy;
use opendal::raw::ConcurrentTasks;
use opendal::Executor;
2 changes: 1 addition & 1 deletion core/src/docs/internals/accessor.rs
Original file line number Diff line number Diff line change
@@ -321,6 +321,6 @@
//! [`Access`]: crate::raw::Access
//! [`Operation`]: crate::raw::Operation
//! [`Capability`]: crate::Capability
//! [`AccessorInfo`]: crate::raw::AccessInfo
//! [`AccessorInfo`]: crate::raw::AccessorInfo
//! [`Scheme`]: crate::Scheme
//! [`Builder`]: crate::Builder
3 changes: 3 additions & 0 deletions core/src/docs/mod.rs
Original file line number Diff line number Diff line change
@@ -19,6 +19,8 @@
//!
//! It's highly recommended that you start by reading [`concepts`] first.
#![allow(rustdoc::bare_urls)]

pub mod comparisons;

pub mod concepts;
@@ -27,6 +29,7 @@ pub mod internals;

/// Changes log for all OpenDAL released versions.
#[doc = include_str!("../../CHANGELOG.md")]
#[cfg(not(doctest))]
pub mod changelog {}

#[cfg(not(doctest))]
1 change: 0 additions & 1 deletion core/src/docs/rfcs/3017_remove_write_copy_from.md
Original file line number Diff line number Diff line change
@@ -12,7 +12,6 @@ Remove the `oio::Write::copy_from()` API pending a more thoughtful design.
In [RFC-2083: Writer Sink API](./2083_writer_sink_api.md), we launched an API, initially named `sink` and changed to `copy_from`, that enables data writing from a `Reader` to a `Writer` object.

The current API signature is:

```rust
pub trait Write: Unpin + Send + Sync {
/// Copies data from the given reader to the writer.
4 changes: 2 additions & 2 deletions core/src/docs/upgrade.md
Original file line number Diff line number Diff line change
@@ -205,7 +205,7 @@ There is no public API and raw API changes.

### RFC-2578 Merge Append Into Write

[RFC-2578](crate::docs::rfcs::rfc_2578_merge_append_into_write) merges `append` into `write` and removes `append` API.
[RFC-2578](crate::docs::rfcs::rfc_2758_merge_append_into_write) merges `append` into `write` and removes `append` API.

- For writing a file at once, please use `op.write()` for convenience.
- For appending a file, please use `op.write_with().append(true)` instead of `op.append()`.
@@ -291,7 +291,7 @@ OpenDAL v0.40 removed the origin `range_read` and `range_reader` interfaces, ple

### RFC-3017 Remove Write Copy From

[RFC-3017](opendal::docs::rfcs::rfc_3017_remove_write_copy_from) removes `copy_from` API from the `oio::Write` trait. Users who implements services and layers by hand should remove this API.
[RFC-3017](crate::docs::rfcs::rfc_3017_remove_write_copy_from) removes `copy_from` API from the `oio::Write` trait. Users who implements services and layers by hand should remove this API.

# Upgrade to v0.39

3 changes: 2 additions & 1 deletion core/src/layers/prometheus.rs
Original file line number Diff line number Diff line change
@@ -31,7 +31,8 @@ use prometheus::register_int_counter_vec_with_registry;
use prometheus::HistogramVec;
use prometheus::Registry;

use crate::raw::oio::{ReadOperation, WriteOperation};
use crate::raw::oio::ReadOperation;
use crate::raw::oio::WriteOperation;
use crate::raw::Access;
use crate::raw::*;
use crate::*;
Loading