Add streaming functionality for list objects

[donatello]

Currently, there is a problem in client.list_objects - we cannot access the prefixes within a path, only the objects (when not traversing recursively). This is because the result_fn only takes object properties:

minio-rs/src/s3/args.rs

Lines 1115 to 1135 in 8ecabea

	pub struct ListObjectsArgs<'a> {
	pub extra_headers: Option<&'a Multimap>,
	pub extra_query_params: Option<&'a Multimap>,
	pub region: Option<&'a str>,
	pub bucket: &'a str,
	pub delimiter: Option<&'a str>,
	pub use_url_encoding_type: bool,
	pub marker: Option<&'a str>, // only for ListObjectsV1.
	pub start_after: Option<&'a str>, // only for ListObjectsV2.
	pub key_marker: Option<&'a str>, // only for GetObjectVersions.
	pub max_keys: Option<u16>,
	pub prefix: Option<&'a str>,
	pub continuation_token: Option<&'a str>, // only for ListObjectsV2.
	pub fetch_owner: bool, // only for ListObjectsV2.
	pub version_id_marker: Option<&'a str>, // only for GetObjectVersions.
	pub include_user_metadata: bool, // MinIO extension for ListObjectsV2.
	pub recursive: bool,
	pub use_api_v1: bool,
	pub include_versions: bool,
	pub result_fn: &'a dyn Fn(Vec<Item>) -> bool,
	}

Results function is called as follows:

minio-rs/src/s3/client.rs

Lines 2810 to 2835 in 8ecabea

	while !stop {
	if args.include_versions {
	let resp = self.list_object_versions(&lov_args).await?;
	stop = !resp.is_truncated;
	if resp.is_truncated {
	lov_args.key_marker = resp.next_key_marker;
	lov_args.version_id_marker = resp.next_version_id_marker;
	}
	stop = stop || !(args.result_fn)(resp.contents);
	} else if args.use_api_v1 {
	let resp = self.list_objects_v1(&lov1_args).await?;
	stop = !resp.is_truncated;
	if resp.is_truncated {
	lov1_args.marker = resp.next_marker;
	}
	stop = stop || !(args.result_fn)(resp.contents);
	} else {
	let resp = self.list_objects_v2(&lov2_args).await?;
	stop = !resp.is_truncated;
	if resp.is_truncated {
	lov2_args.start_after = resp.start_after;
	lov2_args.continuation_token = resp.next_continuation_token;
	}
	stop = stop || !(args.result_fn)(resp.contents);
	}
	}

So looks like no way to iterate over prefixes in this function interface.

Of course we can use the lower level list_objects_v2 in the client to do this, but maybe we can make a different interface that is easier to use.

This PR is currently in draft for discussion.

[balamurugana]

So looks like no way to iterate over prefixes in this function interface.

Of course we can use the lower level list_objects_v2 in the client to do this, but maybe we can make a different interface that is easier to use.

@donatello You have to use lower level S3 APIs for this specific needs. The objective of higher level list_objects() is to provide one view for all lower level object listing APIs. As you are interested only with prefixes and there is no way to fetch prefixes without object information as per S3 specification, using lower level S3 APIs is the right choice.

[donatello]

So looks like no way to iterate over prefixes in this function interface.

Of course we can use the lower level list_objects_v2 in the client to do this, but maybe we can make a different interface that is easier to use.

@donatello You have to use lower level S3 APIs for this specific needs. The objective of higher level list_objects() is to provide one view for all lower level object listing APIs. As you are interested only with prefixes and there is no way to fetch prefixes without object information as per S3 specification, using lower level S3 APIs is the right choice.

@balamurugana I want to actually list folders and objects at a prefix and get all of them. Adding a streaming API for this should be ok. We already have it in other SDKs.

[donatello]

@balamurugana This is ready.