implements session object handle iterator, with caching

[keldonin]

Hello,

This PR is an alternate candidate for implementing #106.

It implements the Iterator trait for object handlers, given a search template. The iterator is able to fetch values ahead, and cache the results. The cache size can be specified at iterator build time.

design choices

• The iterator will return an Option<Result<ObjectHandle>>. This is because calls to C_FindObjects() may fail, and the caller may want to manage such condition. It is similar to what the std::io::Lines iterator implements.
• I didn't replace the find_objects() method. It seems to me it serves a different goal, ease of use. On the other hand, the iterator allows the caller a more fine-grained control over how object handles are retrieved from the token, and allows for streaming. Both can coexist, IMO.
• as with the recent find_objects change, every effort is made to avoid unnecessary calls to C_FindObjects().
• The iterator implements the Drop trait, to call C_FindObjectsFinal().
• Thanks to the design choices made for the Session object, the iterator features no locking mechanism for its state variables. It assumes that the iterator object will never be accessed concurrently from different threads.
• No code was added to prevent creating two iterators: It relies upon the underlying token to prevent running two searches simultaneously.
• Because the next() method of an iterator cannot return a Result, I had to create a new macro from get_pkcs11!() that does not carry the syntactic sugar ?.
• The implementation tries to address the unlikely case when C_FindObjects and C_FindObjectsFinal are not implemented. But... perhaps the library should refuse to work with such library, and detect that at library loading time?

For your review,

[wiktor-k]

Okay, I think it looks very nice 👍

I've added a couple of personal, tiny nitpicks.

I also wonder if it'd be a good idea to add Closes: https://github.com/parallaxsecond/rust-cryptoki/pull/109 in the commit message trailer (since it appears to be the same functionality but with a separate method).

One further improvement would be re-implementing find_objects in terms of your iter_objects (it should be as simple as just calling iter_objects and then collect on the result 🤔 )

Thanks for your time! 👋

[wiktor-k]

No code was added to prevent creating two iterators: It relies upon the underlying token to prevent running two searches simultaneously.

Correct me if I'm wrong but this should be as simple as making iter_objects_with_cache_size take a mut ref to Session. This way as long as the ObjectHandleIterator exists (which would hold a mut ref to Session no other iterators could be created (since mut means "exclusive"). 🤔

[keldonin]

No code was added to prevent creating two iterators: It relies upon the underlying token to prevent running two searches simultaneously.

Correct me if I'm wrong but this should be as simple as making iter_objects_with_cache_size take a mut ref to Session. This way as long as the ObjectHandleIterator exists (which would hold a mut ref to Session no other iterators could be created (since mut means "exclusive"). 🤔

That would work. However, if I combine this with a rewrite of find_objects() as suggested above, it will propagate the need for a mutable ref, leading to a breaking change for that API:

pub fn find_objects(&mut self, template: &[Attribute]) -> Result<Vec<ObjectHandle>>`

Choose your curse 😉

[keldonin]

Actually there is maybe a good reason for not taking a mutable ref:

    for obj in session.iter_objects(&token_object)? {
        let obj = obj?; 

        let attributes = session.get_attributes(obj, &wanted_attr)?;
        ...

In this pattern, attributes are retrieved as objects are fetched. It results in interleaved calls to C_FindObjects() and C_GetAttributeValue(), which is legit. In fact, crypto operations are also legit during a search. implementing session with a mutable ref would prevent this, ans session.get_attributes() is also needing a ref to the session.

If preventing concurrent searches by the library itself is desirable, it can be easily implemented using a flag in session, inside a Cell for interior mutability.

Any thought?

[wiktor-k]

Yeah, all good points. Maybe we could just leave the "no multiple open searches" invariant in docs and that's it. Making it too complex to cover a rare API misuse may not be such a great idea after all 🤔

(I've got a couple of complex ideas how to "solve" it but I worry that the cure will be worse than the disease 😅 )

[keldonin]

I also wonder if it'd be a good idea to add Closes: https://github.com/parallaxsecond/rust-cryptoki/pull/109 in the commit message trailer (since it appears to be the same functionality but with a separate method).

When all the reviews are finished, I can rebase the branch and incorporate the comment.

One further improvement would be re-implementing find_objects in terms of your iter_objects (it should be as simple as just calling iter_objects and then collect on the result 🤔 )

That's been incorporated.

[ionut-arm]

Approach looks good to me, happy to go ahead with it, just a few minor comments

[wiktor-k]

I think it's the last round of feedback and in general it looks very nice already 👌

[wiktor-k]

LGTM 👍 thanks for taking the time to address all remarks :)

[keldonin]

@wiktor-k:

NP, feedback is useful and relevant!

I don't know what is your policy for merging changes, but since you made this request:

I also wonder if it'd be a good idea to add Closes: #109 in the commit message trailer (since it appears to be the same functionality but with a separate method).

I suggested I would rebase this PR, but that would make another round of approvals, and you would loose visibility on the changes before merging. Maybe is it simpler for you to squash merge and add this comment to the commit message?

[hug-dev]

Nitpicking comments you can feel free to ignore if they are not worth it! Otherwise trusting Wiktor's review 🙏

[wiktor-k]

Otherwise trusting Wiktor's review 🙏

Haha, please don't do that 😅 I try to be diligent here but it's still an unpaid side-project so my time is limited.

I still think that given our collective effort the code continously gets better and better (and with tests future refactoring is always possible).

👋

[keldonin]

Thank you for merging this one!

Out of curiosity, are you planning to make a new tagged release at some point in time?

[hug-dev]

Nothing planned for now I think, it's mostly demand-based! We also try to group release with other dependencies in parallaxsecond. If you need one for your projects, we can start to see when will work!

[hug-dev]

On another topic @keldonin, would you like to be a maintainer of this crate? Since you have shown interest in it in the past few weeks with very nice improvements I was wondering if you would like to help us maintaining it?
There is no special asks or responsibility apart from the power to be able to approve and merge PRs!

[keldonin]

Nothing planned for now I think, it's mostly demand-based! We also try to group release with other dependencies in parallaxsecond. If you need one for your projects, we can start to see when will work!

Hi @hug-dev. Nothing pressing for now, just asking. Thank you for the clarification!

[keldonin]

On another topic @keldonin, would you like to be a maintainer of this crate? Since you have shown interest in it in the past few weeks with very nice improvements I was wondering if you would like to help us maintaining it? There is no special asks or responsibility apart from the power to be able to approve and merge PRs!

I would be honoured.

Time-wise, something I can perform during my free time (probably like you? 🤔) - that I share with other activities - there is life out there 😂.

So, glad to give support here, to the extent of my availability.

[hug-dev]

That's great, thank you!
There is absolutely no ask from our side, put into it as much time as you want! It's mostly so that we can share the merging power so there is less chance that a PR stays too long in limbo state/unreviewed 😃

[hug-dev]

It's done now! If you want, you can join the #parsec Slack channel on the CNCF Slack workspace :) More info here