Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix undefined behaviour in backing store #1558

Merged
merged 19 commits into from
Sep 12, 2024
Merged

Fix undefined behaviour in backing store #1558

merged 19 commits into from
Sep 12, 2024

Conversation

Ndiritu
Copy link
Contributor

@Ndiritu Ndiritu commented Sep 6, 2024

Fixes a bug encountered while marking all properties in an InMemoryBackingStore as dirty.

Context

When iterating over a Map entrySet(), any modification to the underlying map causes undefined behavior during the iteration.

The InMemoryBackingStore iterates over the underlying store's entrySet() and currently calls ensureCollectionPropertyIsConsistent() which may update the underlying Map by calling set() to update the collection size & flag the collection as dirty in case of additions/deletions.

Undefined behaviour manifests as repeated iterations over the same set of items in the underlying store causing performance issues. The situation is amplified when we have a collection of BackedModels where each BackedModel has a collection property that has been updated and we need to mark all properties of all the BackedModels as dirty.

Changes

This PR:

  • Updates check for collection size consistency to prevent updating the underlying map during iteration.
  • Reduces redundant recursive calls when cascading dirty tracking to nested BackedModel elements
  • Separates concerns by allowing get() to determine whether a value is dirty or not which matches its interface definition. getValueFromWrapper() now only unwraps values from the underlying store
  • Fixes bug to ensure collection properties are checked for consistency before determining if they are dirty when we want to return only changed values

The sample test case would previously fail after running for ~60s vs with this fix, it passes in ~3s on the same hardware.

closes microsoftgraph/msgraph-sdk-java#2106

@baywet
Copy link
Member

baywet commented Sep 9, 2024

You might want to look at the follow PRs and see if any of the fixes/optimizations could be ported here.
microsoft/kiota-dotnet#95
microsoft/kiota-dotnet#311
microsoft/kiota-dotnet#326

@Ndiritu Ndiritu changed the title Fix infinite loop in backing store Fix undefined behaviour in backing store Sep 9, 2024
@Ndiritu Ndiritu marked this pull request as ready for review September 10, 2024 09:46
@Ndiritu Ndiritu requested a review from a team as a code owner September 10, 2024 09:46
@Ndiritu
Copy link
Contributor Author

Ndiritu commented Sep 10, 2024

You might want to look at the follow PRs and see if any of the fixes/optimizations could be ported here. microsoft/kiota-dotnet#95 microsoft/kiota-dotnet#311 microsoft/kiota-dotnet#326

Missing KiotaSerialization helper updates. To be tracked by #1561 next sprint

baywet
baywet previously approved these changes Sep 10, 2024
Copy link
Member

@baywet baywet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like @andrueastman revie as well before we merge this one

Copy link
Member

@andrueastman andrueastman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple of comments from me. Otherwise this looks good to me.
Thanks for looking into this @Ndiritu

Object[] items = getObjectArrayFromCollectionWrapper(collectionTuple);

for (final Object item : items) {
if (!(item instanceof BackedModel)) break;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think for this we may need to handle the case for maps in separately as this may be an incorrect assumption for them.
Since getObjectArrayFromCollectionWrapper will also unwrap the values of a map as an array, the key values pairs in the map are not guaranteed to have the same type. A map i.e. additionalData can have primitives and complex backed models too..

Object[] items = getObjectArrayFromCollectionWrapper(collectionTuple);
for (final Object item : items) {
if (!(item instanceof BackedModel)) break;
nestedBackedModelsToEnumerate.add((BackedModel) item);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar concern here, the cast will most likely fail if we called getObjectArrayFromCollectionWrapper on a map that has mixed types and the first item happened to be a backed model.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this. Made these updates & added tests.

Tests also exposed a bug where we now need to propagate returnOnlyChangedValues to nested BackedModels to ensure nested models with collection properties are checked. We have limited the collection checks to happen during enumeration only when we want to return changed values.

Copy link

Copy link
Member

@andrueastman andrueastman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏼

@Ndiritu Ndiritu merged commit aa93e78 into main Sep 12, 2024
9 checks passed
@Ndiritu Ndiritu deleted the fix/backing-store branch September 12, 2024 10:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

Infinite loop when setting the team property on a Group under specific circumstances
4 participants