diff --git a/API-INTERNAL.md b/API-INTERNAL.md index b853a457..38a5d76c 100644 --- a/API-INTERNAL.md +++ b/API-INTERNAL.md @@ -109,10 +109,13 @@ subscriber callbacks receive the data in a different format than they normally e
remove()

Remove a key from Onyx and update the subscribers

-
evictStorageAndRetry()
-

If we fail to set or merge we must handle this by -evicting some data from Onyx and then retrying to do -whatever it is we attempted to do.

+
retryOperation()
+

Handles storage operation failures based on the error type:

+
broadcastUpdate()

Notifies subscribers and writes current value to cache

@@ -147,14 +150,31 @@ It will also mark deep nested objects that need to be entirely replaced during t
unsubscribeFromKey(subscriptionID)

Disconnects and removes the listener from the Onyx key.

-
mergeCollectionWithPatches(collectionKey, collection, mergeReplaceNullPatches)
+
setWithRetry(params, retryAttempt)
+

Writes a value to our store with the given key. +Serves as core implementation for Onyx.set() public function, the difference being +that this internal function allows passing an additional retryAttempt parameter to retry on failure.

+
+
multiSetWithRetry(data, retryAttempt)
+

Sets multiple keys and values. +Serves as core implementation for Onyx.multiSet() public function, the difference being +that this internal function allows passing an additional retryAttempt parameter to retry on failure.

+
+
setCollectionWithRetry(params, retryAttempt)
+

Sets a collection by replacing all existing collection members with new values. +Any existing collection members not included in the new data will be removed. +Serves as core implementation for Onyx.setCollection() public function, the difference being +that this internal function allows passing an additional retryAttempt parameter to retry on failure.

+
+
mergeCollectionWithPatches(params, retryAttempt)

Merges a collection based on their keys. Serves as core implementation for Onyx.mergeCollection() public function, the difference being -that this internal function allows passing an additional mergeReplaceNullPatches parameter.

+that this internal function allows passing an additional mergeReplaceNullPatches parameter and retries on failure.

-
partialSetCollection(collectionKey, collection)
+
partialSetCollection(params, retryAttempt)

Sets keys in a collection by replacing all targeted collection members with new values. -Any existing collection members not included in the new data will not be removed.

+Any existing collection members not included in the new data will not be removed. +Retries on failure.

clearOnyxUtilsInternals()

Clear internal variables used in this file, useful in test environments.

@@ -406,12 +426,13 @@ subscriber callbacks receive the data in a different format than they normally e Remove a key from Onyx and update the subscribers **Kind**: global function - + -## evictStorageAndRetry() -If we fail to set or merge we must handle this by -evicting some data from Onyx and then retrying to do -whatever it is we attempted to do. +## retryOperation() +Handles storage operation failures based on the error type: +- Storage capacity errors: evicts data and retries the operation +- Invalid data errors: logs an alert and throws an error +- Other errors: retries the operation **Kind**: global function @@ -507,33 +528,87 @@ Disconnects and removes the listener from the Onyx key. | --- | --- | | subscriptionID | Subscription ID returned by calling `OnyxUtils.subscribeToKey()`. | + + +## setWithRetry(params, retryAttempt) +Writes a value to our store with the given key. +Serves as core implementation for `Onyx.set()` public function, the difference being +that this internal function allows passing an additional `retryAttempt` parameter to retry on failure. + +**Kind**: global function + +| Param | Description | +| --- | --- | +| params | set parameters | +| params.key | ONYXKEY to set | +| params.value | value to store | +| params.options | optional configuration object | +| retryAttempt | retry attempt | + + + +## multiSetWithRetry(data, retryAttempt) +Sets multiple keys and values. +Serves as core implementation for `Onyx.multiSet()` public function, the difference being +that this internal function allows passing an additional `retryAttempt` parameter to retry on failure. + +**Kind**: global function + +| Param | Description | +| --- | --- | +| data | object keyed by ONYXKEYS and the values to set | +| retryAttempt | retry attempt | + + + +## setCollectionWithRetry(params, retryAttempt) +Sets a collection by replacing all existing collection members with new values. +Any existing collection members not included in the new data will be removed. +Serves as core implementation for `Onyx.setCollection()` public function, the difference being +that this internal function allows passing an additional `retryAttempt` parameter to retry on failure. + +**Kind**: global function + +| Param | Description | +| --- | --- | +| params | collection parameters | +| params.collectionKey | e.g. `ONYXKEYS.COLLECTION.REPORT` | +| params.collection | Object collection keyed by individual collection member keys and values | +| retryAttempt | retry attempt | + -## mergeCollectionWithPatches(collectionKey, collection, mergeReplaceNullPatches) +## mergeCollectionWithPatches(params, retryAttempt) Merges a collection based on their keys. Serves as core implementation for `Onyx.mergeCollection()` public function, the difference being -that this internal function allows passing an additional `mergeReplaceNullPatches` parameter. +that this internal function allows passing an additional `mergeReplaceNullPatches` parameter and retries on failure. **Kind**: global function | Param | Description | | --- | --- | -| collectionKey | e.g. `ONYXKEYS.COLLECTION.REPORT` | -| collection | Object collection keyed by individual collection member keys and values | -| mergeReplaceNullPatches | Record where the key is a collection member key and the value is a list of tuples that we'll use to replace the nested objects of that collection member record with something else. | +| params | mergeCollection parameters | +| params.collectionKey | e.g. `ONYXKEYS.COLLECTION.REPORT` | +| params.collection | Object collection keyed by individual collection member keys and values | +| params.mergeReplaceNullPatches | Record where the key is a collection member key and the value is a list of tuples that we'll use to replace the nested objects of that collection member record with something else. | +| params.isProcessingCollectionUpdate | whether this is part of a collection update operation. | +| retryAttempt | retry attempt | -## partialSetCollection(collectionKey, collection) +## partialSetCollection(params, retryAttempt) Sets keys in a collection by replacing all targeted collection members with new values. Any existing collection members not included in the new data will not be removed. +Retries on failure. **Kind**: global function | Param | Description | | --- | --- | -| collectionKey | e.g. `ONYXKEYS.COLLECTION.REPORT` | -| collection | Object collection keyed by individual collection member keys and values | +| params | collection parameters | +| params.collectionKey | e.g. `ONYXKEYS.COLLECTION.REPORT` | +| params.collection | Object collection keyed by individual collection member keys and values | +| retryAttempt | retry attempt | diff --git a/lib/Onyx.ts b/lib/Onyx.ts index 4d8a169c..cac0ca1c 100644 --- a/lib/Onyx.ts +++ b/lib/Onyx.ts @@ -150,71 +150,7 @@ function disconnect(connection: Connection): void { * @param options optional configuration object */ function set(key: TKey, value: OnyxSetInput, options?: SetOptions): Promise { - // When we use Onyx.set to set a key we want to clear the current delta changes from Onyx.merge that were queued - // before the value was set. If Onyx.merge is currently reading the old value from storage, it will then not apply the changes. - if (OnyxUtils.hasPendingMergeForKey(key)) { - delete OnyxUtils.getMergeQueue()[key]; - } - - const skippableCollectionMemberIDs = OnyxUtils.getSkippableCollectionMemberIDs(); - if (skippableCollectionMemberIDs.size) { - try { - const [, collectionMemberID] = OnyxUtils.splitCollectionMemberKey(key); - if (skippableCollectionMemberIDs.has(collectionMemberID)) { - // The key is a skippable one, so we set the new value to null. - // eslint-disable-next-line no-param-reassign - value = null; - } - } catch (e) { - // The key is not a collection one or something went wrong during split, so we proceed with the function's logic. - } - } - - // Onyx.set will ignore `undefined` values as inputs, therefore we can return early. - if (value === undefined) { - return Promise.resolve(); - } - - const existingValue = cache.get(key, false); - // If the existing value as well as the new value are null, we can return early. - if (existingValue === undefined && value === null) { - return Promise.resolve(); - } - - // Check if the value is compatible with the existing value in the storage - const {isCompatible, existingValueType, newValueType} = utils.checkCompatibilityWithExistingValue(value, existingValue); - if (!isCompatible) { - Logger.logAlert(logMessages.incompatibleUpdateAlert(key, 'set', existingValueType, newValueType)); - return Promise.resolve(); - } - - // If the change is null, we can just delete the key. - // Therefore, we don't need to further broadcast and update the value so we can return early. - if (value === null) { - OnyxUtils.remove(key); - OnyxUtils.logKeyRemoved(OnyxUtils.METHOD.SET, key); - return Promise.resolve(); - } - - const valueWithoutNestedNullValues = utils.removeNestedNullValues(value) as OnyxValue; - const hasChanged = options?.skipCacheCheck ? true : cache.hasValueChanged(key, valueWithoutNestedNullValues); - - OnyxUtils.logKeyChanged(OnyxUtils.METHOD.SET, key, value, hasChanged); - - // This approach prioritizes fast UI changes without waiting for data to be stored in device storage. - const updatePromise = OnyxUtils.broadcastUpdate(key, valueWithoutNestedNullValues, hasChanged); - - // If the value has not changed or the key got removed, calling Storage.setItem() would be redundant and a waste of performance, so return early instead. - if (!hasChanged) { - return updatePromise; - } - - return Storage.setItem(key, valueWithoutNestedNullValues) - .catch((error) => OnyxUtils.evictStorageAndRetry(error, set, key, valueWithoutNestedNullValues)) - .then(() => { - OnyxUtils.sendActionToDevTools(OnyxUtils.METHOD.SET, key, valueWithoutNestedNullValues); - return updatePromise; - }); + return OnyxUtils.setWithRetry({key, value, options}); } /** @@ -225,47 +161,7 @@ function set(key: TKey, value: OnyxSetInput, options * @param data object keyed by ONYXKEYS and the values to set */ function multiSet(data: OnyxMultiSetInput): Promise { - let newData = data; - - const skippableCollectionMemberIDs = OnyxUtils.getSkippableCollectionMemberIDs(); - if (skippableCollectionMemberIDs.size) { - newData = Object.keys(newData).reduce((result: OnyxMultiSetInput, key) => { - try { - const [, collectionMemberID] = OnyxUtils.splitCollectionMemberKey(key); - // If the collection member key is a skippable one we set its value to null. - // eslint-disable-next-line no-param-reassign - result[key] = !skippableCollectionMemberIDs.has(collectionMemberID) ? newData[key] : null; - } catch { - // The key is not a collection one or something went wrong during split, so we assign the data to result anyway. - // eslint-disable-next-line no-param-reassign - result[key] = newData[key]; - } - - return result; - }, {}); - } - - const keyValuePairsToSet = OnyxUtils.prepareKeyValuePairsForStorage(newData, true); - - const updatePromises = keyValuePairsToSet.map(([key, value]) => { - // When we use multiSet to set a key we want to clear the current delta changes from Onyx.merge that were queued - // before the value was set. If Onyx.merge is currently reading the old value from storage, it will then not apply the changes. - if (OnyxUtils.hasPendingMergeForKey(key)) { - delete OnyxUtils.getMergeQueue()[key]; - } - - // Update cache and optimistically inform subscribers on the next tick - cache.set(key, value); - return OnyxUtils.scheduleSubscriberUpdate(key, value); - }); - - return Storage.multiSet(keyValuePairsToSet) - .catch((error) => OnyxUtils.evictStorageAndRetry(error, multiSet, newData)) - .then(() => { - OnyxUtils.sendActionToDevTools(OnyxUtils.METHOD.MULTI_SET, undefined, newData); - return Promise.all(updatePromises); - }) - .then(() => undefined); + return OnyxUtils.multiSetWithRetry(data); } /** @@ -374,7 +270,7 @@ function merge(key: TKey, changes: OnyxMergeInput): * @param collection Object collection keyed by individual collection member keys and values */ function mergeCollection(collectionKey: TKey, collection: OnyxMergeCollectionInput): Promise { - return OnyxUtils.mergeCollectionWithPatches(collectionKey, collection, undefined, true); + return OnyxUtils.mergeCollectionWithPatches({collectionKey, collection, isProcessingCollectionUpdate: true}); } /** @@ -608,16 +504,16 @@ function update(data: OnyxUpdate[]): Promise { if (!utils.isEmptyObject(batchedCollectionUpdates.merge)) { promises.push(() => - OnyxUtils.mergeCollectionWithPatches( + OnyxUtils.mergeCollectionWithPatches({ collectionKey, - batchedCollectionUpdates.merge as OnyxMergeCollectionInput, - batchedCollectionUpdates.mergeReplaceNullPatches, - true, - ), + collection: batchedCollectionUpdates.merge as OnyxMergeCollectionInput, + mergeReplaceNullPatches: batchedCollectionUpdates.mergeReplaceNullPatches, + isProcessingCollectionUpdate: true, + }), ); } if (!utils.isEmptyObject(batchedCollectionUpdates.set)) { - promises.push(() => OnyxUtils.partialSetCollection(collectionKey, batchedCollectionUpdates.set as OnyxSetCollectionInput)); + promises.push(() => OnyxUtils.partialSetCollection({collectionKey, collection: batchedCollectionUpdates.set as OnyxSetCollectionInput})); } }); @@ -655,63 +551,7 @@ function update(data: OnyxUpdate[]): Promise { * @param collection Object collection keyed by individual collection member keys and values */ function setCollection(collectionKey: TKey, collection: OnyxSetCollectionInput): Promise { - let resultCollection: OnyxInputKeyValueMapping = collection; - let resultCollectionKeys = Object.keys(resultCollection); - - // Confirm all the collection keys belong to the same parent - if (!OnyxUtils.doAllCollectionItemsBelongToSameParent(collectionKey, resultCollectionKeys)) { - Logger.logAlert(`setCollection called with keys that do not belong to the same parent ${collectionKey}. Skipping this update.`); - return Promise.resolve(); - } - - const skippableCollectionMemberIDs = OnyxUtils.getSkippableCollectionMemberIDs(); - if (skippableCollectionMemberIDs.size) { - resultCollection = resultCollectionKeys.reduce((result: OnyxInputKeyValueMapping, key) => { - try { - const [, collectionMemberID] = OnyxUtils.splitCollectionMemberKey(key, collectionKey); - // If the collection member key is a skippable one we set its value to null. - // eslint-disable-next-line no-param-reassign - result[key] = !skippableCollectionMemberIDs.has(collectionMemberID) ? resultCollection[key] : null; - } catch { - // Something went wrong during split, so we assign the data to result anyway. - // eslint-disable-next-line no-param-reassign - result[key] = resultCollection[key]; - } - - return result; - }, {}); - } - resultCollectionKeys = Object.keys(resultCollection); - - return OnyxUtils.getAllKeys().then((persistedKeys) => { - const mutableCollection: OnyxInputKeyValueMapping = {...resultCollection}; - - persistedKeys.forEach((key) => { - if (!key.startsWith(collectionKey)) { - return; - } - if (resultCollectionKeys.includes(key)) { - return; - } - - mutableCollection[key] = null; - }); - - const keyValuePairs = OnyxUtils.prepareKeyValuePairsForStorage(mutableCollection, true, undefined, true); - const previousCollection = OnyxUtils.getCachedCollection(collectionKey); - - // Preserve references for unchanged items in setCollection - const preservedCollection = OnyxUtils.preserveCollectionReferences(keyValuePairs); - - const updatePromise = OnyxUtils.scheduleNotifyCollectionSubscribers(collectionKey, preservedCollection, previousCollection); - - return Storage.multiSet(keyValuePairs) - .catch((error) => OnyxUtils.evictStorageAndRetry(error, setCollection, collectionKey, collection)) - .then(() => { - OnyxUtils.sendActionToDevTools(OnyxUtils.METHOD.SET_COLLECTION, undefined, mutableCollection); - return updatePromise; - }); - }); + return OnyxUtils.setCollectionWithRetry({collectionKey, collection}); } const Onyx = { diff --git a/lib/OnyxUtils.ts b/lib/OnyxUtils.ts index 9150ff86..155e0d67 100644 --- a/lib/OnyxUtils.ts +++ b/lib/OnyxUtils.ts @@ -29,7 +29,11 @@ import type { OnyxUpdate, OnyxValue, Selector, - OnyxSetCollectionInput, + MergeCollectionWithPatchesParams, + SetCollectionParams, + SetParams, + OnyxMultiSetInput, + RetriableOnyxOperation, } from './types'; import type {FastMergeOptions, FastMergeResult} from './utils'; import utils from './utils'; @@ -50,6 +54,23 @@ const METHOD = { CLEAR: 'clear', } as const; +// IndexedDB errors that indicate storage capacity issues where eviction can help +const IDB_STORAGE_ERRORS = [ + 'quotaexceedederror', // Browser storage quota exceeded +] as const; + +// SQLite errors that indicate storage capacity issues where eviction can help +const SQLITE_STORAGE_ERRORS = [ + 'database or disk is full', // Device storage is full + 'disk I/O error', // File system I/O failure, often due to insufficient space or corrupted storage + 'out of memory', // Insufficient RAM or storage space to complete the operation +] as const; + +const STORAGE_ERRORS = [...IDB_STORAGE_ERRORS, ...SQLITE_STORAGE_ERRORS]; + +// Max number of retries for failed storage operations +const MAX_STORAGE_OPERATION_RETRY_ATTEMPTS = 5; + type OnyxMethod = ValueOf; // Key/value store of Onyx key and arrays of values to merge @@ -920,22 +941,36 @@ function reportStorageQuota(): Promise { } /** - * If we fail to set or merge we must handle this by - * evicting some data from Onyx and then retrying to do - * whatever it is we attempted to do. + * Handles storage operation failures based on the error type: + * - Storage capacity errors: evicts data and retries the operation + * - Invalid data errors: logs an alert and throws an error + * - Other errors: retries the operation */ -function evictStorageAndRetry( - error: Error, - onyxMethod: TMethod, - ...args: Parameters -): Promise { - Logger.logInfo(`Failed to save to storage. Error: ${error}. onyxMethod: ${onyxMethod.name}`); +function retryOperation(error: Error, onyxMethod: TMethod, defaultParams: Parameters[0], retryAttempt: number | undefined): Promise { + const currentRetryAttempt = retryAttempt ?? 0; + const nextRetryAttempt = currentRetryAttempt + 1; + + Logger.logInfo(`Failed to save to storage. Error: ${error}. onyxMethod: ${onyxMethod.name}. retryAttempt: ${currentRetryAttempt}/${MAX_STORAGE_OPERATION_RETRY_ATTEMPTS}`); if (error && Str.startsWith(error.message, "Failed to execute 'put' on 'IDBObjectStore'")) { Logger.logAlert('Attempted to set invalid data set in Onyx. Please ensure all data is serializable.'); throw error; } + const errorMessage = error?.message?.toLowerCase?.(); + const errorName = error?.name?.toLowerCase?.(); + const isStorageCapacityError = STORAGE_ERRORS.some((storageError) => errorName?.includes(storageError) || errorMessage?.includes(storageError)); + + if (nextRetryAttempt > MAX_STORAGE_OPERATION_RETRY_ATTEMPTS) { + Logger.logAlert(`Storage operation failed after 5 retries. Error: ${error}. onyxMethod: ${onyxMethod.name}.`); + return Promise.resolve(); + } + + if (!isStorageCapacityError) { + // @ts-expect-error No overload matches this call. + return onyxMethod(defaultParams, nextRetryAttempt); + } + // Find the first key that we can remove that has no subscribers in our blocklist const keyForRemoval = cache.getKeyForEviction(); if (!keyForRemoval) { @@ -951,7 +986,7 @@ function evictStorageAndRetry onyxMethod(...args)); + return remove(keyForRemoval).then(() => onyxMethod(defaultParams, nextRetryAttempt)); } /** @@ -1286,21 +1321,222 @@ function updateSnapshots(data: OnyxUpdate[], mergeFn: typeof Onyx.merge): Array< return promises; } +/** + * Writes a value to our store with the given key. + * Serves as core implementation for `Onyx.set()` public function, the difference being + * that this internal function allows passing an additional `retryAttempt` parameter to retry on failure. + * + * @param params - set parameters + * @param params.key ONYXKEY to set + * @param params.value value to store + * @param params.options optional configuration object + * @param retryAttempt retry attempt + */ +function setWithRetry({key, value, options}: SetParams, retryAttempt?: number): Promise { + // When we use Onyx.set to set a key we want to clear the current delta changes from Onyx.merge that were queued + // before the value was set. If Onyx.merge is currently reading the old value from storage, it will then not apply the changes. + if (OnyxUtils.hasPendingMergeForKey(key)) { + delete OnyxUtils.getMergeQueue()[key]; + } + + if (skippableCollectionMemberIDs.size) { + try { + const [, collectionMemberID] = OnyxUtils.splitCollectionMemberKey(key); + if (skippableCollectionMemberIDs.has(collectionMemberID)) { + // The key is a skippable one, so we set the new value to null. + // eslint-disable-next-line no-param-reassign + value = null; + } + } catch (e) { + // The key is not a collection one or something went wrong during split, so we proceed with the function's logic. + } + } + + // Onyx.set will ignore `undefined` values as inputs, therefore we can return early. + if (value === undefined) { + return Promise.resolve(); + } + + const existingValue = cache.get(key, false); + + // If the existing value as well as the new value are null, we can return early. + if (existingValue === undefined && value === null) { + return Promise.resolve(); + } + + // Check if the value is compatible with the existing value in the storage + const {isCompatible, existingValueType, newValueType} = utils.checkCompatibilityWithExistingValue(value, existingValue); + if (!isCompatible) { + Logger.logAlert(logMessages.incompatibleUpdateAlert(key, 'set', existingValueType, newValueType)); + return Promise.resolve(); + } + + // If the change is null, we can just delete the key. + // Therefore, we don't need to further broadcast and update the value so we can return early. + if (value === null) { + OnyxUtils.remove(key); + OnyxUtils.logKeyRemoved(OnyxUtils.METHOD.SET, key); + return Promise.resolve(); + } + + const valueWithoutNestedNullValues = utils.removeNestedNullValues(value) as OnyxValue; + const hasChanged = options?.skipCacheCheck ? true : cache.hasValueChanged(key, valueWithoutNestedNullValues); + + OnyxUtils.logKeyChanged(OnyxUtils.METHOD.SET, key, value, hasChanged); + + // This approach prioritizes fast UI changes without waiting for data to be stored in device storage. + const updatePromise = OnyxUtils.broadcastUpdate(key, valueWithoutNestedNullValues, hasChanged); + + // If the value has not changed and this isn't a retry attempt, calling Storage.setItem() would be redundant and a waste of performance, so return early instead. + if (!hasChanged && !retryAttempt) { + return updatePromise; + } + + return Storage.setItem(key, valueWithoutNestedNullValues) + .catch((error) => OnyxUtils.retryOperation(error, setWithRetry, {key, value: valueWithoutNestedNullValues, options}, retryAttempt)) + .then(() => { + OnyxUtils.sendActionToDevTools(OnyxUtils.METHOD.SET, key, valueWithoutNestedNullValues); + return updatePromise; + }); +} + +/** + * Sets multiple keys and values. + * Serves as core implementation for `Onyx.multiSet()` public function, the difference being + * that this internal function allows passing an additional `retryAttempt` parameter to retry on failure. + * + * @param data object keyed by ONYXKEYS and the values to set + * @param retryAttempt retry attempt + */ +function multiSetWithRetry(data: OnyxMultiSetInput, retryAttempt?: number): Promise { + let newData = data; + + if (skippableCollectionMemberIDs.size) { + newData = Object.keys(newData).reduce((result: OnyxMultiSetInput, key) => { + try { + const [, collectionMemberID] = OnyxUtils.splitCollectionMemberKey(key); + // If the collection member key is a skippable one we set its value to null. + // eslint-disable-next-line no-param-reassign + result[key] = !skippableCollectionMemberIDs.has(collectionMemberID) ? newData[key] : null; + } catch { + // The key is not a collection one or something went wrong during split, so we assign the data to result anyway. + // eslint-disable-next-line no-param-reassign + result[key] = newData[key]; + } + + return result; + }, {}); + } + + const keyValuePairsToSet = OnyxUtils.prepareKeyValuePairsForStorage(newData, true); + + const updatePromises = keyValuePairsToSet.map(([key, value]) => { + // When we use multiSet to set a key we want to clear the current delta changes from Onyx.merge that were queued + // before the value was set. If Onyx.merge is currently reading the old value from storage, it will then not apply the changes. + if (OnyxUtils.hasPendingMergeForKey(key)) { + delete OnyxUtils.getMergeQueue()[key]; + } + + // Update cache and optimistically inform subscribers on the next tick + cache.set(key, value); + return OnyxUtils.scheduleSubscriberUpdate(key, value); + }); + + return Storage.multiSet(keyValuePairsToSet) + .catch((error) => OnyxUtils.retryOperation(error, multiSetWithRetry, newData, retryAttempt)) + .then(() => { + OnyxUtils.sendActionToDevTools(OnyxUtils.METHOD.MULTI_SET, undefined, newData); + return Promise.all(updatePromises); + }) + .then(() => undefined); +} + +/** + * Sets a collection by replacing all existing collection members with new values. + * Any existing collection members not included in the new data will be removed. + * Serves as core implementation for `Onyx.setCollection()` public function, the difference being + * that this internal function allows passing an additional `retryAttempt` parameter to retry on failure. + * + * @param params - collection parameters + * @param params.collectionKey e.g. `ONYXKEYS.COLLECTION.REPORT` + * @param params.collection Object collection keyed by individual collection member keys and values + * @param retryAttempt retry attempt + */ +function setCollectionWithRetry({collectionKey, collection}: SetCollectionParams, retryAttempt?: number): Promise { + let resultCollection: OnyxInputKeyValueMapping = collection; + let resultCollectionKeys = Object.keys(resultCollection); + + // Confirm all the collection keys belong to the same parent + if (!OnyxUtils.doAllCollectionItemsBelongToSameParent(collectionKey, resultCollectionKeys)) { + Logger.logAlert(`setCollection called with keys that do not belong to the same parent ${collectionKey}. Skipping this update.`); + return Promise.resolve(); + } + + if (skippableCollectionMemberIDs.size) { + resultCollection = resultCollectionKeys.reduce((result: OnyxInputKeyValueMapping, key) => { + try { + const [, collectionMemberID] = OnyxUtils.splitCollectionMemberKey(key, collectionKey); + // If the collection member key is a skippable one we set its value to null. + // eslint-disable-next-line no-param-reassign + result[key] = !skippableCollectionMemberIDs.has(collectionMemberID) ? resultCollection[key] : null; + } catch { + // Something went wrong during split, so we assign the data to result anyway. + // eslint-disable-next-line no-param-reassign + result[key] = resultCollection[key]; + } + + return result; + }, {}); + } + resultCollectionKeys = Object.keys(resultCollection); + + return OnyxUtils.getAllKeys().then((persistedKeys) => { + const mutableCollection: OnyxInputKeyValueMapping = {...resultCollection}; + + persistedKeys.forEach((key) => { + if (!key.startsWith(collectionKey)) { + return; + } + if (resultCollectionKeys.includes(key)) { + return; + } + + mutableCollection[key] = null; + }); + + const keyValuePairs = OnyxUtils.prepareKeyValuePairsForStorage(mutableCollection, true, undefined, true); + const previousCollection = OnyxUtils.getCachedCollection(collectionKey); + + // Preserve references for unchanged items in setCollection + const preservedCollection = OnyxUtils.preserveCollectionReferences(keyValuePairs); + + const updatePromise = OnyxUtils.scheduleNotifyCollectionSubscribers(collectionKey, preservedCollection, previousCollection); + + return Storage.multiSet(keyValuePairs) + .catch((error) => OnyxUtils.retryOperation(error, setCollectionWithRetry, {collectionKey, collection}, retryAttempt)) + .then(() => { + OnyxUtils.sendActionToDevTools(OnyxUtils.METHOD.SET_COLLECTION, undefined, mutableCollection); + return updatePromise; + }); + }); +} + /** * Merges a collection based on their keys. * Serves as core implementation for `Onyx.mergeCollection()` public function, the difference being - * that this internal function allows passing an additional `mergeReplaceNullPatches` parameter. + * that this internal function allows passing an additional `mergeReplaceNullPatches` parameter and retries on failure. * - * @param collectionKey e.g. `ONYXKEYS.COLLECTION.REPORT` - * @param collection Object collection keyed by individual collection member keys and values - * @param mergeReplaceNullPatches Record where the key is a collection member key and the value is a list of + * @param params - mergeCollection parameters + * @param params.collectionKey e.g. `ONYXKEYS.COLLECTION.REPORT` + * @param params.collection Object collection keyed by individual collection member keys and values + * @param params.mergeReplaceNullPatches Record where the key is a collection member key and the value is a list of * tuples that we'll use to replace the nested objects of that collection member record with something else. + * @param params.isProcessingCollectionUpdate whether this is part of a collection update operation. + * @param retryAttempt retry attempt */ function mergeCollectionWithPatches( - collectionKey: TKey, - collection: OnyxMergeCollectionInput, - mergeReplaceNullPatches?: MultiMergeReplaceNullPatches, - isProcessingCollectionUpdate = false, + {collectionKey, collection, mergeReplaceNullPatches, isProcessingCollectionUpdate = false}: MergeCollectionWithPatchesParams, + retryAttempt?: number, ): Promise { if (!isValidNonEmptyCollectionForMerge(collection)) { Logger.logInfo('mergeCollection() called with invalid or empty value. Skipping this update.'); @@ -1416,7 +1652,14 @@ function mergeCollectionWithPatches( }); return Promise.all(promises) - .catch((error) => evictStorageAndRetry(error, mergeCollectionWithPatches, collectionKey, resultCollection as OnyxMergeCollectionInput)) + .catch((error) => + retryOperation( + error, + mergeCollectionWithPatches, + {collectionKey, collection: resultCollection as OnyxMergeCollectionInput, mergeReplaceNullPatches, isProcessingCollectionUpdate}, + retryAttempt, + ), + ) .then(() => { sendActionToDevTools(METHOD.MERGE_COLLECTION, undefined, resultCollection); return promiseUpdate; @@ -1428,11 +1671,14 @@ function mergeCollectionWithPatches( /** * Sets keys in a collection by replacing all targeted collection members with new values. * Any existing collection members not included in the new data will not be removed. + * Retries on failure. * - * @param collectionKey e.g. `ONYXKEYS.COLLECTION.REPORT` - * @param collection Object collection keyed by individual collection member keys and values + * @param params - collection parameters + * @param params.collectionKey e.g. `ONYXKEYS.COLLECTION.REPORT` + * @param params.collection Object collection keyed by individual collection member keys and values + * @param retryAttempt retry attempt */ -function partialSetCollection(collectionKey: TKey, collection: OnyxSetCollectionInput): Promise { +function partialSetCollection({collectionKey, collection}: SetCollectionParams, retryAttempt?: number): Promise { let resultCollection: OnyxInputKeyValueMapping = collection; let resultCollectionKeys = Object.keys(resultCollection); @@ -1472,7 +1718,7 @@ function partialSetCollection(collectionKey: TKe const updatePromise = scheduleNotifyCollectionSubscribers(collectionKey, preservedCollection, previousCollection); return Storage.multiSet(keyValuePairs) - .catch((error) => evictStorageAndRetry(error, partialSetCollection, collectionKey, collection)) + .catch((error) => retryOperation(error, partialSetCollection, {collectionKey, collection}, retryAttempt)) .then(() => { sendActionToDevTools(METHOD.SET_COLLECTION, undefined, mutableCollection); return updatePromise; @@ -1528,7 +1774,7 @@ const OnyxUtils = { scheduleNotifyCollectionSubscribers, remove, reportStorageQuota, - evictStorageAndRetry, + retryOperation, broadcastUpdate, hasPendingMergeForKey, prepareKeyValuePairsForStorage, @@ -1555,6 +1801,9 @@ const OnyxUtils = { preserveCollectionReferencesAfterMerge, logKeyChanged, logKeyRemoved, + setWithRetry, + multiSetWithRetry, + setCollectionWithRetry, }; GlobalSettings.addGlobalSettingsChangeListener(({enablePerformanceMetrics}) => { @@ -1590,7 +1839,7 @@ GlobalSettings.addGlobalSettingsChangeListener(({enablePerformanceMetrics}) => { // @ts-expect-error Reassign reportStorageQuota = decorateWithMetrics(reportStorageQuota, 'OnyxUtils.reportStorageQuota'); // @ts-expect-error Complex type signature - evictStorageAndRetry = decorateWithMetrics(evictStorageAndRetry, 'OnyxUtils.evictStorageAndRetry'); + retryOperation = decorateWithMetrics(retryOperation, 'OnyxUtils.retryOperation'); // @ts-expect-error Reassign broadcastUpdate = decorateWithMetrics(broadcastUpdate, 'OnyxUtils.broadcastUpdate'); // @ts-expect-error Reassign @@ -1601,6 +1850,12 @@ GlobalSettings.addGlobalSettingsChangeListener(({enablePerformanceMetrics}) => { tupleGet = decorateWithMetrics(tupleGet, 'OnyxUtils.tupleGet'); // @ts-expect-error Reassign subscribeToKey = decorateWithMetrics(subscribeToKey, 'OnyxUtils.subscribeToKey'); + // @ts-expect-error Reassign + setWithRetry = decorateWithMetrics(setWithRetry, 'OnyxUtils.setWithRetry'); + // @ts-expect-error Reassign + multiSetWithRetry = decorateWithMetrics(multiSetWithRetry, 'OnyxUtils.multiSetWithRetry'); + // @ts-expect-error Reassign + setCollectionWithRetry = decorateWithMetrics(setCollectionWithRetry, 'OnyxUtils.setCollectionWithRetry'); }); export type {OnyxMethod}; diff --git a/lib/types.ts b/lib/types.ts index 7b9ddb2a..c802814c 100644 --- a/lib/types.ts +++ b/lib/types.ts @@ -380,6 +380,31 @@ type SetOptions = { skipCacheCheck?: boolean; }; +type SetParams = { + key: TKey; + value: OnyxSetInput; + options?: SetOptions; +}; + +type SetCollectionParams = { + collectionKey: TKey; + collection: OnyxSetCollectionInput; +}; + +type MergeCollectionWithPatchesParams = { + collectionKey: TKey; + collection: OnyxMergeCollectionInput; + mergeReplaceNullPatches?: MultiMergeReplaceNullPatches; + isProcessingCollectionUpdate?: boolean; +}; + +type RetriableOnyxOperation = + | typeof OnyxUtils.setWithRetry + | typeof OnyxUtils.multiSetWithRetry + | typeof OnyxUtils.setCollectionWithRetry + | typeof OnyxUtils.mergeCollectionWithPatches + | typeof OnyxUtils.partialSetCollection; + /** * Represents the options used in `Onyx.init()` method. */ @@ -488,6 +513,10 @@ export type { OnyxValue, Selector, SetOptions, + SetParams, + SetCollectionParams, + MergeCollectionWithPatchesParams, MultiMergeReplaceNullPatches, MixedOperationsQueue, + RetriableOnyxOperation, }; diff --git a/tests/perf-test/OnyxUtils.perf-test.ts b/tests/perf-test/OnyxUtils.perf-test.ts index 5a817b85..ea8844a2 100644 --- a/tests/perf-test/OnyxUtils.perf-test.ts +++ b/tests/perf-test/OnyxUtils.perf-test.ts @@ -9,7 +9,7 @@ import OnyxUtils, {clearOnyxUtilsInternals} from '../../lib/OnyxUtils'; import type GenericCollection from '../utils/GenericCollection'; import type {OnyxUpdate} from '../../lib/Onyx'; import createDeferredTask from '../../lib/createDeferredTask'; -import type {OnyxInputKeyValueMapping} from '../../lib/types'; +import type {OnyxInputKeyValueMapping, RetriableOnyxOperation} from '../../lib/types'; const ONYXKEYS = { TEST_KEY: 'test', @@ -281,7 +281,7 @@ describe('OnyxUtils', () => { const changedReportActions = Object.fromEntries( Object.entries(mockedReportActionsMap).map(([k, v]) => [k, randBoolean() ? v : createRandomReportAction(Number(v.reportActionID))] as const), ) as GenericCollection; - await measureAsyncFunction(() => OnyxUtils.partialSetCollection(collectionKey, changedReportActions), { + await measureAsyncFunction(() => OnyxUtils.partialSetCollection({collectionKey, collection: changedReportActions}), { beforeEach: async () => { await Onyx.setCollection(collectionKey, mockedReportActionsMap as GenericCollection); }, @@ -502,12 +502,12 @@ describe('OnyxUtils', () => { }); }); - describe('evictStorageAndRetry', () => { + describe('retryOperation', () => { test('one call', async () => { const error = new Error(); - const onyxMethod = jest.fn() as typeof Onyx.set; + const onyxMethod = jest.fn() as RetriableOnyxOperation; - await measureAsyncFunction(() => OnyxUtils.evictStorageAndRetry(error, onyxMethod, '', null), { + await measureAsyncFunction(() => OnyxUtils.retryOperation(error, onyxMethod, {key: '', value: null}, 1), { beforeEach: async () => { mockedReportActionsKeys.forEach((key) => OnyxCache.addLastAccessedKey(key, false)); OnyxCache.addLastAccessedKey(`${ONYXKEYS.COLLECTION.EVICTABLE_TEST_KEY}1`, false); diff --git a/tests/unit/cacheEvictionTest.ts b/tests/unit/cacheEvictionTest.ts index 8e9e83b0..56585b50 100644 --- a/tests/unit/cacheEvictionTest.ts +++ b/tests/unit/cacheEvictionTest.ts @@ -45,7 +45,7 @@ test('Cache eviction', () => { const setItemMock = jest.fn(originalSetItem).mockImplementationOnce( () => new Promise((_resolve, reject) => { - reject(); + reject(new Error('out of memory')); }), ); StorageMock.setItem = setItemMock; diff --git a/tests/unit/onyxUtilsTest.ts b/tests/unit/onyxUtilsTest.ts index 25f6d6d3..b4f49e54 100644 --- a/tests/unit/onyxUtilsTest.ts +++ b/tests/unit/onyxUtilsTest.ts @@ -4,6 +4,7 @@ import type {GenericDeepRecord} from '../types'; import utils from '../../lib/utils'; import type {Collection, OnyxCollection} from '../../lib/types'; import type GenericCollection from '../utils/GenericCollection'; +import StorageMock from '../../lib/storage'; const testObject: GenericDeepRecord = { a: 'a', @@ -82,6 +83,8 @@ Onyx.init({ beforeEach(() => Onyx.clear()); +afterEach(() => jest.clearAllMocks()); + describe('OnyxUtils', () => { describe('splitCollectionMemberKey', () => { describe('should return correct values', () => { @@ -155,11 +158,14 @@ describe('OnyxUtils', () => { } as GenericCollection); // Replace with new collection data - await OnyxUtils.partialSetCollection(ONYXKEYS.COLLECTION.ROUTES, { - [routeA]: {name: 'New Route A'}, - [routeB]: {name: 'New Route B'}, - [routeC]: {name: 'New Route C'}, - } as GenericCollection); + await OnyxUtils.partialSetCollection({ + collectionKey: ONYXKEYS.COLLECTION.ROUTES, + collection: { + [routeA]: {name: 'New Route A'}, + [routeB]: {name: 'New Route B'}, + [routeC]: {name: 'New Route C'}, + } as GenericCollection, + }); expect(result).toEqual({ [routeA]: {name: 'New Route A'}, @@ -185,7 +191,7 @@ describe('OnyxUtils', () => { [routeA]: {name: 'Route A'}, } as GenericCollection); - await OnyxUtils.partialSetCollection(ONYXKEYS.COLLECTION.ROUTES, {} as GenericCollection); + await OnyxUtils.partialSetCollection({collectionKey: ONYXKEYS.COLLECTION.ROUTES, collection: {} as GenericCollection}); expect(result).toEqual({ [routeA]: {name: 'Route A'}, @@ -209,9 +215,11 @@ describe('OnyxUtils', () => { [routeA]: {name: 'Route A'}, } as GenericCollection); - await OnyxUtils.partialSetCollection(ONYXKEYS.COLLECTION.ROUTES, { - // @ts-expect-error invalidRoute is not a valid key - [invalidRoute]: {name: 'Invalid Route'}, + await OnyxUtils.partialSetCollection({ + collectionKey: ONYXKEYS.COLLECTION.ROUTES, + collection: { + [invalidRoute]: {name: 'Invalid Route'}, + } as GenericCollection, }); expect(result).toEqual({ @@ -410,4 +418,44 @@ describe('OnyxUtils', () => { ]); }); }); + + describe('retryOperation', () => { + const retryOperationSpy = jest.spyOn(OnyxUtils, 'retryOperation'); + const genericError = new Error('Generic storage error'); + const invalidDataError = new Error("Failed to execute 'put' on 'IDBObjectStore': invalid data"); + const memoryError = new Error('out of memory'); + + it('should retry only one time if the operation is firstly failed and then passed', async () => { + StorageMock.setItem = jest.fn(StorageMock.setItem).mockRejectedValueOnce(genericError).mockImplementation(StorageMock.setItem); + + await Onyx.set(ONYXKEYS.TEST_KEY, {test: 'data'}); + + // Should be called once, since Storage.setItem if failed only once + expect(retryOperationSpy).toHaveBeenCalledTimes(1); + }); + + it('should stop retrying after MAX_STORAGE_OPERATION_RETRY_ATTEMPTS retries for failing operation', async () => { + StorageMock.setItem = jest.fn().mockRejectedValue(genericError); + + await Onyx.set(ONYXKEYS.TEST_KEY, {test: 'data'}); + + // Should be called 6 times: initial attempt + 5 retries (MAX_STORAGE_OPERATION_RETRY_ATTEMPTS) + expect(retryOperationSpy).toHaveBeenCalledTimes(6); + }); + + it("should throw error for if operation failed with \"Failed to execute 'put' on 'IDBObjectStore': invalid data\" error", async () => { + StorageMock.setItem = jest.fn().mockRejectedValueOnce(invalidDataError); + + await expect(Onyx.set(ONYXKEYS.TEST_KEY, {test: 'data'})).rejects.toThrow(invalidDataError); + }); + + it('should not retry in case of storage capacity error and no keys to evict', async () => { + StorageMock.setItem = jest.fn().mockRejectedValue(memoryError); + + await Onyx.set(ONYXKEYS.TEST_KEY, {test: 'data'}); + + // Should only be called once since there are no evictable keys + expect(retryOperationSpy).toHaveBeenCalledTimes(1); + }); + }); });