Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Firestore: Make client-side indexing code tree-shakeable #7902

Open
wants to merge 43 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
39f2008
Add two more size reports: "CSI Auto Indexing" and "CSI Delete All In…
dconeybe Dec 15, 2023
dd357bf
Rename size reports for CSI, and include disablePersistentCacheIndexA…
dconeybe Dec 15, 2023
5c68017
work towards making index auto-creation code tree-shakeable
dconeybe Dec 15, 2023
aacdb90
persistent_cache_index_manager.ts: make PersistentCacheIndexManager a…
dconeybe Dec 18, 2023
8d99691
Merge remote-tracking branch 'origin/master' into CsiTreeShake
dconeybe Dec 18, 2023
dfca741
query_engine.test.ts: fixed build
dconeybe Dec 18, 2023
d878f45
local_store.test.ts: fix build
dconeybe Dec 18, 2023
558574e
local_store_indexeddb.test.ts: fix build
dconeybe Dec 18, 2023
90cc022
Merge remote-tracking branch 'origin/master' into CsiTreeShake
dconeybe Dec 21, 2023
fbf3b35
fixed unit tests
dconeybe Dec 21, 2023
3e1287b
fix disablePersistentCacheIndexAutoCreation() and deleteAllPersistent…
dconeybe Dec 21, 2023
08afa67
Move CSI code from IndexManager to FieldIndexManagementApiImpl
dconeybe Dec 21, 2023
cd321f9
mess to switch code from using index manager to the new class
dconeybe Dec 21, 2023
7a965ad
Revert "mess to switch code from using index manager to the new class"
dconeybe Dec 21, 2023
ea333de
Revert "Move CSI code from IndexManager to FieldIndexManagementApiImpl"
dconeybe Dec 21, 2023
59b51ae
Introduce FieldIndexManagementApiFactory and create FieldIndexManagem…
dconeybe Dec 21, 2023
4114f11
Move IndexManager reference into FieldIndexManagementApiImpl
dconeybe Dec 21, 2023
0a337ed
Make IndexBackfiller pluggable in component_provider.ts
dconeybe Dec 21, 2023
2b6ee85
Move IndexBackfiller from 'local' to 'index' and fix some related, an…
dconeybe Dec 21, 2023
1f640e2
Add onNewInstance(() method to the factory
dconeybe Dec 22, 2023
7573c23
Delete FieldIndexManagementApiFactory in favor of an initialize() method
dconeybe Dec 22, 2023
d79a9bb
fix local_store.test.ts and test_field_index_management_api.ts
dconeybe Dec 22, 2023
8b269cd
Remove .only and yarn prettier, which I forgot on the last commit
dconeybe Dec 22, 2023
d50e5bc
fix local_store_indexeddb.test.ts
dconeybe Dec 22, 2023
0bce38f
fix spec_test_runner.ts
dconeybe Dec 22, 2023
955e86d
local_store_indexeddb.test.ts: remove .only()
dconeybe Dec 22, 2023
64ad7ef
fix query_engine.test.ts
dconeybe Dec 22, 2023
1b57573
local_store_indexeddb.test.ts: avoid spurious and confusing error if …
dconeybe Dec 22, 2023
79b6105
Wire up IndexBackfiller
dconeybe Dec 22, 2023
cd9cb53
Move IndexManager code into FieldIndexManagementApiImpl
dconeybe Dec 22, 2023
921b068
Merge remote-tracking branch 'origin/master' into CsiTreeShake
dconeybe Dec 24, 2023
12c47db
Merge remote-tracking branch 'origin/master' into CsiTreeShake
dconeybe Jan 2, 2024
9142f67
Merge remote-tracking branch 'origin/master' into CsiTreeShake
dconeybe Jan 3, 2024
9f2fe66
Merge remote-tracking branch 'origin/master' into CsiTreeShake
dconeybe Jan 4, 2024
7be9012
Rewrite everything
dconeybe Jan 5, 2024
3afba98
undo noisy, unrelated import ordering changes
dconeybe Jan 5, 2024
bfcc207
yarn changeset
dconeybe Jan 5, 2024
2ae3cee
Remove changes from PR 7921 (Firestore: Add support for MOCK_PERSISTE…
dconeybe Jan 5, 2024
ccfe206
Remove changes from PR 7922 (Firestore: local_store_indexeddb.test.ts…
dconeybe Jan 5, 2024
0096fa9
tweak changelog
dconeybe Jan 5, 2024
0b34fdd
Merge remote-tracking branch 'origin/master' into CsiTreeShake
dconeybe Jan 9, 2024
485020c
Merge remote-tracking branch 'origin/master' into CsiTreeShake_PR7902
dconeybe Jan 11, 2024
bef8149
Merge remote-tracking branch 'origin/master' into CsiTreeShake
dconeybe May 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changeset/silly-vans-fix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
'@firebase/firestore': patch
'firebase': patch
---

Refactor client-side indexing to be tree-shakable. This can reduce code size by up to 23 kB for applications that enable IndexedDb persistence but do _not_ call either `enablePersistentCacheIndexAutoCreation()` or `setIndexConfiguration()`.
55 changes: 17 additions & 38 deletions packages/firestore/src/api/persistent_cache_index_manager.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,8 @@

import {
firestoreClientDeleteAllFieldIndexes,
firestoreClientSetPersistentCacheIndexAutoCreationEnabled,
firestoreClientDisablePersistentCacheIndexAutoCreation,
firestoreClientEnablePersistentCacheIndexAutoCreation,
FirestoreClient
} from '../core/firestore_client';
import { cast } from '../util/input_validation';
Expand Down Expand Up @@ -76,7 +77,12 @@ export function getPersistentCacheIndexManager(
export function enablePersistentCacheIndexAutoCreation(
indexManager: PersistentCacheIndexManager
): void {
setPersistentCacheIndexAutoCreationEnabled(indexManager, true);
indexManager._client.verifyNotTerminated();
firestoreClientEnablePersistentCacheIndexAutoCreation(indexManager._client)
.then(_ => logDebug('enablePersistentCacheIndexAutoCreation() succeeded.'))
.catch(error =>
logWarn('enablePersistentCacheIndexAutoCreation() failed', error)
);
}

/**
Expand All @@ -87,7 +93,12 @@ export function enablePersistentCacheIndexAutoCreation(
export function disablePersistentCacheIndexAutoCreation(
indexManager: PersistentCacheIndexManager
): void {
setPersistentCacheIndexAutoCreationEnabled(indexManager, false);
indexManager._client.verifyNotTerminated();
firestoreClientDisablePersistentCacheIndexAutoCreation(indexManager._client)
.then(_ => logDebug('disablePersistentCacheIndexAutoCreation() succeeded.'))
.catch(error =>
logWarn('disablePersistentCacheIndexAutoCreation() failed', error)
);
}

/**
Expand All @@ -100,41 +111,9 @@ export function deleteAllPersistentCacheIndexes(
indexManager: PersistentCacheIndexManager
): void {
indexManager._client.verifyNotTerminated();

const promise = firestoreClientDeleteAllFieldIndexes(indexManager._client);

promise
.then(_ => logDebug('deleting all persistent cache indexes succeeded'))
.catch(error =>
logWarn('deleting all persistent cache indexes failed', error)
);
}

function setPersistentCacheIndexAutoCreationEnabled(
indexManager: PersistentCacheIndexManager,
isEnabled: boolean
): void {
indexManager._client.verifyNotTerminated();

const promise = firestoreClientSetPersistentCacheIndexAutoCreationEnabled(
indexManager._client,
isEnabled
);

promise
.then(_ =>
logDebug(
`setting persistent cache index auto creation ` +
`isEnabled=${isEnabled} succeeded`
)
)
.catch(error =>
logWarn(
`setting persistent cache index auto creation ` +
`isEnabled=${isEnabled} failed`,
error
)
);
firestoreClientDeleteAllFieldIndexes(indexManager._client)
.then(_ => logDebug('deleteAllPersistentCacheIndexes() succeeded.'))
.catch(error => logWarn('deleteAllPersistentCacheIndexes() failed', error));
}

/**
Expand Down
126 changes: 81 additions & 45 deletions packages/firestore/src/core/component_provider.ts
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ import { JsonProtoSerializer } from '../remote/serializer';
import { hardAssert } from '../util/assert';
import { AsyncQueue } from '../util/async_queue';
import { Code, FirestoreError } from '../util/error';
import { logDebug } from '../util/log';

import { DatabaseInfo } from './database_info';
import { EventManager, newEventManager } from './event_manager';
Expand Down Expand Up @@ -90,6 +91,7 @@ export interface ComponentConfiguration {
* cache. Implementations override `initialize()` to provide all components.
*/
export interface OfflineComponentProvider {
asyncQueue: AsyncQueue;
persistence: Persistence;
sharedClientState: SharedClientState;
localStore: LocalStore;
Expand All @@ -109,16 +111,23 @@ export interface OfflineComponentProvider {
export class MemoryOfflineComponentProvider
implements OfflineComponentProvider
{
asyncQueue!: AsyncQueue;
persistence!: Persistence;
sharedClientState!: SharedClientState;
localStore!: LocalStore;
gcScheduler!: Scheduler | null;
indexBackfillerScheduler!: Scheduler | null;
indexBackfillerScheduler: Scheduler | null = null;
synchronizeTabs = false;

serializer!: JsonProtoSerializer;

get schedulers(): Scheduler[] {
const schedulers = [this.gcScheduler, this.indexBackfillerScheduler];
return schedulers.filter(scheduler => !!scheduler) as Scheduler[];
}

async initialize(cfg: ComponentConfiguration): Promise<void> {
this.asyncQueue = cfg.asyncQueue;
this.serializer = newSerializer(cfg.databaseInfo.databaseId);
this.sharedClientState = this.createSharedClientState(cfg);
this.persistence = this.createPersistence(cfg);
Expand All @@ -128,10 +137,6 @@ export class MemoryOfflineComponentProvider
cfg,
this.localStore
);
this.indexBackfillerScheduler = this.createIndexBackfillerScheduler(
cfg,
this.localStore
);
}

createGarbageCollectionScheduler(
Expand All @@ -141,13 +146,6 @@ export class MemoryOfflineComponentProvider
return null;
}

createIndexBackfillerScheduler(
cfg: ComponentConfiguration,
localStore: LocalStore
): Scheduler | null {
return null;
}

createLocalStore(cfg: ComponentConfiguration): LocalStore {
return newLocalStore(
this.persistence,
Expand All @@ -166,8 +164,9 @@ export class MemoryOfflineComponentProvider
}

async terminate(): Promise<void> {
this.gcScheduler?.stop();
this.indexBackfillerScheduler?.stop();
for (const scheduler of this.schedulers) {
scheduler.stop();
}
this.sharedClientState.shutdown();
await this.persistence.shutdown();
}
Expand Down Expand Up @@ -215,6 +214,8 @@ export class IndexedDbOfflineComponentProvider extends MemoryOfflineComponentPro
indexBackfillerScheduler!: Scheduler | null;
synchronizeTabs = false;

private primaryStateListenerNotified = false;

constructor(
protected readonly onlineComponentProvider: OnlineComponentProvider,
protected readonly cacheSizeBytes: number | undefined,
Expand All @@ -237,19 +238,30 @@ export class IndexedDbOfflineComponentProvider extends MemoryOfflineComponentPro
// NOTE: This will immediately call the listener, so we make sure to
// set it after localStore / remoteStore are started.
await this.persistence.setPrimaryStateListener(() => {
if (this.gcScheduler && !this.gcScheduler.started) {
this.gcScheduler.start();
}
if (
this.indexBackfillerScheduler &&
!this.indexBackfillerScheduler.started
) {
this.indexBackfillerScheduler.start();
}
this.primaryStateListenerNotified = true;
this.startSchedulers();
return Promise.resolve();
});
}

private startSchedulers(): void {
if (!this.primaryStateListenerNotified) {
return;
}

for (const scheduler of this.schedulers) {
if (!scheduler.started) {
scheduler.start();
}
}
}

installIndexBackfillerScheduler(scheduler: IndexBackfillerScheduler): void {
hardAssert(!this.indexBackfillerScheduler);
this.indexBackfillerScheduler = scheduler;
this.startSchedulers();
}

createLocalStore(cfg: ComponentConfiguration): LocalStore {
return newLocalStore(
this.persistence,
Expand All @@ -268,14 +280,6 @@ export class IndexedDbOfflineComponentProvider extends MemoryOfflineComponentPro
return new LruScheduler(garbageCollector, cfg.asyncQueue, localStore);
}

createIndexBackfillerScheduler(
cfg: ComponentConfiguration,
localStore: LocalStore
): Scheduler | null {
const indexBackfiller = new IndexBackfiller(localStore, this.persistence);
return new IndexBackfillerScheduler(cfg.asyncQueue, indexBackfiller);
}

createPersistence(cfg: ComponentConfiguration): IndexedDbPersistence {
const persistenceKey = indexedDbStoragePrefix(
cfg.databaseInfo.databaseId,
Expand Down Expand Up @@ -305,6 +309,30 @@ export class IndexedDbOfflineComponentProvider extends MemoryOfflineComponentPro
}
}

export function indexedDbOfflineComponentProviderInstallFieldIndexPlugin(
componentProvider: IndexedDbOfflineComponentProvider
): void {
if (componentProvider.indexBackfillerScheduler) {
return;
}

logDebug(
'Installing IndexBackfillerScheduler into OfflineComponentProvider ' +
'to support persistent cache indexing.'
);

const indexBackfiller = new IndexBackfiller(
componentProvider.localStore,
componentProvider.persistence
);
const scheduler = new IndexBackfillerScheduler(
componentProvider.asyncQueue,
indexBackfiller
);

componentProvider.installIndexBackfillerScheduler(scheduler);
}

/**
* Provides all components needed for Firestore with multi-tab IndexedDB
* persistence.
Expand All @@ -316,6 +344,8 @@ export class IndexedDbOfflineComponentProvider extends MemoryOfflineComponentPro
export class MultiTabOfflineComponentProvider extends IndexedDbOfflineComponentProvider {
synchronizeTabs = true;

private isPrimary: boolean | null = null;

constructor(
protected readonly onlineComponentProvider: OnlineComponentProvider,
protected readonly cacheSizeBytes: number | undefined
Expand Down Expand Up @@ -346,27 +376,33 @@ export class MultiTabOfflineComponentProvider extends IndexedDbOfflineComponentP
// NOTE: This will immediately call the listener, so we make sure to
// set it after localStore / remoteStore are started.
await this.persistence.setPrimaryStateListener(async isPrimary => {
this.isPrimary = isPrimary;

await syncEngineApplyPrimaryState(
this.onlineComponentProvider.syncEngine,
isPrimary
);
if (this.gcScheduler) {
if (isPrimary && !this.gcScheduler.started) {
this.gcScheduler.start();
} else if (!isPrimary) {
this.gcScheduler.stop();
}
}
if (this.indexBackfillerScheduler) {
if (isPrimary && !this.indexBackfillerScheduler.started) {
this.indexBackfillerScheduler.start();
} else if (!isPrimary) {
this.indexBackfillerScheduler.stop();
}
}

this.startOrStopSchedulers();
});
}

private startOrStopSchedulers(): void {
for (const scheduler of this.schedulers) {
if (this.isPrimary === true && !scheduler.started) {
scheduler.start();
} else if (this.isPrimary === false) {
scheduler.stop();
}
}
}

installIndexBackfillerScheduler(scheduler: IndexBackfillerScheduler): void {
hardAssert(!this.indexBackfillerScheduler);
this.indexBackfillerScheduler = scheduler;
this.startOrStopSchedulers();
}

createSharedClientState(cfg: ComponentConfiguration): SharedClientState {
const window = getWindow();
if (!WebStorageSharedClientState.isAvailable(window)) {
Expand Down
Loading
Loading