Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RateLimiterMongo: add option not to create indexes (a must for for serverless environments) #103

Open
tomyam1 opened this issue Jan 12, 2021 · 1 comment
Labels
enhancement New feature or request

Comments

@tomyam1
Copy link

tomyam1 commented Jan 12, 2021

We use RateLimiterMongo on a server running on AWS lambda.

Since RateLimiterMongo calls createIndex on creation, it results in calling createIndex for every lambda invocation that is a cold start.

This has two issues:

  1. Causes lambda to crash with UnhandledPromiseRejection, for example
Unhandled Promise Rejection 	{"errorType":"Runtime.UnhandledPromiseRejection","errorMessage":"MongoError: Pool was force destroyed","reason":{"errorType":"MongoError","errorMessage":"Pool was force destroyed","name":"MongoError","stack":["MongoError: Pool was force destroyed","    at Pool.destroy (/var/task/node_modules/mongodb/lib/core/connection/pool.js:679:21)","    at Server.destroy (/var/task/node_modules/mongodb/lib/core/topologies/server.js:902:15)","    at Timeout._onTimeout (/var/task/node_modules/mongodb/lib/core/topologies/replset.js:357:26)","    at listOnTimeout (internal/timers.js:554:17)","    at processTimers (internal/timers.js:497:7)"]},"promise":{},"stack":["Runtime.UnhandledPromiseRejection: MongoError: Pool was force destroyed","    at process.<anonymous> (/var/runtime/index.js:35:15)","    at process.emit (events.js:326:22)","    at process.EventEmitter.emit (domain.js:483:12)","    at processPromiseRejections (internal/process/promises.js:209:33)","    at processTicksAndRejections (internal/process/task_queues.js:98:32)"]}

This is because createIndex is called without waiting for the result. If the its called in one lambda call and it finished before its done it will throw in the following lambda invocation.

  1. Performance

I would recommend:

  1. Adding option to choose if createIndex should be called at all
  2. Add a method public method to call createIndex with async
  3. Catch errors from the createIndex calls in _initCollection with rethrow. It's very difficult to find the reason for the UnhandledPromiseRejection otherwise.

This is what we currently use to alleviate the issue:

/**
 * Rate limiter that doesn't create indexes on initialize
 * Instead we create the indexes once on init
 */
export class RateLimiterMongoose extends RateLimiterMongo {
    public readonly tableName: string;
    protected client: Connection;
    protected _collection: Collection

    /**
     * Override _initCollection and don't call createIndex
     *
     * Source: https://github.com/animir/node-rate-limiter-flexible/blob/v2.2.1/lib/RateLimiterMongo.js#L78
     */
    protected _initCollection(): void {
        this._collection = this.client.collection(this.tableName);
    }

    /**
     * Create the indexes required, but unlike the original _initCollection, use await to catch for errors
     */
    public async initIndexes(): Promise<void> {
        const collection = this.client.collection(this.tableName);
        await collection.createIndex({ expire: -1 }, { expireAfterSeconds: 0 });
        await collection.createIndex(Object.assign({}, this.indexKeyPrefix, { key: 1 }), { unique: true });
    }
}
@animir animir added the enhancement New feature or request label Jan 13, 2021
@animir animir changed the title Add option not to create indexes on RateLimiterMongo (a must for for serverless environments) RateLimiterMongo: add option not to create indexes (a must for for serverless environments) Oct 30, 2021
@Systerr
Copy link

Systerr commented Sep 20, 2023

This will be useful for testing too.
In our case we are using mongodb-memory server for testing and this affect us too

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants