MongoDB PHP Driver 1.15+ increases PHP memory consumption

[jazzsequence]

Bug Report

We (Pantheon / https://github.com/pantheon-systems) compile PHP manually and build a variety of extensions into our PHP image. Recently, I attempted updating MongoDB from 1.14.2 (what we were previously using) to 1.16.2 and received reports that PHP was taking much longer to resolve according to New Relic data. Rolling back to 1.15.x did not resolve, but the difference can be seen after we reverted back to 1.14.2:

We don't actually run MongoDB but provide the driver for users who wish to make MongoDB connections.

Environment

OS: Centos 8 / Debian 20.10.7
PHP: 8+
https://v80-php-info.pantheonsite.io/
https://v81-php-info.pantheonsite.io/
https://v82-php-info.pantheonsite.io/
(We're currently in the process of rolling back the MongoDB driver update on the platform, so the above phpinfo sites may or may not show 1.16/1.14)

Happy to provide any other details I can. I will say that after the update, I needed to manually install libbson and libmongoc, so my suspicion is that those updates are related.

[GromNaN]

I took a screenshot so that we keep track of the phpinfo (same for the 3 PHP versions)

[jazzsequence]

Thanks @GromNaN I only had that thought after I created the ticket. 😅

[jmikola]

received reports that PHP was taking much longer to resolve according to New Relic data. Rolling back to 1.15.x did not resolve, but the difference can be seen after we reverted back to 1.14.2:

What are these reports measuring? In the "Web transactions time" graph you shared, what is "2.64k" referring to for the minute starting at 9:08am? The X axis is clearly time but I don't understand what the Y axis is measuring.

I will say that after the update, I needed to manually install libbson and libmongoc, so my suspicion is that those updates are related.

A few questions on this:

• Is "after the update" referring to installing version 1.16.x of the mongodb extension?
• Does "manually install libbson and libmongoc" refer to specifying --with-mongodb-system-libs=yes when configuring the build, as described in the manual installation docs?
• Can you elaborate on "needed" there? What problems did you encounter attempting to build the driver using the bundled submodules for libmongoc and libmongocrypt?

---

To walk through exactly what the mongodb extension does when loaded (assuming the application doesn't interact with it further), we can look at the various init hooks in php_phongo.c:

• RINIT initializes several internal HashTables
• GINIT initializes another HashTable, increments an internal counter, and resets some memory
• MINIT does a bit more, but the most notable thing is invoking mongoc_init

mongoc_init has had few changes in recent versions, but the most notable I can think of is AWS credential caching in mongodb/mongo-c-driver@8eaac64; however, the PHP driver isn't yet using that functionality so it should not even be compiled per the #if defined(MONGOC_ENABLE_MONGODB_AWS_AUTH) check.

Is there any way for you to get more detail tracing from New Relic that might pinpoint an increase in time spent in various PHP extensions' init hooks? That would help narrow this down further and provide some assurance we're actually looking in the right place.

[jazzsequence]

What are these reports measuring? In the "Web transactions time" graph you shared, what is "2.64k" referring to for the minute starting at 9:08am? The X axis is clearly time but I don't understand what the Y axis is measuring.

The y axis is ms. So, it's showing transaction time over time. The beginning of the graph in the screenshot (8:50 - ~9:15ish) is with 1.16.2 and the drop is after downgrading back to 1.14.2.

• Does "manually install libbson and libmongoc" refer to specifying --with-mongodb-system-libs=yes when configuring the build, as described in the manual installation docs?
• Can you elaborate on "needed" there? What problems did you encounter attempting to build the driver using the bundled submodules for libmongoc and libmongocrypt?

So we use a build script to pull down packages from a variety of locations, unpack them and compile PHP with various flags depending on what version of PHP we're compiling, etc. The script was already running a git clone of the specified version (based on a config file that includes a number of other extensions) and then running a git submodule update --init.

The "manually install" part came in when the PHP build would fail (even when --with-mongodb-system-libs=yes was passed) because libbson and libmongoc weren't found -- so we had to add a yum install -y mongo-c-driver-devel libbson-devel for our Centos boxes and actually build libmongoc from source for Debian (because the debian version of libmongoc in apt wasn't compatible with 1.16). For both we then ran pkg-config --cflags and pkg-config --libs for libbson-1.0 and libmongoc-1.0 before building (with --with-mongodb-system-libs=yes).

I could go into the whys and wherefores of the above changes that were necessary to add 1.16.x, but the tl:dr; for each is "it didn't build PHP". Does that help? Unfortunately, the repository that has this build script is private, but I could probably pull stuff out into gists if necessary.

The New Relic graph was taken from a customer's active live site. Right now, we don't have a test environment that we can run the same kind of tests on (and non-live environments didn't have enough data to really see the before/after effects), but that is something we can (and want to) set up, using load testing tools to throw traffic at it for brief periods of time so we can analyze the results.

[jmikola]

The New Relic graph was taken from a customer's active live site.

We don't actually run MongoDB but provide the driver for users who wish to make MongoDB connections.

Is the customer actually interacting with the MongoDB driver in some way? My original read of the thread was that simply upgrading the mongodb extension in your PHP image introduced overhead in an application that was not using the driver API, which is why I focused on the init hooks, as those are the only parts of the extension that I expect would be invoked automatically.

That said (and apologies for not clarifying this in my last comment), MINIT is only run once per PHP worker process. So even if mongoc_init is responsible for some performance degradation, I would only expect that to impact the first request served by each worker. I see no reason to suspect the RINIT and GINIT functions, which could be invoked more frequently (definitely RINIT; GINIT might only be as frequent as MINIT depending on the SAPI you're using).

If the customer is using MongoDB, I think we'd need to get them involved here to actually investigate further.

If they're not using MongoDB (and assuming we can trust them on that), then I think it's probably correct to focus on MINIT, but you'd need to be able to reproduce this in a test environment so we can start diving into possible culprits.

[jazzsequence]

I am relatively certain that the customer is not using MongoDB in any way and it was just the driver update the caused the performance overhead. And I'm reasonably confident that that's the case because actual MongoDB usage amongst our customers is rare and we saw this on multiple sites (I can't confirm whether it was all sites, so there may be some common thread among those that we knew about, or it could have actually made PHP slower for everyone and we only heard from the ones that complained the loudest).

I agree that we need to have an environment to throw tests at, so that's in our court, but I wanted to create the issue while it was relatively fresh in mind.

[jmikola]

I wanted to create the issue while it was relatively fresh in mind.

Happy to leave this issue open until more info comes to light. Two additional thoughts below.

Unfortunately, the repository that has this build script is private, but I could probably pull stuff out into gists if necessary.

Sharing as much of the build process as possible would be helpful. Ideally, a script that shows all of the steps for how you install libmongoc (let's focus on building manually from a release tarball) and then compiling a particular PHP version with mongodb as a static extension.

Also, version 1.14.2 of the mongodb extension requires libmongoc 1.22.2+, but any never version of libmongoc will work. To try and isolate the issue between the PHP driver and libmongoc, I'd suggest leaving the extension at 1.14.2 but building against a newer version of libmongoc. We can start with:

• libmongoc 1.24.3, which is required by PHP driver 1.16.2
• libmongoc 1.23.4, which is required by PHP driver 1.15.3

The goal here would be to start with a version that easily reproduces the delay and then jump backwards until we can pinpoint a libmongoc version that introduces the issue (akin to a git bisect to find a bug). This would definitely require a test environment, so something to revisit once that's in place.

In the meantime, I'll peruse libmongoc's fixVersions in JIRA from 1.22 onwards to see if any tickets come up that might relevant for changes to mongoc_init().