Native Dependency loading Error for package sqlite-vec

[matsjfunke]

Why does it try to open vec0.dylib.dylib when only vec0.dylib exsits???

SQLite Vec Extension Loading Error on macOS - vec0.dylib.dylib not found

Description

When launching the Electron app on macOS, it fails to initialize due to an error loading the SQLite vector extension. The error occurs when trying to load the vec0.dylib.dylib file from the sqlite-vec-darwin-arm64 package.

Everything works well in development Environment and the npm run build && electron-builder --mac --arm64 command runs without error the error occurs while opening the app.

Error Message

SqliteError: dlopen(/Users/matsfunke/dev/AppName/AppName.app/Contents/Resources/app.asar/node_modules/sqlite-vec-darwin-arm64/vec0.dylib.dylib, 0x000A): tried: '/Users/matsfunke/dev/AppName/AppName.app/Contents/Resources/app.asar/node_modules/sqlite-vec-darwin-arm64/vec0.dylib.dylib' (errno=20)

Environment

• OS: macOS
• Electron: 30.5.1
• Chrome: 124.0.6367.243
• Node: 20.16.0
• Architecture: arm64 (Apple Silicon)
• better-sqlite: 3.48.0
• sqlite-vec: v0.1.7-alpha.2

Current Configuration

electron-builder.json configuration for unpacking native modules:

 "asar": true,
"asarUnpack": [
  "**/*.dylib",
  "node_modules/better-sqlite3/**/*",
  "node_modules/sqlite-vec/**/*",
  "node_modules/sqlite-vec-darwin-arm64/**/*"
],
  "files": [
    "dist-electron/**/*",
    "dist-react/**/*",
    "node_modules/langchain/**/*",
    "node_modules/@langchain/**/*"
  ],
  "extraResources": [
    "dist-electron/preload.cjs",
    {
      "from": "node_modules/langchain",
      "to": "node_modules/langchain"
    },
    {
      "from": "node_modules/@langchain",
      "to": "node_modules/@langchain"
    },
    {
      "from": "node_modules/better-sqlite3",
      "to": "node_modules/better-sqlite3",
      "filter": ["**/*"]
    },
    {
      "from": "node_modules/sqlite-vec",
      "to": "node_modules/sqlite-vec",
      "filter": ["**/*"]
    },
    {
      "from": "node_modules/sqlite-vec-darwin-arm64",
      "to": "node_modules/sqlite-vec-darwin-arm64",
      "filter": ["**/*"]
    }
  ],

Expected Behavior

The application should successfully load the SQLite vector extension (from AppName.app/Contents/Resources/app.asar/node_modules/sqlite-vec-darwin-arm64/vec0.dylib) and initialize the database.

Actual Behavior

The application fails to start due to being unable to load the SQLite extension from within the asar archive because it tries to load from AppName.app/Contents/Resources/app.asar/node_modules/sqlite-vec-darwin-arm64/vec0.dylib.dylib.

[mmaietta]

So electron-builder doesn't change the paths of what is being imported or even modify your app/bundled code, electron-builder is a packager and not a bundler.

Everything works well in development Environment

That's because it's loading everything from the filesystem, so yeah, that'll work. Packaged apps are self-contained, so your issue is likely a local misconfiguration. The fact that electron-builder doesn't return an error during packaging confirms it's behaving correctly.

Please provide a minimum reproducible repo, because as it currently stands, there's not much I can do to help with the current info provided.

[matsjfunke]

I have created this repo which reproduces the error.

Can't tell you how much it would mean if you help to fix this!!

[mmaietta]

So I added some console logging to the packages being loaded.
Here:
at Database.loadExtension (/Users/mikemaietta/Development/reproduce-sql-error/dist/mac-arm64/ErrorDemo.app/Contents/Resources/app.asar/node_modules/better-sqlite3/lib/methods/wrappers.js:20:14)

 exports.loadExtension = function loadExtension(...args) {
	console.log({ loadExtension: args });
	this[cppdb].loadExtension(...args);
	return this;
};

And here:
at Module.load (file:///Users/mikemaietta/Development/reproduce-sql-error/dist/mac-arm64/ErrorDemo.app/Contents/Resources/app.asar/node_modules/sqlite-vec/index.mjs:56:6

function getLoadablePath() {
  if (!validPlatform(platform, arch)) {
    throw new Error(
      invalidPlatformErrorMessage
    );
  }
  const packageName = platformPackageName(platform, arch);
  const loadablePath = join(
    fileURLToPath(new URL(join("."), import.meta.url)),
    "..",
    packageName,
    `${ENTRYPOINT_BASE_NAME}.${extensionSuffix(platform)}`
  );
  console.log({ loadablePath });
  if (!statSync(loadablePath, { throwIfNoEntry: false })) {
    throw new Error(extensionNotFoundErrorMessage(packageName));
  }

  return loadablePath;
}

The logs returned the correct values.

{
  loadablePath: '/Users/mikemaietta/Development/reproduce-sql-error/dist/mac-arm64/ErrorDemo.app/Contents/Resources/app.asar/node_modules/sqlite-vec-darwin-arm64/vec0.dylib'
}

{
  loadExtension: [
    '/Users/mikemaietta/Development/reproduce-sql-error/dist/mac-arm64/ErrorDemo.app/Contents/Resources/app.asar/node_modules/sqlite-vec-darwin-arm64/vec0.dylib'
  ]
}

I dove a little deeper and found this:
https://github.com/WiseLibs/better-sqlite3/blob/d07834f4bf04a1338fc51930793deb30ad54d3cc/deps/sqlite3/sqlite3.h#L7203-L7209

** ^The sqlite3_load_extension() interface attempts to load an
** [SQLite extension] library contained in the file zFile.  If
** the file cannot be loaded directly, attempts are made to load
** with various operating-system specific extensions added.
** So for example, if "samplelib" cannot be loaded, then names like
** "samplelib.so" or "samplelib.dylib" or "samplelib.dll" might
** be tried also.

It seems that by default, it'll search for the path you provide vec0.dylib, then if not found, it appends .dylib to the path and tries again.

So the error message re: "vec0.dylib.dylib" is kind of confusing as what I think it's really trying to say is that it can't find vec0.dylib, so it tried vec0.dylib.dylib and still couldn't find that.

[Laurin-Notemann]

That actually really valuable input, I think one of the issues is related to this path reproduce-sql-error/ErrorDemo.app/Contents/Resources/app.asar/node_modules/ because if you check the folder there is no app.asar folder only an app.asar as a folder or app.asar.unpacked.

I don't really know what asar but I assume the error has something to do with that

[Laurin-Notemann]

https://github.com/Laurin-Notemann/reproduce-sql-error

Laurin-Notemann/reproduce-sql-error@2519cfe#diff-fba39aeef1a729d5c1bddd5efcaa258830f30151f95be0fc6657978ccad16249L23

I this line here where it hardcodes the path to use app.asar.unpack instead of app.asar. which solves the error, obvs thats just a patchwork fix

[Laurin-Notemann]

The root issue lies in sqlite-vec's path resolution using import.meta.url. When running in an Electron app, import.meta.url returns a virtual path inside the asar archive (which is a file, not a directory), causing the package to look for the native module in the wrong location. The package needs to be updated to handle Electron's asar packaging system by checking if it's running in a packaged Electron environment and using the correct path to app.asar.unpacked instead. This affects any Electron application using sqlite-vec with native modules packaged in asar archives.