Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Having problems trying to use duckdb-async in Next.js #6

Open
Kennyac opened this issue May 12, 2023 · 7 comments
Open

Having problems trying to use duckdb-async in Next.js #6

Kennyac opened this issue May 12, 2023 · 7 comments

Comments

@Kennyac
Copy link

Kennyac commented May 12, 2023

When running on next.js I recieve the following error

  • error ./node_modules/@mapbox/node-pre-gyp/lib/util/nw-pre-gyp/index.html
    Module parse failed: Unexpected token (1:0)
    You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders

<!doctype html>
|
|

Import trace for requested module:
./node_modules/@mapbox/node-pre-gyp/lib/util/nw-pre-gyp/index.html
./node_modules/@mapbox/node-pre-gyp/lib/ sync ^./.*$
./node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js
./node_modules/duckdb/lib/duckdb-binding.js
./node_modules/duckdb/lib/duckdb.js
./node_modules/duckdb-async/dist/duckdb-async.js
./src/app/api/duckdb/route.ts

I have a simple example project showing the problem
https://drive.google.com/file/d/1dceXEXiaH01-SJKxu2ckgoWCTf8f79Rm/view?usp=sharing

unzip form the above link
to reproduce
yarn install
yarn dev
browse to http://localhost:3001/api/duckdb
@aaronjbecker
Copy link

I tried following the error messages and installing a webpack loader for the HTML dependency and wiring that up in my next.config.js, but that just led to more errors related to some dependencies on mock-aws-s3, aws-sdk, and finally another webpack error related to a C# file:
error ./node_modules/node-gyp/lib/Find-VisualStudio.cs Module parse failed: Unexpected token (9:6) You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders | // This script needs to be compatible with PowerShell v2 to run on Windows 2008R2 and Windows 7. | > using System; | using System.Text; | using System.Runtime.InteropServices; Import trace for requested module: ./node_modules/node-gyp/lib/Find-VisualStudio.cs ./node_modules/node-gyp/lib/ sync ^\.\/.*$ ./node_modules/node-gyp/lib/node-gyp.js ./node_modules/@mapbox/node-pre-gyp/lib/util/compile.js ./node_modules/@mapbox/node-pre-gyp/lib/ sync ^\.\/.*$ ./node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js ./node_modules/duckdb/lib/duckdb-binding.js ./node_modules/duckdb/lib/duckdb.js ./node_modules/duckdb-async/dist/duckdb-async.js

At this point I gave up because I don't understand why webpack would need to load a C# file.

All the cascading dependency issues seem related to mapbox integration. Is there a way to isolate that functionality?
For the time being I'm just using DuckDB from Python with a private API on the backend...

@antonycourtney
Copy link
Contributor

I have only done small experiments with Next.js, so forgive any lack of understanding, but...what are you trying to do?

duckdb-async is just a thin wrapper around the duckdb node.js bindings (duckdb npm package). The duckdb npm package contains a build of the DuckDb library (C++ code) wrapped with node.js addins.

So the duckdb npm by module (and, by extension, duckdb-async) can only ever run server-side, in a node.js process.
I'm not entirely sure what Next.js is trying to do here, but the fact that webpack involves typically suggests bundling for the browser. That's not going to work for duckdb or duckd-async.
There is an adjacent project to duckdb, duckdb-wasm, for running DuckDb in a browser using WebAssembly if that's what you are looking for.

Hope that helps.

@mattf96s
Copy link

mattf96s commented Jun 27, 2023 •
edited
Loading

You won't be able to get this working 😅—the original DuckDB node package is the issue. I gave up and instead make a post request to a custom lambda layer as explained in this repo. It's not ideal—there's a wasm version too (if it appropriate for your use case).

See the node issues here:

@Kennyac
Copy link
Author

Kennyac commented Jun 27, 2023

Thanks for your help

My main aim here was use duckdb for a database access layer at the server side

I endded up writing a separate fastify service to handle my requests works but would have been nice for this to be used in directly in next.js

@tobilg
Copy link

tobilg commented Nov 9, 2023

I maintain https://www.npmjs.com/package/duckdb-lambda-x86 for exactly the reason @mattf96s explained. I uses the correct GLIBC versions for Vercel (which runs on Lambda).

@pencilcheck
Copy link

vercel/next.js#49709

but somehow the poster said it doesn't work for duckdb-async, which is weird if this package only is a thin wrapper around duckdb

@pencilcheck
Copy link

Btw, copying my answer from the vercel issue here:

If you are on nextjs 14, try this

  experimental: {
    serverComponentsExternalPackages: ["duckdb", "duckdb-async"],
  },

duckdb-async works with this configuration above on nextjs 14

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@pencilcheck @tobilg @antonycourtney @aaronjbecker @Kennyac @mattf96s