Output to database

[dejurin]

Hi, there 👋

The flex architecture of the project allows you to write integration modules. I see that a database module like MongoDB is not implemented.

Are you considering support in the future? I have now written my own, a variant. I would like to use your program out of the box without my rebuilds.

Thank you.

[philippta]

Yes, that was the intention of having this modular architecture, so that the functionality can be easily extended.

I didn’t have MongoDB planned yet, as I have little insights in what environments people use flyscrape, but I’m happy to support it.

If you would like, and already have a module for it, feel free to submit a pull request and we can discuss it there.

What I would like to see (and my mongodb knowledge is very limited) is a config option to enable it, like so:

output: {
    mongodb: {
        uri: "mongodb://...",
        database: "exampledb",
        collection: "news",
    },
},

[dejurin]

Exactly as you suggested, because it fits in with the organic nature of your programme and parse args, sure ;-)

I actually do that

go run main.go run ./test.js \
  --output.mongodb.uri="mongodb://localhost:27017" \
  --output.mongodb.database="db" \
  --output.mongodb.collection="col"

[philippta]

Yes, those command line arguments directly map to the config object when parsed. 👍🏼