TypeError: Cannot read property 'stream' of undefined

[vishalgakhare]

Trying out in local where nodeJS = v8.0.0, Mongo DB = 3.6.3

"dependencies": {
    "mongoose": "^5.2.15",
    "stream-to-mongo-db": "^1.5.0",
    "xml2js": "^0.4.19"
  }

Error:

node stream.js
/Users/vish/Projects/testnpm/node_modules/mongoose/node_modules/mquery/lib/collection/node.js:131
  return this.collection.find(match, findOptions).stream(streamOptions);
                                                 ^

TypeError: Cannot read property 'stream' of undefined
    at NodeCollection.findStream (/Users/vish/Projects/testnpm/node_modules/mongoose/node_modules/mquery/lib/collection/node.js:131:50)
    at model.Query.Query.stream (/Users/vish/Projects/testnpm/node_modules/mongoose/node_modules/mquery/lib/mquery.js:3017:27)
    at Object.<anonymous> (/Users/vish/Projects/testnpm/stream.js:54:38)
    at Module._compile (module.js:569:30)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:503:32)
    at tryModuleLoad (module.js:466:12)
    at Function.Module._load (module.js:458:3)
    at Function.Module.runMain (module.js:605:10)
    at startup (bootstrap_node.js:158:16)
    at bootstrap_node.js:575:3

The code is same as given in readme markdown.

const streamToMongoDB = require('stream-to-mongo-db').streamToMongoDB;
const mongoose = require('mongoose');

// where the data will come from
const connection = mongoose.connect('mongodb://admin:[email protected]:27017/dbBiz?authSource=admin&authMechanism=SCRAM-SHA-1');
// Event Schema
const eventsSchema = mongoose.Schema(
    {
        accountIdentifier: { type: Number }     
    }, { collection: 'events' }

);
const MyModel = mongoose.model('events', eventsSchema);

// where the data will end up
const outputDBConfig = { dbURL: 'mongodb://admin:[email protected]:27017/dbBiz?authSource=admin&authMechanism=SCRAM-SHA-1', collection: 'bizeventsConverted' };

// create the writable stream
const writableStream = streamToMongoDB(outputDBConfig);

// create readable stream and consume it
const stream = MyModel.find().lean().stream();

/*
//TODO:
// create the readable stream and transform the data before writing it

*/

stream.pipe(writableStream);

stream.on('end', () => {
    console.log('done!');
    connection.close();
});

[fabienjuif]

Hi 👋

As you can see, the exception is in mongoose.

You copy past a line from the README.md and tried to apply it to mongoose collection but I bet this is not the same API:
README.md

    const stream = db.collection(inputDBConfig.collection).find().stream();

Your code

   const stream = MyModel.find().lean().stream();

As you can see the error happens before streamToMongoDB is even called:

TypeError: Cannot read property 'stream' of undefined
    at NodeCollection.findStream (/Users/vish/Projects/testnpm/node_modules/mongoose/node_modules/mquery/lib/collection/node.js:131:50)
    at model.Query.Query.stream (/Users/vish/Projects/testnpm/node_modules/mongoose/node_modules/mquery/lib/mquery.js:3017:27)
    at Object.<anonymous> (/Users/vish/Projects/testnpm/stream.js:54:38)

You should dig around mongoose and find if this is possible to have a stream, and then plug in to streamToMongoDB ?

Maybe you can try to pipe to process.stdout stream if you want to make sure your code works before piping into streamToMongoDB ?

[fabienjuif]

I put the invalid label, feel free to argue if you think I am mistaken ;)

[vishalgakhare]

You're correct - mongoose is not able initialize the stream. Will have check mongoose doc.
A small correction - you are referring to mongodb example in readme, I am using mongoose example.

[fabienjuif]

Ho! You are right!

Is it possible that you open a PR to update the documentation when you'll find a solution ? 🤔

[fabienjuif]

Hi @vishalgakhare did you find a solution?

[vishalgakhare]

Sorry @fabienjuif . I was on vacation, so responding late. The mongoose has replaced stream with cursor function. However, the subsequent pipe, on couldn't be used with the cursor. I also noticed that on event is fired faster than I could apply the transformation. Therefore, I ended up using the cursor inside async function.

[afirdousi]

@vishalgakhare can you share a gist of your solution here? of how did you used async/await in your case to run a time-consuming transformation.

[vishalgakhare]

• This gist uses the cursor. I don't recall the exact error but I think read was too faster to capture each record. Would try to run it later.
• This gist uses the async. It doesn't emit the next doc/record until the eachAsync promise is resolved.