New package for inclusion - vsivsi:file-collection

[edemaine]

Package/project name & description

file-collection enables arbitrary-size binary file support using MongoDB gridFS, so users can easily upload/download files in a Meteor system, keeping everything in the MongoDB.

Major features from the README:

• HTTP upload and download including support for Meteor authentication
• Client and Server integration of resumable.js for robust chunked uploading
• Also compatible with traditional HTTP POST or PUT file uploading
• HTTP range requests support random access for resumable downloads, media seeking, etc.
• Robust file locking allows safe replacement and removal of files even on a busy server
• External changes to the underlying file store automatically synchronize with the Meteor collection
• Designed for efficient handling of millions of small files as well as huge files 10GB and above

Links

• Atmosphere
• Github repository

Current status of the project

• Abandoned / maintenance mode
• I (@edemaine) have a fork that runs on the current version of Meteor (Atmosphere, Github), but it was a quick fix and doesn't address existing PRs etc.

Reasoning

Storing all files in MongoDB makes it particularly easy to backup the state of a Meteor project: just backup the database.

I use the package in my Coauthor system, so it's important to me. I'm willing to maintain it for now, at least trying to review PRs and I have some (slow) plans to improve some functionality. This has been discussed some on vsivsi/meteor-file-collection#172, with support from the original author, @vsivsi.

Progress

• Discussion finished
• Moved / Forked
• Team Created
• Meteor Org Created and set as maintainer
• communitypackages org added as a maintainer on Atmosphere

Related projects

• vsivsi:job-collection

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[juliomac]

Thanks for reopening it.

This Package has been installed almost 12.000 times. It is the only package that allows using gridFS for file storage (pictures and movies) in an easy and secure way after the demise of the complex fs:collection. The alternative https://github.com/VeliovGroup/Meteor-Files is not as an elegant solution for grids as this one is.

It is so popular that It has already been forked many times to replace the older coffescript dependency, like done here https://github.com/brucejo75/meteor-file-collection and here with an non-coffee version https://github.com/swiftcourt/meteor-file-collection .

The other problem is its incompatibility with SSL mongo servers, which is an impediment for using Mongo Atlas, for instance, as detailed here https://forums.meteor.com/t/issues-migrating-from-mlab-to-atlas/54206 that were supposedly solved here https://github.com/sakeena12/meteor-file-collection .

[edemaine]

Just glancing, @sakeena12's https://github.com/sakeena12/meteor-file-collection does look like a good starting point, incorporating @brucejo75's CoffeeScript upgrade and mongoOptions passthrough. Are either you interested in helping maintain this project?

[juliomac]

Hi @edemaine, I don't feel I have enough knowledge to take on this task, specially regarding CoffeeScript.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[brucejo75]

Hi @edmaine,

For basic Mongo file storage I don't recommend this package. vsivsi:file-collection is designed for very large files and able to handle interrupted streams, etc. This is way overkill for simple storage of files. I had used vsivsi:file-collection originally as I was beginning to understand Meteor and it was really the only working Mongo file package at the time.

I ended up switching to a more modern technique simply using fetch(), express & Mongo GridFSBucket. It turned out the code to integrate was vastly simplified compared to vsivsi:file-collection.

-Bruce

[juliomac]

Hi @edmaine,

For basic Mongo file storage I don't recommend this package. vsivsi:file-collection is designed for very large files and able to handle interrupted streams, etc. This is way overkill for simple storage of files. I had used vsivsi:file-collection originally as I was beginning to understand Meteor and it was really the only working Mongo file package at the time.

I ended up switching to a more modern technique simply using fetch(), express & Mongo GridFSBucket. It turned out the code to integrate was vastly simplified compared to vsivsi:file-collection.

-Bruce

How have you done it? Have you got any code example that I could reuse and also replace vsivsi:file-collection for good in my code? My goal is to continue using Mongo for saving small pictures in database.

[brucejo75]

HI @juliomac,

Here is a summary.

Client Side

1. In HTML you will want an <input type="file" id="yourID"...> tag.
2. Create a meteor event handler on input change e.g. for Blaze 'change #yourID': function fileInputChanged(event, template)...
3. In your event handler you will do the file upload. Read up on fetch. On the client you can use it to POST or PUT a file request against a REST interface on your server.
4. It would be good to be familiar with async functions.

Server Side

1. You can use express to process the incoming REST request.
2. Include the mongo package so that you can get access to MongoInternals.
3. In your express handler you can then access GridFSBucket via MongoInternals.
4. Read up on node streams. Once you have a GridFS interface you can then pipe the incoming file to a GridFSBucket upload stream.

Summary of main parts

• an input form that you can browse to a file to upload.
• When you choose the file it will fire your event handler.
• In the handler you send off a REST url request.
• In the server express handler you receive the REST url request.
• Then you engage Mongo's GridFSBucket to pipe the file stream directly into the Mongo DB fs.files collection.

[juliomac]

Great! I will try that!

[juliomac]

1. You can use express to process the incoming REST request.

Hi @brucejo75 , I am trying this in practice, but you are suggesting using Express framework inside Meteor framework just for the process of REST? Could we just use Meteor's own REST processing HTTP method https://docs.meteor.com/api/http.html or simple:rest package ?

[brucejo75]

@juliomac,

1. Express framework is for the server side to process the incoming packets. HTTP does not handle incoming packets. simple:rest looks like it has the capability to define REST routes, so you could use that. I recommend express, pretty standard, quite powerful for handling authentication, errors, etc..
2. HTTP could be used instead of fetch() on the client side. I use fetch() because it is a standard browser feature now and the code is pretty simple.

Background

Here is how I think of the Meteor Server:

1. It is Node, therefore you can use all Node packages with it.
2. Meteor provides some useful communications interfaces:

• publish/subscribe for downloading MongoDB data to your local cache of mongo data (aka minimongo)
• Decent RPC capability with some good error handling (aka methods)
• Shared API model on client and server, useful for getting started.

3. Meteor provides a very surface abstraction to MongoDB, useful for simple projects.

This enables some interesting things like accounts. But there are some caveats:

• publish/subscribe is quite slow over DDP. Most people recommend using methods they are more performant.
• Meteor packages are not being advised in favor of npm packages. So everywhere I can, I use npm packages instead. In fact, I should work on removing as many meteor packages as possible.
• packages like simple:rest have not been touched in over 5 years. Maybe better to use active packages?
• Mongo interface has not kept up with capabilities of mongodb. So having access to MongoInternals is great for using the full capabilities of the Node Mongo DB driver.

[edemaine]

Meteor's webapp package ought to suffice. That's what vsivsi:file-collection uses, and I've also used it for other file access in other projects.

I personally would not use Meteor and Express together. I'd rather have a project that's easy to install with minimal dependencies, and Meteor should support everything you need out of the box. (I could see switching wholesale to Express, but I wouldn't.) But maybe that's just me...

Anyway, this thread doesn't seem like the place to discuss, unless @brucejo75 is suggesting that vsivsi:file-collection should not be added to Meteor Community Packages?

For what it's worth, I still use vsivsi:file-collection everyday in a major project, and am still willing to help maintain it.

[brucejo75]

Thanks @edmaine,

• Yes, I am recommending to not maintain this package. Unless your usage is for multi-GB streaming files this package is harder to use, has a ton of overhead and not much benefit. Also, coffescript - yuck.

• Any of the packages recommended simple:rest or webapp should work fine.

Note: I wouldn't be too afraid to install NPM packages directly. You can see that webapp depends on a number of npm packages. Notably connect. It looks like express used to depend on connect but they removed that dependency. You can also see that connect has been in light maintenance mode for the last couple of years. I think express is the preferred solution these days.

Even though you are using "Meteor" packages they can pull in npm dependencies.

[edemaine]

@brucejo75 I'm not afraid to install NPM packages. But I don't want to run two Node.js server processes (Meteor and Express) for a single project. Unless there's some integration that lets Express apply its hooks from within Meteor (via webapp) or vice versa?

What is the extra overhead of vsivsi:file-collection? The Files subscription?

In my application, I do routinely upload and download 1-100MB files. And you are welcome to your opinion about CoffeeScript, but it's still my preferred language for all projects, so that's actually a positive for me.

[brucejo75]

Hi @edmaine,

Oh, you don't need 2 servers. Simply npm install express into your Meteor server. Then start using the api in your Meteor server code. I think of Meteor as nothing more than some API sugar on top of node. Sure it uses websockets for DDP & methods. But you can still define http endpoints via your favorite package.

If you look closely at webapp it does exactly that with the connect package.

Summary: At its core a Meteor server is simply a node server. And this is a great thing about Meteor, you do not have to use it at all if you choose not to. Very flexible.

It all works together in one server.

Benefits without vsivsi:file-collection

When I removed vsivsi:file-collection I got 3 benefits:

1. Integrating my file upload as I described above was more straightforward and required less code for me than integrating with vsivsi:file-collection.
2. I no longer have the vsivsi:file-collection code in my app.
3. No more coffeescript compiler incompatibilities.

3 big wins.

Sorry for the coffeescript crack earlier. I know some people love it but introduces maintenance & sharing headaches because coffeescript is not widely adopted and it is another toolset that can introduce incompatibilities.

[juliomac]

1. Integrating my file upload as I described above was more straightforward and required less code for me than integrating with vsivsi:file-collection.

Interesting... I would think adding an entire framework like express would add a lot more code than using a single package.

[brucejo75]

@juliomac,

Good thought. But the problem is that vsivsi:file-collection includes express in addition to other packages and its own code.

And look at all the other packages that vsivsi:file-collection depends on and therefore pulls in.

And interestingly, vsivsi:file-collection also includes webapp which pulls in connect which is the precursor to express (express was originally built on connect).

With vsivsi:file-collection you essentially get 2 copies of express plus a bunch of other packages.

Why not just use 1 copy of express?

[juliomac]

@brucejo75 , thanks for the great analysis on this package. I would never know that it was using express behind the scenes and that it was so heavy.

I think you have put out the last argument against the package. I would definitely follow your advice.

If I succeed, I will post here my experiences with the suggested path.

Thanks!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.