GitHub - abuddenb/Meteor-CollectionFS: Meteor webbased filesystem handling up and downloads.

#CollectionFS (pre1)

NOTE: This branch is under active development right now (2014-7-14). It has bugs and the API may continue to change. Please help test it and fix bugs, but don't use in production yet.

If you have Documentation feedback/requests please post on issue 206

CollectionFS is a smart package for Meteor that provides a complete file management solution including uploading, downloading, storage, synchronization, manipulation, and copying. It supports several storage adapters for saving to the local filesystem, GridFS, or S3, and additional storage adapters can be created.

Installation

Install Meteorite, if you have not yet done so, and then:

$ cd <app dir>
$ mrt add collectionFS
$ mrt add <storage adapter package>
$ mrt add <CFS add-on packages>

You must add collectionFS, which is the main package, and at least one storage adapter package. See the Storage Adapters section for a list of the available storage adapter packages. Depending on what you need to do, you may need to add additional add-on packages. These are explained in the documentation sections to which they apply.

To pull down the most recent updates to every package, just run mrt update again at any time. If you're having trouble, you can alternatively try cloning this repo.

Make sure your smart.json says "collectionFS": {} pointing to the latest version

Using the Old API

Adding the collectionFS package currently gives you the 0.4.x release, which is the new API documented here and in development on the devel branch. This API is fairly stable and allows you to do much more than the old API, but it currently lacks complete tests and may change a bit prior to the 1.0 release. If you prefer to use the old API, edit your app's smart.json so that the collectionFS line looks like this:

 "packages": {
   "collectionFS": "0.3.7"
 }

If you have tried the new API, you can remove any other CFS packages that might be in your smart.json file. Then run the following commands:

$ cd <app dir>
$ mrt update
$ meteor add collectionFS

You must call meteor add for all packages that you manually added to smart.json.

Introduction

The CollectionFS package makes available two important global variables: FS.File and FS.Collection.

A FS.File wraps a file and its data on the client or server. It is similar to the browser File object (and can be created from a File object), but it has additional properties and methods.
A FS.Collection provides a collection in which information about files can be stored.

A document from a FS.Collection is represented as a FS.File.

CollectionFS also provides an HTTP upload package that has the necessary mechanisms to upload files, track upload progress reactively, and pause and resume uploads. This can be swapped for a DDP upload package, but we do not currently recommend using DDP for uploads due to known issues with the current DDP spec and large files.

Getting Started

The first step in using this package is to define a FS.Collection.

common.js:

var Images = new FS.Collection("images", {
  stores: [new FS.Store.FileSystem("images", {path: "~/uploads"})]
});

In this example, we've defined a FS.Collection named "images", which will be a new collection in your MongoDB database with the name "cfs.images.filerecord". We've also told it to store the files in ~/uploads on the local filesystem. If you don't specify a path, a cfs/files folder in your app container (bundle directory) will be used.

Your FS.Collection and FS.Store variables do not necessarily have to be global on the client or the server, but be sure to give them the same name (the first argument in each constructor) on both the client and the server.

If you're using a storage adapter that requires sensitive information such as access keys, we recommend supplying that information using environment variables. If you instead decide to pass options to the storage adapter constructor, then be sure that you do that only in the server code (and not simply within a Meteor.isServer block).

Now we can upload a file from the client. Here is an example of doing so from the change event handler of an HTML file input:

Template.myForm.events({
  'change .myFileInput': function(event, template) {
    var files = event.target.files;
    for (var i = 0, ln = files.length; i < ln; i++) {
      Images.insert(files[i], function (err, fileObj) {
        //Inserted new doc with ID fileObj._id, and kicked off the data upload using HTTP
      });
    }
  }
});

You can optionally make this code a bit cleaner by using a provided utility method, FS.Utility.eachFile:

Template.myForm.events({
  'change .myFileInput': function(event, template) {
    FS.Utility.eachFile(event, function(file) {
      Images.insert(file, function (err, fileObj) {
        //Inserted new doc with ID fileObj._id, and kicked off the data upload using HTTP
      });
    });
  }
});

Notice that the only thing we're doing is passing the browser-provided File object to Images.insert(). This will create a FS.File from the File, link it with the Images FS.Collection, and then immediately begin uploading the data to the server with reactive progress updates.

The insert method can directly accept a variety of different file representations as its first argument:

File object (client only)
Blob object (client only)
Uint8Array
ArrayBuffer
Buffer (server only)
A full URL that begins with "http:" or "https:"
A local filepath (server only)
A data URI string

Where possible, streams are used, so in general you should avoid using any of the buffer/binary options unless you have no choice, perhaps because you are generating small files in memory.

The most common usage is to pass a File object on the client or a URL on either the client or the server. Note that when you pass a URL on the client, the actual data download from that URL happens on the server, so you don't need to worry about CORS. In fact, we recommend doing all inserts on the client (managing security through allow/deny), unless you are generating the data on the server.

Inserting a File on the Server

In certain circumstances, however, it might be necessary to perform inserts on the server. The following example demonstrates the steps required to handle a Buffer:

var request = Meteor.require('request');

request.get({url: url, encoding: null}, Meteor.bindEnvironment(function(e, r, buffer){
  var newFile = new FS.File();
  newFile.attachData(buffer, {type: 'image/png'}, function(error){
      if(error) throw error;
      newFile.name('myGraphic.png');
      
      Images.insert(newFile);
  });
})).auth(null, null, true, accessToken);

When calling attachData() with a Buffer, you must provide the MIME type of the file through the second argument as shown above. Omitting this argument yields an exception:

Error: DataMan constructor requires a type argument when passed a Buffer

Further examples can be found here.

Caveats:

Downloading large files into a Buffer could potentially lead to memory issues.
Use of external HTTP library to make authenticated requests will be unnecessary once issue 350 is resolved.

After the Upload

After the server receives the FS.File and all the corresponding binary file data, it saves copies of the file in the stores that you specified.

If any storage adapters fail to save any of the copies in the designated store, the server will periodically retry saving them. After a configurable number of failed attempts at saving, the server will give up.

To configure the maximum number of save attempts, use the maxTries option when creating your store. The default is 5.

Storage Adapters

There are currently three available storage adapters, which are in separate packages. Refer to the package documentation for usage instructions.

cfs-gridfs: Allows you to save data to mongodb GridFS.
cfs-filesystem: Allows you to save to the server filesystem.
cfs-s3: Allows you to save to an Amazon S3 bucket.

Storage adapters also handle retrieving the file data and removing the file data when you delete the file.

File Manipulation

You may want to manipulate files before saving them. For example, if a user uploads a large image, you may want to reduce its resolution, crop it, compress it, etc. before allowing the storage adapter to save it. You may also want to convert to another content type or change the filename or encrypt the file. You can do all of this by defining stream transformations on a store.

transformWrite/transformRead

The most common type of transformation is a "write" transformation, that is, a function that changes the data as it is initially stored. You can define this function using the transformWrite option on any store constructor. If the transformation requires a companion transformation when the data is later read out of the store (such as encrypt/decrypt), you can define a transformRead function as well.

For illustration purposes, here is an example of a transformWrite function that doesn't do anything:

transformWrite: function(fileObj, readStream, writeStream) {
  readStream.pipe(writeStream);
}

The important thing is that you must pipe the readStream to the writeStream before returning from the function. Generally you will manipulate the stream in some way before piping it.

beforeWrite

Sometimes you also need to change a file's metadata before it is saved to a particular store. For example, you might have a transformWrite function that changes the file type, so you need a beforeWrite function that changes the extension and content type to match.

The simplest type of beforeWrite function will return an object with extension, name, or type properties. For example:

beforeWrite: function (fileObj) {
  return {
    extension: 'jpg',
    type: 'image/jpg'
  };
}

This would change the extension and type for that particular store.

Since beforeWrite is passed the fileObj, you can optionally alter that directly. For example, the following would be the same as the previous example assuming the store name is "jpegs":

beforeWrite: function (fileObj) {
  fileObj.extension('jpg', {store: "jpegs", save: false});
  fileObj.type('image/jpg', {store: "jpegs", save: false});
}

(It's best to provide the save: false option to any of the setters you call in beforeWrite.)

Image Manipulation

A common use for transformWrite is to manipulate images before saving them. To get this set up:

Install GraphicsMagick or ImageMagick on your development machine and on any server that will host your app. (The free Meteor deployment servers do not have either of these, so you can't deploy to there.) These are normal operating system applications, so you have to install them using the correct method for your OS. For example, on Mac OSX you can use brew install graphicsmagick assuming you have Homebrew installed.
Add the cfs-graphicsmagick Meteor package to your app: mrt add cfs-graphicsmagick

The following are some examples.

Basic Example

Images = new FS.Collection("images", {
    stores: [
      new FS.Store.FileSystem("images"),
      new FS.Store.FileSystem("thumbs", {
        transformWrite: function(fileObj, readStream, writeStream) {
          // Transform the image into a 10x10px thumbnail
          gm(readStream, fileObj.name()).resize('10', '10').stream().pipe(writeStream);
        }
      })
    ],
    filter: {
      allow: {
        contentTypes: ['image/*'] //allow only images in this FS.Collection
      }
    }
});

Note that this example requires the cfs-filesystem package.

Converting to a Different Image Format

To convert every file to a specific image format, you can pass a GraphicsMagick format string to the stream method, but you will also need to alter the FS.File instance as necessary in a beforeWrite function.

Images = new FS.Collection("images", {
    stores: [
      new FS.Store.FileSystem("images"),
      new FS.Store.FileSystem("thumbs", {
        beforeWrite: function(fileObj) {
          // We return an object, which will change the
          // filename extension and type for this store only.
          return {
            extension: 'png',
            type: 'image/png'
          };
        },
        transformWrite: function(fileObj, readStream, writeStream) {
          // Transform the image into a 10x10px PNG thumbnail
          gm(readStream).resize(60).stream('PNG').pipe(writeStream);
          // The new file size will be automatically detected and set for this store
        }
      })
    ],
    filter: {
      allow: {
        contentTypes: ['image/*'] //allow only images in this FS.Collection
      }
    }
});

Note that this example requires the cfs-filesystem package.

Converting a File Already Stored

You may want to adjust a bunch of images that you've already stored. This can be done easily by streaming out of the store and then back into it. The following example is for illustration purposes, but you should not use it on production data unless you have a throttled queue in a separate process or only a very small number of images.

Images.find().forEach(function (fileObj) {
  var readStream = fileObj.createReadStream('images');
  var writeStream = fileObj.createWriteStream('images');
  gm(readStream).swirl(180).stream().pipe(writeStream);
});

Note that you could also pipe the readStream from one store to the writeStream from another store to move files between stores, for example if you decide to use a different storage adapter and need to quickly and easily migrate the data. (We have not tested this, but it should be possible.)

An FS.File Instance

An FS.File instance is an object with properties similar to this:

{
  _id: '',
  collectionName: '', // this property not stored in DB
  collection: collectionInstance, // this property not stored in DB
  createdByTransform: true, // this property not stored in DB
  data: data, // this property not stored in DB
  original: {
    name: '',
    size: 0,
    type: '',
    updatedAt: date 
  },
  copies: {
    storeName: {
      key: '',
      name: '',
      size: 0,
      type: '',
      createdAt: date,
      updatedAt: date 
    }
  },
  uploadedAt: date,
  anyUserDefinedProp: anything
}

But name, size, type, and updatedAt should be retrieved and set with the methods rather than directly accessing the props:

// get original
fileObj.name();
fileObj.extension();
fileObj.size();
fileObj.formattedSize(); // must add the "numeral" package to your project to use this method
fileObj.type();
fileObj.updatedAt();

// get for the version in a store
fileObj.name({store: 'thumbs'});
fileObj.extension({store: 'thumbs'});
fileObj.size({store: 'thumbs'});
fileObj.formattedSize({store: 'thumbs'}); // must add the "numeral" package to your project to use this method
fileObj.type({store: 'thumbs'});
fileObj.updatedAt({store: 'thumbs'});

// set original
fileObj.name('pic.png');
fileObj.extension('png');
fileObj.size(100);
fileObj.type('image/png');
fileObj.updatedAt(new Date);

// set for the version in a store
fileObj.name('pic.png', {store: 'thumbs'});
fileObj.extension('png', {store: 'thumbs'});
fileObj.size(100, {store: 'thumbs'});
fileObj.type('image/png', {store: 'thumbs'});
fileObj.updatedAt(new Date, {store: 'thumbs'});

These methods can all be used as UI helpers, too:

{{#each myFiles}}
  <p>Original name: {{this.name}}</p>
  <p>Original extension: {{this.extension}}</p>
  <p>Original type: {{this.type}}</p>
  <p>Original size: {{this.size}}</p>
  <p>Thumbnail name: {{this.name store="thumbs"}}</p>
  <p>Thumbnail extension: {{this.extension store="thumbs"}}</p>
  <p>Thumbnail type: {{this.type store="thumbs"}}</p>
  <p>Thumbnail size: {{this.size store="thumbs"}}</p>
{{/each}}

Also, rather than setting the data property directly, you should use the attachData method.

Check out the full public API for FS.File.

Storing FS.File references in your objects

NOTE: At the moment storing FS.File - References in MongoDB on the server side doesn't work. See eg. (https://github.com/CollectionFS/Meteor-cfs-ejson-file/issues/1) (Meteor-Community-Packages#356) (meteor/meteor#1890).

_Instead store the id's of your file objects and then fetch the FS.File-Objects from your CollectionFS - Collection.

Often your files are part of another entity. You can store a reference to the file directly in the entity. You need to add cfs-ejson-file to your packages with mrt add cfs-ejson-file. Then you can do for example:

// Add file reference of the event photo to the event
var file = $('#file').get(0).files[0];
var fileObj = eventPhotos.insert(file);
events.insert({
  name: 'My Event',
  photo: fileObj
});

// Later: Retrieve the event with the photo
var event = events.findOne({name: 'My Event'});
// This loads the data of the photo into event.photo
// You can include it in your collection transform function.
event.photo.getFileRecord();

Demo app

You need to ensure that the client is subscribed to the related photo document, too. There are packages on atmosphere, such as publish-with-relations and smart-publish, that attempt to make this easy.

Filtering

You may specify filters to allow (or deny) only certain content types, file extensions, or file sizes in a FS.Collection. Use the filter option.

Images = new FS.Collection("images", {
  filter: {
    maxSize: 1048576, //in bytes
    allow: {
      contentTypes: ['image/*'],
      extensions: ['png']
    },
    deny: {
      contentTypes: ['image/*'],
      extensions: ['png']
    },
    onInvalid: function (message) {
      if (Meteor.isClient) {
        alert(message);
      } else {
        console.log(message);
      }
    }
  }
});

Alternatively, you can pass your filters object to myFSCollection.addFilters().

To be secure, this must be added on the server; however, you should use the filter option on the client, too, to help catch many of the disallowed uploads there and allow you to display a helpful message with your onInvalid function.

You can mix and match filtering based on extension or content types. The contentTypes array also supports "image/*" and "audio/*" and "video/*" like the "accepts" attribute on the HTML5 file input element.

If a file extension or content type matches any of those listed in allow, it is allowed. If not, it is denied. If it matches both allow and deny, it is denied. Typically, you would use only allow or only deny, but not both. If you do not pass the filter option, all files are allowed, as long as they pass the tests in your FS.Collection.allow() and FS.Collection.deny() functions.

The extension checks are used only when there is a filename. It's possible to upload a file with no name. Thus, you should generally use extension checks only in addition to content type checks, and not instead of content type checks.

The file extensions must be specified without a leading period. Extension matching is case-insensitive.

Security

File uploads and downloads can be secured using standard Meteor allow and deny methods. To best understand how CollectionFS security works, you must first understand that there are two ways in which a user could interact with a file:

She could view or edit information about the file or any custom metadata you've attached to the file record.
She could view or edit the actual file data.

You may find it necessary to secure file records with different criteria from that of file data. This is easy to do.

Here's an overview of the various ways of securing various aspects of files:

To determine who can see file metadata, such as filename, size, content type, and any custom metadata that you set, use normal Meteor publish/subscribe to publish and subscribe to an FS.Collection cursor. This does not allow the user to download the file data.
To determine who can download the actual file, use "download" allow/deny functions. This is a custom type of allow/deny function provided by CollectionFS. The first argument is the userId and the second argument is the FS.File being requested for download.
To determine who can set file metadata, insert files, and upload file data, use "insert" allow/deny functions.
To determine who can update file metadata, use "update" allow/deny functions.
To determine who can remove files, which removes all file data and file metadata, use "remove" allow/deny functions.

The download allow/deny functions can be thought of essentially as allowing or denying "read" access to the file. For a normal Meteor collection, "read" access is defined through pub/sub, but we don't want to send large amounts of binary file data to each client just because they subscribe to the file record. Thus with CFS, pub/sub will get you the file's metadata on the client whereas an HTTP request to the GET URL is required to view or download the file itself. The download allow/deny determines whether this HTTP request will respond with "Access Denied" or not.

Securing Based on User Information

To secure a file based on a user "owner" or "role" or some other piece of custom metadata, you must set this information on the file when originally inserting it. You can then check it in your allow/deny functions.

var fsFile = new FS.File(event.target.files[0]);
fsFile.owner = Meteor.userId();
fsCollection.insert(fsFile, function (err) {
  if (err) throw err;
});

Note that you will want to verify this owner metadata in a deny function since the client could put any user ID there.

Using `insert` Properly

When you need to insert a file that's located on a client, always call myFSCollection.insert on the client. While you could define your own method, pass it the fsFile, and call myFSCollection.insert on the server, the difficulty is with getting the data from the client to the server. When you pass the fsFile to your method, only the file info is sent and not the data. By contrast, when you do the insert directly on the client, it automatically chunks the file's data after insert, and then queues it to be sent chunk by chunk to the server. And then there is the matter of recombining all those chunks on the server and stuffing the data back into the fsFile. So doing client-side inserts actually saves you all of this complex work, and that's why we recommend it.

Calling insert on the server should be done only when you have the file somewhere on the server filesystem already or you're generating the data on the server.

UI Helpers

There are two types of UI helpers available. First, some of the FS.File instance methods will work when called in templates, too. These are available to you automatically and are documented here. Second, some additional useful helpers are provided in the optional cfs-ui package. These make it easy to render a delete button or an upload progress bar and more. Refer to the cfs-ui readme.