Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement fast uploader #8

Open
milliondreams opened this issue Apr 18, 2014 · 0 comments
Open

Implement fast uploader #8

milliondreams opened this issue Apr 18, 2014 · 0 comments
Assignees

Comments

@milliondreams
Copy link
Member

Currently snackfs relies on HDFS 'copyFromLocal' or equivalent command to send data to snackfs. This command doesn't understand snackfs internals and treats it as just another (HDFS compatible) filesystem, writing chunks of bytes to output stream.

Since snackfs is a fat client and all the logic to create inodes and blocks/subblocks from the file resides on the client, we can use it improve upload performance by splitting the files into chunks equal to the "subblock" size and writing multiple subblocks to different C* nodes in parallel.

Once all the subblocks are written, we can update the inode in one go.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants