Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added script that I used to move files from files folder up to S3 #6

Merged
merged 1 commit into from
Nov 17, 2011
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.rdoc
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ This Redmine[http://www.redmine.org] plugin makes file attachments be stored on
3. cp vendor/plugins/redmine_s3/config/s3.yml.example config/s3.yml
4. Edit config/s3.yml with your favourite editor
5. Restart mongrel/upload to production/whatever
6. _Optional:_ Run 'rake redmine_s3:files_to_s3' to upload files in your files folder to s3

== Options Overview
* The bucket specified in s3.yml will be created automatically when the plugin is loaded (this is generally when the server starts).
Expand Down
57 changes: 57 additions & 0 deletions lib/tasks/files_to_s3.rake
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
namespace :redmine_s3 do
task :files_to_s3 => :environment do
require 'thread'

def s3_file_path(file_path)
File.basename(file_path).split('_').first + '/' + File.basename(file_path)
end

# updates a single file on s3
def update_file_on_s3(file, objects)
file_path = s3_file_path(file)
conn = RedmineS3::Connection.conn
object = objects[file_path]

# get the file modified time, which will stay nil if the file doesn't exist yet
# we could check if the file exists, but this saves a head request
s3_mtime = object.last_modified rescue nil

# put it on s3 if the file has been updated or it doesn't exist on s3 yet
if s3_mtime.nil? || s3_mtime < File.mtime(file)
fileObj = File.open(file, 'r')
RedmineS3::Connection.put(file_path, fileObj.read)
fileObj.close

puts "Put file " + File.basename(file)
else
puts File.basename(file) + ' is up-to-date on S3'
end
end

# enqueue all of the files to be "worked" on
fileQ = Queue.new
Dir.glob(RAILS_ROOT + '/files/*').each do |file|
fileQ << file
end

# init the connection, and grab the ObjectCollection object for the bucket
conn = RedmineS3::Connection.establish_connection
objects = conn.buckets[RedmineS3::Connection.bucket].objects

# create some threads to start syncing all of the queued files with s3
threads = Array.new
8.times do
threads << Thread.new do
while !fileQ.empty?
update_file_on_s3(fileQ.pop, objects)
end
end
end

# wait on all of the threads to finish
threads.each do |thread|
thread.join
end

end
end