Skip to content

Commit

Permalink
Use thread pooling to prevent large number of threads
Browse files Browse the repository at this point in the history
  • Loading branch information
tonyvince committed Jul 4, 2023
1 parent b59efda commit 147caa1
Show file tree
Hide file tree
Showing 3 changed files with 10 additions and 3 deletions.
1 change: 1 addition & 0 deletions Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ source 'https://rubygems.org'

ruby "3.2.2"

gem 'concurrent-ruby'
gem 'pry'

group :test do
Expand Down
2 changes: 2 additions & 0 deletions Gemfile.lock
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ GEM
addressable (2.8.4)
public_suffix (>= 2.0.2, < 6.0)
coderay (1.1.3)
concurrent-ruby (1.2.2)
crack (0.4.5)
rexml
diff-lcs (1.5.0)
Expand Down Expand Up @@ -36,6 +37,7 @@ PLATFORMS
arm64-darwin-20

DEPENDENCIES
concurrent-ruby
pry
rspec
webmock
Expand Down
10 changes: 7 additions & 3 deletions lib/image_downloader.rb
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
require 'open-uri'
require 'net/http'
require 'concurrent'

class ImageDownloader
def self.download_images(urls, target_dir = "./images")
Dir.mkdir(target_dir) unless File.exist?(target_dir)

threads = urls.map do |url|
Thread.new do
pool = Concurrent::FixedThreadPool.new(10)

urls.each do |url|
pool.post do
begin
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
Expand All @@ -28,6 +31,7 @@ def self.download_images(urls, target_dir = "./images")
end
end

threads.each(&:join) # Wait for all threads to finish
pool.shutdown
pool.wait_for_termination
end
end

0 comments on commit 147caa1

Please sign in to comment.