Skip to content

Ruby's libarchive and open-uri gems can use a ton of memory if you're not careful. In this repo I will explore various use cases.

License

Notifications You must be signed in to change notification settings

billzajac/ruby-libarchive-and-open-uri-memory-tests

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

4 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Ruby libarchive and open-uri memory tests

Ruby’s libarchive and open-uri gems can use a ton of memory if you’re not careful.
This repo is a just number of test scripts I used to explore various use cases.

Use Case

When creating scripts for something like Chef, you might need to download a large tarball. If you’re not careful, the process of downloading and potentially extracting that tarball could end up trying to use more memory than your system has.

Observations

The main observation seems to be that if you open the file handles for writing inside of the read loop, that you will end up storing the entire file in memory at some point. This seems obvious now that I’ve seen it, but my initial expectation was exactly the opposite.

It’s also worth noting that libarchive allows you to chunk your data read, which it doesn’t do by default. If you don’t send a parameter to Archive#read_data() it will store the entire file in memory at some point.

The scripts

  • wget_best.rb
  • profile_tools.rb is a library
    • The very simple ProfileTools class has the method show_mem() which finds the RSS using ps $$

Links

To do

Read more at: http://code.google.com/p/libarchive/wiki/Examples

  • In glancing at the Examples, it looks like libarchive was created to allow for chaining.

About

Ruby's libarchive and open-uri gems can use a ton of memory if you're not careful. In this repo I will explore various use cases.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages