Motivation for RedAmber

[baarkerlounger]

What's the motivation for creating/using RedAmber over Rover-df?

[heronshoes]

Thank you for visiting RedAmber discussions!

• The trigger was to try Red Arrow to speed up Rover's csv loading, and RedAmber's speed based on Red Arrow is advantageous, especially when the data is large.
• RedAmber allows nil for all types, while Rover cannot have nil for Numeric data types (due to Numo/NArray limitations, Float can have NaN). I think this is not sufficient for a general-purpose data frame library.
• RedAmber uses blocks for columns and rows manipulation and can be written Ruby-like manner.

Rover works well with machine learning libraries, the code is simple and beautiful, and I have a lot of respect for it because it inspired me (with poor OSS experience) to create my own library.

[heronshoes]

For additional information, this is a simple benchmark for Rover-df and RedAmber.

# csv_load_penguins.yml
prelude: |
  require 'rover'
  require 'red_amber'

  penguins_csv = 'tmp/penguins.csv'

  unless File.exist?(penguins_csv)
    require 'datasets-arrow'
    ds = Datasets::Penguins.new
    RedAmber::DataFrame.new(ds).save(penguins_csv)
  end

benchmark:
  'penguins by Rover': Rover.read_csv(penguins_csv)
  'penguins by RedAmber': RedAmber::DataFrame.load(penguins_csv)

And run above with;

benchmark-driver csv_load_penguins.yml

Results;

Warming up --------------------------------------
   penguins by Rover     49.494 i/s -      54.000 times in 1.091036s (20.20ms/i)
penguins by RedAmber    616.594 i/s -     620.000 times in 1.005524s (1.62ms/i)
Calculating -------------------------------------
   penguins by Rover     38.195 i/s -     148.000 times in 3.874902s (26.18ms/i)
penguins by RedAmber    574.486 i/s -      1.849k times in 3.218528s (1.74ms/i)

Comparison:
penguins by RedAmber:       574.5 i/s 
   penguins by Rover:        38.2 i/s - 15.04x  slower

[baarkerlounger]

Would be interesting to also benchmark against https://github.com/ankane/polars-ruby

[heronshoes]

Yes. Comparison above is old and unfair... Arrow (Arrow2) based Polars will be super fast. Also Polars has lazy() to delay evaluation. RedAmber eagerly materialize all result. That results slow operation for a series of operation (queries).