Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update arrow version #8

Merged
merged 2 commits into from
Jun 10, 2024
Merged

Update arrow version #8

merged 2 commits into from
Jun 10, 2024

Conversation

Swoorup
Copy link
Owner

@Swoorup Swoorup commented Jun 9, 2024

Much better actually. Reduce code.

Copy link

gooroo-dev bot commented Jun 9, 2024

There are no known files to review.

Currently, I support these languages:

  • Ada: .adb, .ads
  • Assembly: .asm, .s
  • C: .c, .h
  • C++: .cpp, .hpp, .hxx, .cc, .cxx
  • C#: .cs
  • Clojure: .clj, .cljs, .cljc
  • COBOL: .cob, .cbl
  • Dart: .dart
  • Elixir: .ex, .exs
  • Erlang: .erl, .hrl
  • F#: .fs, .fsi, .fsx
  • Fortran: .f, .f90, .f95, .f03, .f08
  • Go: .go
  • Groovy: .groovy, .gvy, .gy, .gsh
  • Haskell: .hs, .lhs
  • Java: .java, .jav
  • JavaScript: .js, .jsx, .ts, .tsx, .coffee, .flow, .gs
  • Julia: .jl
  • Kotlin: .kt, .kts
  • Lua: .lua
  • MATLAB: .m, .mat
  • Markup Languages: .md, .markdown, .yaml, .yml, .json
  • Objective-C: .m, .mm
  • Pascal: .pas, .pp
  • PHP: .php, .phtml, .php3, .php4, .php5, .phps, .phpt
  • Perl: .pl, .pm
  • Python: .py
  • R: .r, .rdata, .rds
  • Ruby: .rb, .erb, .haml, .slim
  • Rust: .rs
  • Scala: .scala, .sc
  • Shell Scripting: .sh, .bash, .ps1, .bat, .cmd
  • SQL: .sql
  • Swift: .swift
  • TypeScript: .ts, .tsx
  • VHDL: .vhdl, .vhd
  • Verilog: .v, .vh
  • HTML: .html, .htm
  • CSS: .css, .sass, .scss
  • XML: .xml, .xsd, .xsl, .xslt

@aldanor
Copy link
Contributor

aldanor commented Jun 10, 2024

@Swoorup looks like at least this has to be addressed first: apache/arrow-rs#5623

@Swoorup
Copy link
Owner Author

Swoorup commented Jun 10, 2024

@Swoorup looks like at least this has to be addressed first: apache/arrow-rs#5623

Thanks for pointing that out.

Copy link

gooroo-dev bot commented Jun 10, 2024

🔄 New Commits Added

Author Commit Name Date
Swoorup Update UnionArray::try_new usage 2024-06-10 16:35:14 UTC

Reviewing these new changes. To re-review the whole PR, mention me and say review.

🐞Mistake 🤪Typo 🚨Security 🚀Performance 💪Best Practices 📖Readability ❓Others
0 0 0 0 1 1 0

Summary of Changes

  • ✅ Corrected the use of Buffer to ScalarBuffer to match the updated Arrow API.
  • ➖ Removed unnecessary field_type_ids field and related code.
  • 📖 Improved readability by simplifying the array construction logic.

Identified Issues

ID Type Details Severity Confidence
1 💪Best Practices The panic! calls in finish and finish_cloned methods could be replaced with Result 🟠Medium 🟠Medium
2 📖Readability The let children block is repeated in both finish and finish_cloned methods 🟡Low 🟢High

Code Snippets for Fixes

Fix for Issue 1

File: arrow_convert_derive/src/derive_enum.rs
Lines: 378, 404

fn finish(&mut self) -> arrow::array::ArrayRef {
    let union_fields = match <#original_name as arrow_convert::field::ArrowField>::data_type().clone() {
        arrow::datatypes::DataType::Union(fields, _) => fields,
        _ => return Err(arrow::error::ArrowError::ComputeError("datatype is not a union".to_string())),
    };

    // rest of the code remains unchanged
}

fn finish_cloned(&self) -> arrow::array::ArrayRef {
    let union_fields = match <#original_name as arrow_convert::field::ArrowField>::data_type().clone() {
        arrow::datatypes::DataType::Union(fields, _) => fields,
        _ => return Err(arrow::error::ArrowError::ComputeError("datatype is not a union".to_string())),
    };

    // rest of the code remains unchanged
}

Fix for Issue 2

File: arrow_convert_derive/src/derive_enum.rs
Lines: 378, 404

let children = vec![#(
    <#mutable_variant_array_types as arrow::array::ArrayBuilder>::finish(&mut self.#variant_names),
)*];
// Use `children` in both methods

General Review

  • The changes are well-aligned with the updated Arrow API.
  • The removal of field_type_ids simplifies the code.
  • Consider replacing panic! with Result for better error handling.
  • The repeated code block for children can be refactored for better readability.

Yours, Gooroo.dev
Your thoughts matter! Please react or reply.

@codecov-commenter
Copy link

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.11%. Comparing base (4ba848f) to head (aa7688f).
Report is 2 commits behind head on main.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main       #8      +/-   ##
==========================================
- Coverage   93.19%   93.11%   -0.09%     
==========================================
  Files           9        9              
  Lines        1764     1743      -21     
==========================================
- Hits         1644     1623      -21     
  Misses        120      120              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Swoorup Swoorup merged commit 4ccca87 into main Jun 10, 2024
5 checks passed
Copy link

gooroo-dev bot commented Jun 10, 2024

🔄 New Commits Added

Author Commit Name Date
Swoorup Update UnionArray::try_new usage 2024-06-10 16:35:14 UTC

Reviewing these new changes. To re-review the whole PR, mention me and say review.

🐞Mistake 🤪Typo 🚨Security 🚀Performance 💪Best Practices 📖Readability ❓Others
0 0 0 0 1 1 0

Summary of Incoming Changes

  • ✅ Corrects the usage of arrow::buffer::Buffer to arrow::buffer::ScalarBuffer, aligning with the updated Arrow library.
  • ➖ Removes the field_type_ids field and its associated logic, simplifying the code.
  • 📖 Improves readability by removing unnecessary variables and simplifying expressions.

Identified Issues

ID Type Details Severity Confidence
1 💪Best Practices Consider using collect() directly to create ScalarBuffer without intermediate collection. 🟡Low 🟠Medium
2 📖Readability Comments could be added to explain the removal of field_type_ids for better understanding. 🟡Low 🟠Medium

Code Snippets for Fixes

Issue 1: Best Practices

File: arrow_convert_derive/src/derive_enum.rs
Lines: 8-9

quote! { Some(self.offsets.iter().cloned().collect::<arrow::buffer::ScalarBuffer<i32>>()) },

Proposed Change:

quote! { Some(arrow::buffer::ScalarBuffer::from_slice_ref(&self.offsets)) },

Explanation: Directly using from_slice_ref avoids unnecessary intermediate collection and improves code efficiency.

Issue 2: Readability

File: arrow_convert_derive/src/derive_enum.rs
Lines: 258-260, 275-277, 378-380
Proposed Change:

// Removed field_type_ids as it is no longer necessary with the updated Arrow library.

Explanation: Adding comments to explain the removal of field_type_ids helps future developers understand the rationale behind the changes.

General Review

The code quality and style are generally good. The most important issue is the potential overuse of intermediate collections, which can be optimized for better performance. Additionally, adding comments to explain significant changes would improve the readability and maintainability of the code.

Yours, Gooroo.dev
Got thoughts? Don't hesitate to reply or add a reaction.

Copy link

gooroo-dev bot commented Jun 10, 2024

🔄 New Commits Added

Author Commit Name Date
Swoorup Update UnionArray::try_new usage 2024-06-10 16:35:14 UTC

Reviewing these new changes. To re-review the whole PR, mention me and say review.

🐞Mistake 🤪Typo 🚨Security 🚀Performance 💪Best Practices 📖Readability ❓Others
0 0 0 0 1 1 0

Summary of Changes

  • ✅ Corrected the usage of arrow::buffer::Buffer to arrow::buffer::ScalarBuffer for handling offsets and type IDs.
  • ➖ Removed the field_type_ids field and its related initializations and usages.
  • 📖 Improved the code readability by simplifying the try_new calls for UnionArray.

Identified Issues

ID Type Details Severity Confidence
1 💪Best Practices The unwrap calls in UnionArray::try_new should be replaced with proper error handling. 🟠Medium 🟠Medium
2 📖Readability The removal of field_type_ids and num_variants could be explained with comments for future readers. 🟡Low 🟡Low

Code Snippets for Fixes

Issue 1: Proper Error Handling

Lines: 377-380, 403-406

// Current code
.unwrap()

// Proposed change
.expect("Failed to create UnionArray")

Issue 2: Add Comments for Removed Fields

Lines: 243-245, 276-278

// Current code
// Removed code related to field_type_ids and num_variants

// Proposed change
// Removed field_type_ids and num_variants as they are no longer needed with the updated arrow version.

General Review

The overall code quality is good, and the changes are clear and concise. The switch to ScalarBuffer is appropriate and aligns with the updated arrow library. However, the use of unwrap should be avoided to prevent potential runtime panics. Adding comments to explain the removal of certain fields would be beneficial for maintaining the code in the future.

Yours, Gooroo.dev
Your feedback is important! Please react or reply.

@Swoorup Swoorup deleted the sj-update-arrow branch June 10, 2024 16:58
Copy link

gooroo-dev bot commented Jun 10, 2024

🔄 New Commits Added

Author Commit Name Date
Swoorup Update UnionArray::try_new usage 2024-06-10 16:35:14 UTC

Reviewing these new changes. To re-review the whole PR, mention me and say review.

🐞Mistake 🤪Typo 🚨Security 🚀Performance 💪Best Practices 📖Readability ❓Others
0 0 0 0 1 1 0

Summary of Incoming Changes

  • ✅ Updated the usage of arrow::buffer::Buffer to arrow::buffer::ScalarBuffer to align with the new version of the Arrow library.
  • ➖ Removed unnecessary field_type_ids and related logic, simplifying the code.
  • 📖 Improved readability by removing redundant code.

Identified Issues

ID Type Details Severity Confidence
1 💪Best Practices The field_type_ids field and related logic is removed but not documented. 🟡Low 🟠Medium
2 📖Readability The removal of field_type_ids is not clearly explained in comments. 🟡Low 🟠Medium

Proposed Fixes

Issue 1: Best Practices

File: arrow_convert_derive/src/derive_enum.rs
Lines: Around 253

Proposed Change:
Add a comment explaining the removal of field_type_ids to maintain clarity for future developers.

// The `field_type_ids` field and related logic have been removed as they are no longer necessary
// with the updated Arrow library version.

Issue 2: Readability

File: arrow_convert_derive/src/derive_enum.rs
Lines: Around 253

Proposed Change:
Include a comment to explain the simplification.

// Simplified by removing `field_type_ids` which is no longer required.

General Review

The code quality is generally good, and the style is consistent. The most important issues are the lack of documentation explaining the removal of field_type_ids and related logic, which could confuse future maintainers. Adding comments to clarify these changes would significantly improve readability and adherence to best practices.

Yours, Gooroo.dev
Your thoughts matter! Please reply or add a reaction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants