Ruby 3.3 #4
Merged
Ruby 3.3 #4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
judy
commented
Dec 26, 2023
judy
commented
- Update magnus for ruby-head compatibility
- Enable stable-api-compiled-fallback
- Bump version to 0.0.6
- INTERNAL: Update libraries (INTERNAL: Update libraries #1)
- Set up a fork with native builds
- Update README.md
- Re-cargo
Older versions of magnus fail to compile on Ruby 3.3.0-dev with: ``` Compiling tiktoken-rs v0.3.2 (https://github.com/IAPark/tiktoken-rs.git#5231fbf4) error[E0609]: no field `len` on type `RString__bindgen_ty_1__bindgen_ty_1` --> /tmp/bundle/ruby/3.3.0+0/gems/tiktoken_ruby-0.0.5/ext/tiktoken_ruby/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/magnus-0.4.4/src/r_string.rs:366:57 | 366 | slice::from_raw_parts(h.ptr as *const u8, h.len as usize) | ^^^ unknown field | = note: available fields are: `ptr`, `aux` error[E0609]: no field `len` on type `RString__bindgen_ty_1__bindgen_ty_1` --> /tmp/bundle/ruby/3.3.0+0/gems/tiktoken_ruby-0.0.5/ext/tiktoken_ruby/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/magnus-0.4.4/src/r_string.rs:954:19 | 954 | h.len as usize | ^^^ unknown field | = note: available fields are: `ptr`, `aux` error[E0609]: no field `len` on type `RString__bindgen_ty_1__bindgen_ty_2` --> /tmp/bundle/ruby/3.3.0+0/gems/tiktoken_ruby-0.0.5/ext/tiktoken_ruby/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/magnus-0.4.4/src/r_string.rs:996:44 | 996 | value.as_internal().as_ref().as_.embed.len | ^^^ unknown field | = note: available fields are: `ary` For more information about this error, try `rustc --explain E0609`. error: could not compile `magnus` (lib) due to 3 previous errors warning: build failed, waiting for other jobs to finish... make: *** [Makefile:564: target/release/libtiktoken_ruby.so] Error 101 ```
Update magnus for ruby-head compatibility
This should solve an issue with compiling on Ruby 3.3.0-dev
* bundle update && cargo update * Update rb_sys in gemspec * Change to original tiktoken-rs library @IAPark's fork was originally created to remove async-openai as an added library, due to unnecessary build issues. The upstream library has since made that library optional. --------- Co-authored-by: Clinton Judy <[email protected]>
I've had a number of compilation issues with this gem on different platforms. I don't want us to have to install Rust everywhere as an additional dependency for a single gem. So I created this fork that does cross-compilation in Github Actions. There's one other problem I discovered, which is that Github's package management appears to ignore platforms when uploading gems. When you upload a gem with the same version number but a different platform, Github says you can't upload it since that version has already been uploaded. Rubygems.org doesn't have this problem. HOWEVER, I don't want to push the same gem name to Rubygems.org and get it confused with the upstream project. This is why I renamed the gem to rz_tiktoken_ruby. Please don't confuse it for the original (but do try it out if you need a version pre-built for your platform.)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.