forked from IAPark/tiktoken_ruby
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
I've had a number of compilation issues with this gem on different platforms. I don't want us to have to install Rust everywhere as an additional dependency for a single gem. So I created this fork that does cross-compilation in Github Actions. There's one other problem I discovered, which is that Github's package management appears to ignore platforms when uploading gems. When you upload a gem with the same version number but a different platform, Github says you can't upload it since that version has already been uploaded. Rubygems.org doesn't have this problem. HOWEVER, I don't want to push the same gem name to Rubygems.org and get it confused with the upstream project. This is why I renamed the gem to rz_tiktoken_ruby. Please don't confuse it for the original (but do try it out if you need a version pre-built for your platform.)
- Loading branch information
Showing
5 changed files
with
47 additions
and
32 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
# frozen_string_literal: true | ||
|
||
module Tiktoken | ||
VERSION = "0.0.5" | ||
VERSION = "0.0.6" | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -3,25 +3,29 @@ | |
require_relative "lib/tiktoken_ruby/version" | ||
|
||
Gem::Specification.new do |spec| | ||
spec.name = "tiktoken_ruby" | ||
spec.name = "rz_tiktoken_ruby" | ||
spec.version = Tiktoken::VERSION | ||
spec.authors = ["IAPark"] | ||
spec.email = ["[email protected]"] | ||
spec.authors = ["IAPark", "judy"] | ||
spec.email = ["[email protected]", "[email protected]"] | ||
|
||
spec.summary = "Ruby wrapper for Tiktoken" | ||
spec.description = "An unofficial Ruby wrapper for Tiktoken, " \ | ||
"a BPE tokenizer written by and used by OpenAI. It can be used to " \ | ||
"count the number of tokens in text before sending it to OpenAI APIs." | ||
|
||
spec.homepage = "https://github.com/IAPark/tiktoken_ruby" | ||
spec.description = <<~TEXT | ||
An unofficial Ruby wrapper for Tiktoken, a BPE tokenizer written by and used by OpenAI. It | ||
can be used to count the number of tokens in text before sending it to OpenAI APIs. This | ||
is a fork of tiktoken_ruby by IAPark, which has been cross-compiled for multiple platforms. | ||
This way compilation with Rust extensions doesn't need to happen wherever you are deploying it. | ||
TEXT | ||
|
||
spec.homepage = "https://github.com/retailzipline/tiktoken_ruby" | ||
spec.license = "MIT" | ||
spec.required_ruby_version = ">= 2.7.0" | ||
spec.required_rubygems_version = ">= 3.1.0" | ||
spec.platform = Gem::Platform::RUBY | ||
|
||
spec.metadata["homepage_uri"] = spec.homepage | ||
spec.metadata["source_code_uri"] = "https://github.com/IAPark/tiktoken_ruby" | ||
spec.metadata["source_code_uri"] = "https://github.com/retailzipline/tiktoken_ruby" | ||
spec.metadata["documentation_uri"] = "https://rubydoc.info/github/IAPark/tiktoken_ruby/main" | ||
spec.metadata["allowed_push_host"] = "https://rubygems.org" | ||
|
||
# spec.metadata["changelog_uri"] = "TODO: Put your gem's CHANGELOG.md URL here." | ||
|
||
|