feat: easier and more flexible loading of sqlite extensions

[flavorjones]

Using sqlite extensions in a Rails app is currently a bit tricky.

Assuming you can figure out how to compile it, where do you put the extension on disk? If it's provided by a gem, how do you get a reliable path that works robustly across environments and upgrades?

Do you monkeypatch the sqlite3 adapter to call #load_extension, or do you explicitly inject the extension yourself before any query that needs it?

Using sqlite extensions shouldn't be this hard.

The goal of this PR is to enable injection of sqlite extensions by specifying the name of a Ruby constant in a Rails database config:

development:
  adapter: sqlite3
  database: storage/development.sqlite3
  extensions:
    - SQLean::Crypto

0. Establishing a packaging convention

Over this past weekend, I packaged up the https://github.com/nalgeon/sqlean/ extensions as a Ruby gem: https://github.com/flavorjones/sqlean-ruby

The gem offers a set of modules that simply return the filesystem path of the extension files. Each module implements this interface, which I'll call _ExtensionSpecifier:

interface _ExtensionSpecifier
  def to_path: () → String
end

So, for example, on my machine:

SQLean::Crypto.to_path
=> "/home/flavorjones/.../ruby/3.3.6/lib/ruby/gems/3.3.0/gems/sqlean-0.2.0-x86_64-linux-gnu/lib/sqlean/dist/x86_64-linux-gnu/crypto"

which means I can load the extension like so:

db = SQLite3::Database.new(":memory:")
db.enable_load_extension(true)
db.load_extension(SQLean::Crypto.to_path)

I think this is a reasonable self-describing pattern for an extensions gem to offer. Assuming a future world where other people package up their extensions like this ...

1. Database#load_extension contract

Database#load_extension now supports _ExtensionSpecifier in addition to filesystem paths.

When passed an object that responds to #to_path, that method's return value is used as the filesystem path for the extension, allowing a terser calling convention than above:

db.load_extension(SQLean::Crypto)

2. Database#new supports extensions:

Database#new supports an extensions: keyword argument which accepts:

• String filesystem path
• _ExtensionSpecifier object

3. Future work

I'll open a PR with Rails to constantize class names that appear in config/database.yml so they're then passed through to the gem. That should unlock usage like:

development:
  adapter: sqlite3
  database: storage/development.sqlite3
  extensions:
    - SQLean::Crypto

[tenderlove]

When passed an object that responds to #sqlite_extension_path, that method's return value is used as the filesystem path for the extension, allowing a terser calling convention than above:

What about using the method #to_path instead? Then it could work with these modules or a Path object?

• the name of a constant that is an _ExtensionSpecifier

Unsure if I like this. I don't know if I like that SQLite gets in to the business of knowing how to constantize things. I feel this should be a responsibility of the Rails SQLite adapter layer rather than the SQLite gem itself.

I could see something like:

Database.new(":memory:", extensions: [SQLean::Crypto])

Is there a particular use case where we would want to support strings in the extensions array outside of Rails?

[flavorjones]

@tenderlove Thanks for the feedback!

What about using the method #to_path instead? Then it could work with these modules or a Path object?

Oh, that's a better idea. I'll switch it up.

I feel this should be a responsibility of the Rails SQLite adapter layer rather than the SQLite gem itself.

I don't disagree? But the Rails sqlite adapter is currently just shoveling the ActiveRecord::DatabaseConfigurations::HashConfig hash to SQLite3::Database.new with very little modification or sanity-checking. I actually started adding the const_get code to the sqlite adapter and it felt weird treating this one value so differently from the rest of the config.

Doing it in Rails also limits the use of SQLite extensions to Rails 8.1+, which might be the best reason to do it in this gem.

Is there a particular use case where we would want to support strings in the extensions array outside of Rails?

Not that I know of, beyond perhaps other web frameworks with the same problem space.

[flavorjones]

Updated to use #to_path instead of #sqlite_extension_path.

[flavorjones]

@tenderlove Thanks again for the feedback. I've removed constantization from this PR since I can point people at https://fractaledmind.github.io/2023/12/24/enhancing-rails-installing-extensions/ for an explanation of how to load extensions in an existing Rails app.

I'll submit a PR to Rails for constantizing the database config extension strings.