Enable tokenizers in shortfin packages on Linux x86_64.

[ScottTodd]

Fixes #679.

Multi-arch support is hacky at the moment, but we are only building for x86_64 today and not aarch64. I think the ideal path forward there is to publish our own multi-platform images that include all of our deps (including rust and ccache), so I started on that with ScottTodd/nod-ai-base-docker-images@bdc5cc5. Another option is to continue using the upstream manylinux dockerfiles and install more deps on-demand like yum install -y rust cargo).

Tested package build times and sizes:

Description	Logs	Duration	Package size	Cache hit rate
Baseline rc build	logs here	5m15s	2.6mb
New rc build	logs here	8m40s	11.8mb
Baseline dev build 1	logs here	2m49s	1.2mb	0%
Baseline dev build 2	logs here	2m9s	1.2mb	45%
New dev build 1	logs here	4m1s	5.8mb	58%
New dev build 2	logs here	3m52s	5.8mb	67%

Dev package builds will be used for #584, so keeping those times low is a priority. Could disable tokenizers there, at least as long as using the native tokenizers library is optional

Observations:

• 30s docker pull before, 45s docker pull after
  • Could try to prune more dependencies
• 365 build actions before, 412 build actions after
  • Still lower cache hit rates than expected

[marbre]

Let's wait for a signal from @stellaraccident before the release to decide if this should be included in the stable releases.

[ScottTodd]

Let's wait for a signal from @stellaraccident before the release to decide if this should be included in the stable releases.

SGTM!