Scheduling of upcoming PRs

[philpax]

I've been merging PRs fairly haphazardly, but they're all starting to conflict with each other. I propose the following merge order and then we ship 0.1.0:

• Sync to llama.cpp + GGML version as of 20230407 09:57 AM UTC. #119 - most pressing, latest version of llama/ggml logic
• fix(llama): buffer tokens until valid UTF-8 #122 - fixes byte token behaviour
• Reserve more eval memory and use ggml scratch buffers #116 - assuming that this is the correct fix
• Standalone loader #125 - blocked on getting it working, but this lays the foundation for loading models of other formats and should let us support all three GGML formats
• Ported quantize.cpp #84 - blocked on fixing the logic, but means users won't need upstream llama.cpp at all
• WIP: Bloom Inference #85 - finally, implements our first non-LLaMA model (with hopefully more to come), and will likely depend on loading logic touched by Add GGJT loader #114 and Ported quantize.cpp #84, and needs to be updated to match Sync to llama.cpp + GGML version as of 20230407 09:57 AM UTC. #119 and Reserve more eval memory and use ggml scratch buffers #116
• and then cleanup work to integrate all of these together (esp with Ported quantize.cpp #84, which introduces loading helper functions that could be used by Add GGJT loader #114 and WIP: Bloom Inference #85)

No other PRs that significantly touch logic will be merged outside of this order unless they're critical to getting things working.

Please confirm you're OK with this, or suggest an alternate plan: @setzer22 @KerfuffleV2 @iacore @FloppyDisck @hhamud

[KerfuffleV2]

It looks reasonable to me, but that's not worth much. I have no qualifications for juggling pull requests.

Is putting the GGML Rust bindings in a separate repo/project something that's on the roadmap? (I've been looking at trying to use it with my RWKV stuff for example, but that's going to require adding some ops and such.)

[philpax]

We might, yeah - separating ggml-rs out is a step towards that. For now, you should be able to use it as a Git dependency, but the additional ops would be a little annoying to manage. Let's see how we feel after the release.

[iacore]

ggml is not published to crates.io

you can use a specific git commit of this repo for now

[hhamud]

I'm fine with this

[KerfuffleV2]

By the way, #114 already works with a 1 line code change (and the nicer way to do it is obvious). Unless iacore doesn't like the idea of the non-mmap approach that one really just needs code cleanups to be suitable for merging.

[iacore]

Which PRs are considered "done"? Those should be merged first. I don't care about the order.

I want to fix the converting back and forth of f16_ (should be named tensor_element_type) after this is all sorted out

[FloppyDisck]

It sounds good to me

[philpax]

Quick update on this: #84 is dependent on #125 in order to support all three model types (it currently only supports GGMJ), so we can't merge that yet. #85 is now very out of sync (sorry, @hhamud!) but hopefully the changes that have been introduced since will make it easier to reconstruct and to share as much code as possible.

[philpax]

Our thinking here is that we'll get two models in (at this stage, LLaMA and BLOOM) and design a trait-based API for them, but the reason that hasn't happened yet is because many of the foundational elements are/were still in churn. Things are beginning to stabilise now, so we're in a better position to scope this out properly. That's also why we haven't released on crates.io yet - we're taking the time to make sure the first release can be used with all compatible models, and that its API is usable for a wide range of applications.

It would also be nice to see a greater emphasis on testing, which seems like it would be very difficult to incorporate into this repository at this point.

That's something we're aware of, but there's no easy answers here - there's no way to test it without weights, and there are no weights for LLaMA that are small or legally unburdened enough to use in CI. The individual components can't really be tested in isolation without some form of weights, either.

Once/if there are LLaMA-arch compatible weights that can be used for deterministic testing, I imagine that this will improve significantly. Open to suggestions on what else can be done here.

[danforbes]

I took a stab at creating a Model trait and a Llama implementation...is this the kind of thing you had in mind? main...danforbes:llama-rs:dfo/refactor/traits

[hhamud]

I took a stab at creating a Model trait and a Llama implementation...is this the kind of thing you had in mind? main...danforbes:llama-rs:dfo/refactor/traits

I had something quite similar in my bloom PR #85 where I attempted to implement a trait for both llama and the bloom models as they are quite similar, however may need something more flexible with incoming #75 model.

[danforbes]

Yes, I definitely referred to your changes! Unfortunately, they had fallen rather behind the tip of this repo, and I also tried to make fewer structural changes than you did.

[danforbes]

Here is a branch that incorporates the Bloom additions by @hhamud: main...danforbes:llama-rs:dfo/model/bloom