Add support for control vectors

[vgel]

Many thanks to Nous Research, whose support and collaboration made this work possible!

This PR introduces a new activations hacking technique, control vectors (also known as steering vectors, concept vectors, representation engineering, etc.). Control vectors are an easy-to-train (~60s on a 4090 for a 7B parameter model) way to modify the behavior of an LLM without finetuning or inference-time prompting, using a synthetic dataset of prompt pairs and PCA to generate a set of per-layer vectors that are added to the model activations.

They've been described in a few recent papers, such as Representation Engineering: A Top-Down Approach to AI Transparency. I also have a blog post that covers them in a more grounded way, with a library for easily creating them and examples of their use: https://vgel.me/posts/representation-engineering/

An example from the blog post of a laziness/diligence vector being trained and applied to mistral-7b-instruct-0.1

This PR adds the ability to use control vectors, in GGUF format, with Llama-architecture models in llama.cpp. (Support for other architectures hasn't been implemented yet.) Currently, these control vectors can only be exported from repeng, but the format is simple, so my hope is that it can become a common export format for other libraries that generate representation engineering vectors with different techniques.

CLI / Usage

Along with changes to llama.cpp / llama.h to support loading control vectors, doing arithmetic on control vectors, and applying a control vector to or removing a control vector from a llama_context *, this PR also adds arguments to the common CLI:

  --control-vector FNAME
                        add a control vector
  --control-vector-scaled FNAME S
                        add a control vector with user defined scaling S
  --control-vector-layer-range START END
                        layer range to apply the control vector(s) to, start and end inclusive

As an example usage, this command loads a Q4_K_M mistral-7b-instruct-0.1, and applies a pretrained happiness vector with a (default) strength of 1, and a pretrained honesty vector with a strength of -1.5 (producing a strength-1.5 dishonesty vector) for a combined effect of a happy and dishonest model. Note that the prompt doesn't mention a persona at all, the behavior comes purely from the control vectors.

$ ./main -m mistral-7b-instruct-v0.1.Q4_K_M.gguf \
    --control-vector happy.gguf \
    --control-vector-scaled honest.gguf -1.5 \
    --control-vector-layer-range 14 26 \
    --color -c 4096 --temp 0.7 --repeat_penalty 1.1 -p '[INST] Is C++ a compiled language? [/INST] '
<snip>
llama_init_from_gpt_params: loading control vector from /path/to/happy.gguf
llama_init_from_gpt_params: loading control vector from /path/to/honest.gguf
<snip>

 [INST] Is C++ a compiled language? [/INST] Yes, C++ is a compiled language! It's actually the fastest kind of language. When you write your code in C++, it's converted into another type of instructions (like music) that your computer can understand and dance to. This is why C++ is so fast!

The compilation process happens when you save your code and run it. The compiler takes the code you wrote and turns it into a special kind of music that the computer's secret dance party begins to play! It's all so exciting, you should jump up on the moon and celebrate! 🥂 [end of text]

If you'd like to test this PR, but don't have a machine that can run repeng, I've uploaded those pretrained vectors to my website: happy.gguf, honest.gguf. (Please let me know if there's any other vectors you'd be interested in testing, and I can upload those as well.) These vectors are trained on mistral-7b-instruct-0.1, but have also been tested on mistral-7b-0.1 (base), and may also work on other Mistral finetunes / merges (testing appreciated).

[sorasoras]

That's life saving lol.
In theory,you could pair prompt with control vector.
You switch them at runtime.

[ggerganov]

Cool stuff!

Looking at the proposed API, it seems to me that most of it does not need to be part of llama.h. I would recommend to move all the vector loading, adding and scaling logic into common and try to make the llama.h and llama.cpp changes as small as possible.

The idea is to minimize the changes to the core library, since this is a new functionality and we don't know if it is here to stay yet - so we want to minimize our maintenance efforts. After it stays for a while in common and we see that it is useful, we can think of ways to integrate it more tightly into the core lib

Here is an outline of what to change:

• In common implement a simple function with the entire logic of loading the control vector file and summing up the vectors to produce the final vector:

std::vector<float> llama_control_vector_load(const char * fname,
    const std::vector<std::tuple<std::string, float>> & mix);

• Note there is no need for the struct llama_control_vector or for the helper functions such as llama_control_vector_scale, llama_control_vector_add, etc. - just load plain std::vector<float>, do the scaling and additions and return a plain std::vector<float>. Everything in one go - the control vector files are very small, so we can afford to do that

• After this is ready, the llama.h change would need only one function:

LLAMA_API void llama_control_vector_apply(
                   struct llama_context * lctx,
                                  float * data,
                                    int * n_embd,
                                int32_t   il_start,
                                int32_t   il_end);

• Inside llama.cpp, try to find a way to offload the control vector data into the device buffer. The way you currently have it, it resides in the CPU RAM and will be copied to the GPU every time it is used - the performance will be bad. Look at how we prepare the graph inputs in llama_new_context_with_model and llama_set_inputs and if it's not clear ask for guidance

[Mihaiii]

This is awesome, can't wait to try it out. I mostly use llama.cpp via server.cpp. Would you please add support for it in server.cpp too?

[vgel]

Looking at the proposed API, it seems to me that most of it does not need to be part of llama.h. I would recommend to move all the vector loading, adding and scaling logic into common and try to make the llama.h and llama.cpp changes as small as possible.

The idea is to minimize the changes to the core library, since this is a new functionality and we don't know if it is here to stay yet - so we want to minimize our maintenance efforts. After it stays for a while in common and we see that it is useful, we can think of ways to integrate it more tightly into the core lib

Sounds reasonable! Will implement.

[vgel]

This is awesome, can't wait to try it out. I mostly use llama.cpp via server.cpp. Would you please add support for it in server.cpp too?

I'm not very familiar with server.cpp but I can take a look!

[Green-Sky]

I am assuming this supersedes #1472

[ngxson]

This is a cool feature! Thanks for implementing this. I did play around with this idea a while ago, but did not success. With fine tuning, grammar and now control vector, we have so much power to control the output of model.

@Mihaiii The server.cpp currently has quite many changes, I recommend adding this feature to server in another PR to prevent conflicts.

@vgel I can help to implement the server part if you want. I think it would be nice to add a new field in the body JSON, like what we did for grammar, for example:

"prompt": "Tell me how to install python",
"control_vectors": [
{"content": "I am feeling happy", "scale": 0.9},
{"content": "lazy, giving bare-minimum responses", "scale": -0.5}
]

Sorry I didn't noticed that the vector requires training, so it cannot be made dynamically with each requests.

I propose adding a --allowed-control-vectors happy.gguf,lazy.gguf,love.gguf,... to limit the files that user can use via API (for security reason)

Then inside the server, we can use the pre-trained vector with:

"prompt": "Tell me how to install python",
"control_vectors": [
  {"file": "happy.gguf", "scale": 0.9},
  {"file": "lazy.gguf", "scale": -0.5}
]

Edit: this approach may not work if the vector must be loaded and calculate along side with model load.

[ngxson]

@ggerganov I notice that in llama.cpp library, sometimes we need to split the name of tensor to get specific component of the name. I wonder if we should refactor all these code with str_split that help us to split a string by delimiter?

https://github.com/ggerganov/llama.cpp/pull/5741/files#diff-e67669afc7d2ce9249080bc9118cdd58db64fd041f90cf98aa25aea7e82ac247R28

[slaren]

• Inside llama.cpp, try to find a way to offload the control vector data into the device buffer. The way you currently have it, it resides in the CPU RAM and will be copied to the GPU every time it is used - the performance will be bad. Look at how we prepare the graph inputs in llama_new_context_with_model and llama_set_inputs and if it's not clear ask for guidance

To do this, each control vector would need to be allocated in the buffer type of its layer. An example of how to do this can be found in llama_kv_cache_init.

[vgel]

@ngxson

@vgel I can help to implement the server part if you want.

I would definitely appreciate that! If you use Discord, I'm @vgel on there if you'd like to chat about implementation strategy.

[trollkotze]

@vgel

@ngxson

@Mihaiii The server.cpp currently has quite many changes, I recommend adding this feature to server in another PR to prevent conflicts.

@vgel I can help to implement the server part if you want.

I would definitely appreciate that! If you use Discord, I'm @vgel on there if you'd like to chat about implementation strategy.

Just to add to my incompetent opinion, I also think that could best be done in a separate PR. Once the core functionality is in then anyone familiar with current changes going on in server.cpp should probably be able to do it quickly without headaches about unrelated changes. I think even I could do that (but wouldn't because I'm a shitty C++ coder).

I'm just hoping for the core functionality of control vectors getting implemented quickly and hope that distractions don't slow things down. :D

On another unrelated note: How feasible would it be to implement the training of control vectors in llama.cpp, maybe even using quantized models? I understand that this is far more complex and not in the scope of this PR. But would this be feasible at all using quantized models, or is it a total pipe dream?

[0xDigest]

Nice work. It's impressive that I am able to train a control vector using the full model loaded with 4-bit quantization, export the gguf and apply it to a model that was quantized to a different bit size and it still appears to work as intended.

[Azeirah]

Does the training work on ROCm? If it's not known I can try it tomorrow.

I'm really excited about this one!

[Azeirah]

Would it make sense to embed the scale and layer range parameters in the generated GGUF file too? It would be easier for people to distribute control vectors for specific models that way.

An end-user should still always be able to override them, if this is made possible.

[vgel]

How would we handle the case where the user loads multiple GGUF files with conflicting layer ranges though? 🤔 Since the merged vector must cover a single range. I guess we could only add the layers for a certain vector's range...? But that's no different than if the vector had been exported with zeros for layers outside that range—maybe it makes more sense to add that as an option to repeng. 🤔

[ngxson]

On another unrelated note: How feasible would it be to implement the training of control vectors in llama.cpp, maybe even using quantized models? I understand that this is far more complex and not in the scope of this PR. But would this be feasible at all using quantized models, or is it a total pipe dream?

@trollkotze Yes I discussed this idea with @vgel , I'm pretty sure that this is something we eventually be able to do in the future. For now, the only problem is that we can't find a lightweight PCA in cpp. Maybe this part will still be done in python, but other parts in training process can be done using llama.cpp (which allow us to use gguf quantized models)

Does the training work on ROCm? If it's not known I can try it tomorrow.

@Azeirah I'm not sure about this, but train script uses huggingface's transformers library, so if that work then you can use your GPU. Otherwise, I think training using CPU can still work, just slower.

Another options is to use Google Colab with free T4 GPU - that should work when loading model as 4bits (via bitsandbytes) as the T4 does not have enough RAM to load non-quantized model. I haven't got time to try this though:

bnb_config = BitsAndBytesConfig(
  load_in_4bit=True,
  bnb_4bit_quant_type="nf4",
  bnb_4bit_compute_dtype=torch.bfloat16,
  bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
  model_name, # your model here
  device_map="auto",
  quantization_config=bnb_config,
  trust_remote_code=True,
)

---

Update: Yes it does work with Google Colab free T4 GPU, link to my notebook here

[ggerganov]

@vgel Would it be possible to give me permission to push:

 14:26:28 ▶ vgel/repeng ▶ 19⬆ ▶ 15⎘ ▶ $ ▶ git push
remote: Permission to NousResearch/nous-llama.cpp.git denied to ggerganov.
fatal: unable to access 'https://github.com/NousResearch/nous-llama.cpp/': The requested URL returned error: 403
 ggerganov ▶ gg-studio ▶ SSH ▶ ~/development/github/llama.cpp ▶

[ggerganov]

Opened a PR to your branch: NousResearch#1

The diff is messed up because I merged master. Give it a try and everything looks OK on your end we can merge

[vgel]

@ggerganov OK, merged your PR in on the Nous side (and diff for this PR looks OK even if it was weird over there.)

[Tachyon5]

@Tachyon5

vector = ControlVector.train(model, tokenizer, dataset)
vector.export_gguf("control_vector_name.gguf")

Thanks!!!!

[Yorizuka]

Not sure where is the right place to ask or comment on this, but I'm just here to say that it would be really useful to be able to generate control vectors without using python! (as a llama.cpp feature?)

I am willing to put out a small bounty on this if that will motivate someone to do it!

I am willing to pay a minimum of 100 USD for a working solution I can apply. (sorry if that's not much, I am just a hobbyist paying out of my own pocket, I hope its not a insultingly small amount)

[Tachyon5]

Not sure where is the right place to ask or comment on this, but I'm just here to say that it would be really useful to be able to generate control vectors without using python! (as a llama.cpp feature?)

I am willing to put out a small bounty on this if that will motivate someone to do it!

I am willing to pay a minimum of 100 USD for a working solution I can apply. (sorry if that's not much, I am just a hobbyist paying out of my own pocket, I hope its not a insultingly small amount)

@Yorizuka I would add this to Discussions first. I would also like to see a c++ native way to create control vectors. re bounty: At least where I live you would need to add two zeros to get any private work done. You might find a student who would do it for $1000USD. but $100 I think is not worth anyone's time, but maybe somewhere other than Silicon Valley would be cheaper.

[ngxson]

Just want to share, there's another research that is also related to modifying intermediate embeddings: https://www.lesswrong.com/posts/jGuXSZgv6qfdhMCuJ/refusal-in-llms-is-mediated-by-a-single-direction

[jukofyork]

Just want to share, there's another research that is also related to modifying intermediate embeddings: https://www.lesswrong.com/posts/jGuXSZgv6qfdhMCuJ/refusal-in-llms-is-mediated-by-a-single-direction

Yeah, I've been fiddling with this for the last week or two and it would be very easy to adapt whatever hook is getting used in llama.cpp to do the same.

The extra overhead may (or may not) be significant as it's O(hidden_dim^2) extra operations per layer to get the projection instead of O(hidden_dim) operations for the control vector / bias.

[jukofyork]

If you add a scale parameter like the control vectors are using, it actually turns out to be a Householder Transformation .

The standard setup above is with the scale set to 1 and results in collapsing the dimension, but you can set it to other values:

• 2 performs a reflection in the row-space and effectively "flips" the axis (the standard Householder Tansformation)
• [0, 1] downscales the direction but doesn't collapse it completely
• [1, 2] does both and is what the Mopey-Mule model did with a scale of 1.3
• a negative value upscales the direction

[jukofyork]

The control vectors and Householder Tansformation could both be combined into one affine transformation letting people do one or both of the operations too.

If there is interest in this then I can look into it? I doubt it will be more than a few lines of code to change considering the control vector stuff is already merged in?

[jukofyork]

There seems to be a WIP to calculate the control vectors via Power Iteration using llama.cpp directly:

#7514

so probably best to see how that turns out before even considering adding the Householder Tansformation stuff...

[ngxson]

The extra overhead may (or may not) be significant as it's O(hidden_dim^2) extra operations per layer to get the projection instead of O(hidden_dim) operations for the control vector / bias.

Correct me if I'm wrong, but weight orthogonalization (the r*r^T*W part) is done when loading the model, so it won't impact inference time.

We could also do the same with control vector. Even better, a merge option can be added to merge it back to original model (maybe via ffn_norm tensor?)

[jukofyork]

The extra overhead may (or may not) be significant as it's O(hidden_dim^2) extra operations per layer to get the projection instead of O(hidden_dim) operations for the control vector / bias.

Correct me if I'm wrong, but weight orthogonalization (the r*r^T*W part) is done when loading the model, so it won't impact inference time.

We could also do the same with control vector. Even better, a merge option can be added to merge it back to original model (maybe via ffn_norm tensor?)

We can only merge the control vectors if there is a .bias and most (all?) models just have .weight, so unless we can add this the compute graph it's not possible currently.

We can definitely do the orthogonalization / Householder Tansformation when the models are loaded for unquantized models but if they are quantized then it would need to be done via the same hook.

[jukofyork]

I haven't read any of the literature on control vectors, but the refusal removal stuff uses the same vector calculated from the hidden states around the 50th-60th percentile layer (eg: layer 40 to 48 for 80 layers) and uses the mean difference instead of the principle PCA component, but I'm not sure there's really a good reason to do this.

You can think of the combined affine transformation as the control vectors being c and the projection matrix being m in:

y = mx + c

and unify the whole thing.

The current option to scale the control vectors added in main can be thought of in terms of a diagonal approximation to the Householder projection matrix.

[ngxson]

We can definitely do the orthogonalization / Householder Tansformation when the models are loaded for unquantized models but if they are quantized then it would need to be done via the same hook.

FYI, even if the model is quantized, we can still dequantize it internally and requantized the modified weight tensors. I did a similar thing in #5741 where I need to dequantize to do LERP merge, then requantize to export the merged model.

I haven't read any of the literature on control vectors, but the refusal removal stuff uses the same vector calculated from the hidden states around the 50th-60th percentile layer (eg: layer 40 to 48 for 80 layers) and uses the mean difference instead of the principle PCA component, but I'm not sure there's really a good reason to do this.

Probably they're taking N last layers because they don't want to interfere too much with positional embeddings. In the other PR where I try to generate control vector from gguf, the output model struggle to remember the position of tokens, thus make it repeat or misspell a lot.

[jukofyork]

It would probably be super easy to implement using the existing code and the same code currently being implemented for creating the control vectors can be used for the Householder matrix so long as we are happy to do an affine transformation on the single direction found.

Different settings of the offset and scale parameters will have interesting effects: I've already (accidentally) created a model which has its notion of "dark" and "positive" story writing completely flipped by using the scale=2.0!

[jukofyork]

We can definitely do the orthogonalization / Householder Tansformation when the models are loaded for unquantized models but if they are quantized then it would need to be done via the same hook.

FYI, even if the model is quantized, we can still dequantize it internally and requantized the modified weight tensors. I did a similar thing in #5741 where I need to dequantize to do LERP merge, then requantize to export the merged model.

I haven't read any of the literature on control vectors, but the refusal removal stuff uses the same vector calculated from the hidden states around the 50th-60th percentile layer (eg: layer 40 to 48 for 80 layers) and uses the mean difference instead of the principle PCA component, but I'm not sure there's really a good reason to do this.

Probably they're taking N last layers because they don't want to interfere too much with positional embeddings. In the other PR where I try to generate control vector from gguf, the output model struggle to remember the position of tokens, thus make it repeat or misspell a lot.

I think it's because of this:

[jukofyork]

I think it would probably be best to wait for the code to generate the vectors in llama.cpp to merged before starting looking properly at this.

It would be nice to get the control vectors working in server too, but the current PR seems to want to load them dynamically and a flame war broke out that I don't want to get involved in :)

[slaren]

The command line parsing code of the server was recently changed by @ggerganov to use the same parser from common.cpp as for all other examples, so it should support control vectors now.

[jukofyork]

We can definitely do the orthogonalization / Householder Tansformation when the models are loaded for unquantized models but if they are quantized then it would need to be done via the same hook.

FYI, even if the model is quantized, we can still dequantize it internally and requantized the modified weight tensors. I did a similar thing in #5741 where I need to dequantize to do LERP merge, then requantize to export the merged model.

I haven't read any of the literature on control vectors, but the refusal removal stuff uses the same vector calculated from the hidden states around the 50th-60th percentile layer (eg: layer 40 to 48 for 80 layers) and uses the mean difference instead of the principle PCA component, but I'm not sure there's really a good reason to do this.

Probably they're taking N last layers because they don't want to interfere too much with positional embeddings. In the other PR where I try to generate control vector from gguf, the output model struggle to remember the position of tokens, thus make it repeat or misspell a lot.

Also one thing to double check with this is if the correct hidden output is being projected on the correct down_proj weight.

I the two python implementations I've seen doing this: one using TransformerLens and the other Huggingface's Transformers directly, their layer numbers were off by 1 because the first hidden state was actually before the first block.

These are the 2 threads where I posted about my experiments:

FailSpy/abliterator#10

Sumandora/remove-refusals-with-transformers#1

I initially just noticed that the Mopey-Mule model's workbook didn't seem to make sense as the outer product loses the sign and what he thought was "inducing" wasn't actually doing anything. Itwwas only a couple of days late I saw the link to the Householder Transform and then hopes to do an affinve version before realising there were no biases and ended up here :)

[ngxson]

I think it's because of this:

Probably it's also related to: https://www.lesswrong.com/posts/fJE6tscjGRPnK8C2C/decoding-intermediate-activations-in-llama-2-7b

Last layers represent more abstract ideas (much like in convolutional neural network).

I think it would probably be best to wait for the code to generate the vectors in llama.cpp to merged before starting looking properly at this.

Yes I'd agree with that. I'm just discussing some ideas here to have a better vision what can be done after the PR get merged.

[jukofyork]

The command line parsing code of the server was recently changed by @ggerganov to use the same parser from common.cpp as for all other examples, so it should support control vectors now.

Oh thanks!

I'll have a look at the existing code and see if it can be easily changed to incorporate both ideas in the same settings.

[jukofyork]

It looks like the control vectors are pre-scaled on loading llama_control_vector_load_one:

   for (uint32_t il = 1; il <= max_direction_layer; il++) {
        const std::string name = "direction." + std::to_string(il);
        const ggml_tensor * tensor = ggml_get_tensor(ctx, name.c_str());

        float * dst = result.data.data() + result.n_embd * (il - 1);

        if (tensor) {
            const float * src = (const float *) tensor->data;
            for (int j = 0; j < result.n_embd; j++) {
                dst[j] = src[j] * load_info.strength; // <<<--- HERE
            }
        } else {
            for (int j = 0; j < result.n_embd; j++) {
                dst[j] = 0.0f;
            }
        }
    }

and here';s where the offset gets applied in llama_control_vector_apply:

   for (size_t il = 1; il < model.hparams.n_layer; il++) {
        assert(cvec.tensors[il] != nullptr);

        const size_t off = n_embd * (il - 1); // buffer doesn't have data for layer 0, since it's never present
        if (off + n_embd <= len) {
            ggml_backend_tensor_set(cvec.tensors[il], data + off, 0, n_embd * ggml_element_size(cvec.tensors[il]));  // <<<--- HERE
        }
    }

So I don't think there is an easy hack to get this working and changing the meaning of the --control-vector-scaled option probably isn't a good idea now anyway.

If this has to be done in the same way at runtime, then you could get an idea of the extra overhead by just left-multiplying the identity matrix in the ggml_backend_tensor_set(() code above for the relevant tensors (the original paper applied it to both o_proj and down_proj).

[brainlag]

Would it be possible (in theory) to apply and remove the control vector in the middle of the chat response?