WISE推理问题

[xchenriang]

您好，我在使用WISE方法对Qwen2.5-7B进行知识编辑后，在推理时出现了一些问题，以下是我的代码与配置文件：

alg_name: "WISE"
model_name: "/home/models/qwen2.5"
device: 0, 1, 2, 3, 4, 5, 6

mask_ratio: 0.2
edit_lr: 0.08
n_iter: 20
norm_constraint: 1.0
act_margin: [5.0, 20.0, 10.0] # alpha, beta, gamma
act_ratio: 0.88
save_freq: 500
merge_freq: 1000
merge_alg: 'ties'
objective_optimization: 'only_label'
inner_params:
- model.layers[23].mlp.down_proj.weight


## alternative: WISE-Merge, WISE-Retrieve

# for merge (if merge)
densities: 0.53
weights: 1.0

# for retrieve (if retrieve, pls set to True)
retrieve: True
replay: False # True --> will replay the past editing instances: see https://arxiv.org/abs/2405.14768 Appendix B.3

model_parallel: True
use_chat_template: True

save_path: "/home/EasyEdit-main/wise_model/wise.pt"

import os
import json
from easyeditor import BaseEditor
from easyeditor import WISEHyperParams

prompts = ["我的姓名是", "我的年龄是", "我的身高是"]
rephrase_prompts = ["我叫什么?", "我多大岁数?", "我有多高?"]
target_new = ["王磊", "23岁", "188cm"]
locality_prompts = ["世界最古老的运河是哪个", "黄土高原和华北平原的分界线是什么", "青藏高原和塔里木盆地的分界线是什么"]
locality_ans = ["灵渠", "太行山脉", "昆仑山脉"]
loc_prompts = [p + " " + a for p, a in zip(locality_prompts, locality_ans)]

locality_inputs = {
    'neighborhood':{
        'prompt': locality_prompts,
        'ground_truth': locality_ans
    },
}
hparams = WISEHyperParams.from_hparams('./hparams/WISE/qwen2.5-7b.yaml')
editor = BaseEditor.from_hparams(hparams)
metrics, edited_model, _ = editor.edit(
    prompts=prompts,
    rephrase_prompts=rephrase_prompts,
    target_new=target_new,
    loc_prompts=loc_prompts,
    locality_inputs=locality_inputs,
    sequential_edit=True # or True
)

prompt = "我叫什么名字?"
inputs = editor.tok.encode(prompt, return_tensors='pt').to(edited_model.device)
outputs = edited_model.generate(inputs, max_new_tokens=16)
predict = editor.tok.decode(outputs[0], skip_special_tokens=True)
print("-"*20)
print("prompt is", prompt)
print("predict is", predict)

结果如下：

2024-12-26 21:01:44,541 - easyeditor.editors.editor - INFO - 0 editing: 我的姓名是 -> 王磊  

 {'pre': {'rewrite_acc': [0.0], 'portability': {}, 'rephrase_acc': [0.0]}, 'case_id': 0, 'requested_rewrite': {'prompt': '我的姓名是', 'target_new': '王磊', 'ground_truth': '<|endoftext|>', 'portability': {}, 'locality': {'neighborhood': {'prompt': '世界最古老的运河是哪个', 'ground_truth': '灵渠'}}, 'loc_prompt': '世界最古老的运河是哪个 灵渠', 'rephrase_prompt': '我叫什么?'}, 'post': {'rewrite_acc': [0.75], 'locality': {'neighborhood_acc': [1.0]}, 'portability': {}, 'rephrase_acc': [0.75]}}
12/26/2024 21:01:44 - INFO - easyeditor.editors.editor -   0 editing: 我的姓名是 -> 王磊  

 {'pre': {'rewrite_acc': [0.0], 'portability': {}, 'rephrase_acc': [0.0]}, 'case_id': 0, 'requested_rewrite': {'prompt': '我的姓名是', 'target_new': '王磊', 'ground_truth': '<|endoftext|>', 'portability': {}, 'locality': {'neighborhood': {'prompt': '世界最古老的运河是哪个', 'ground_truth': '灵渠'}}, 'loc_prompt': '世界最古老的运河是哪个 灵渠', 'rephrase_prompt': '我叫什么?'}, 'post': {'rewrite_acc': [0.75], 'locality': {'neighborhood_acc': [1.0]}, 'portability': {}, 'rephrase_acc': [0.75]}}
2024-12-26 21:01:44,909 - easyeditor.editors.editor - INFO - 1 editing: 我的年龄是 -> 23岁  

 {'pre': {'rewrite_acc': [0.0], 'portability': {}, 'rephrase_acc': [0.25]}, 'case_id': 1, 'requested_rewrite': {'prompt': '我的年龄是', 'target_new': '23岁', 'ground_truth': '<|endoftext|>', 'portability': {}, 'locality': {'neighborhood': {'prompt': '黄土高原和华北平原的分界线是什么', 'ground_truth': '太行山脉'}}, 'loc_prompt': '黄土高原和华北平原的分界线是什么 太行山脉', 'rephrase_prompt': '我多大岁数?'}, 'post': {'rewrite_acc': [1.0], 'locality': {'neighborhood_acc': [1.0]}, 'portability': {}, 'rephrase_acc': [0.25]}}
12/26/2024 21:01:44 - INFO - easyeditor.editors.editor -   1 editing: 我的年龄是 -> 23岁  

 {'pre': {'rewrite_acc': [0.0], 'portability': {}, 'rephrase_acc': [0.25]}, 'case_id': 1, 'requested_rewrite': {'prompt': '我的年龄是', 'target_new': '23岁', 'ground_truth': '<|endoftext|>', 'portability': {}, 'locality': {'neighborhood': {'prompt': '黄土高原和华北平原的分界线是什么', 'ground_truth': '太行山脉'}}, 'loc_prompt': '黄土高原和华北平原的分界线是什么 太行山脉', 'rephrase_prompt': '我多大岁数?'}, 'post': {'rewrite_acc': [1.0], 'locality': {'neighborhood_acc': [1.0]}, 'portability': {}, 'rephrase_acc': [0.25]}}
2024-12-26 21:01:45,281 - easyeditor.editors.editor - INFO - 2 editing: 我的身高是 -> 188cm  

 {'pre': {'rewrite_acc': [0.2], 'portability': {}, 'rephrase_acc': [0.2]}, 'case_id': 2, 'requested_rewrite': {'prompt': '我的身高是', 'target_new': '188cm', 'ground_truth': '<|endoftext|>', 'portability': {}, 'locality': {'neighborhood': {'prompt': '青藏高原和塔里木盆地的分界线是什么', 'ground_truth': '昆仑山脉'}}, 'loc_prompt': '青藏高原和塔里木盆地的分界线是什么 昆仑山脉', 'rephrase_prompt': '我有多高?'}, 'post': {'rewrite_acc': [1.0], 'locality': {'neighborhood_acc': [1.0]}, 'portability': {}, 'rephrase_acc': [1.0]}}
12/26/2024 21:01:45 - INFO - easyeditor.editors.editor -   2 editing: 我的身高是 -> 188cm  

 {'pre': {'rewrite_acc': [0.2], 'portability': {}, 'rephrase_acc': [0.2]}, 'case_id': 2, 'requested_rewrite': {'prompt': '我的身高是', 'target_new': '188cm', 'ground_truth': '<|endoftext|>', 'portability': {}, 'locality': {'neighborhood': {'prompt': '青藏高原和塔里木盆地的分界线是什么', 'ground_truth': '昆仑山脉'}}, 'loc_prompt': '青藏高原和塔里木盆地的分界线是什么 昆仑山脉', 'rephrase_prompt': '我有多高?'}, 'post': {'rewrite_acc': [1.0], 'locality': {'neighborhood_acc': [1.0]}, 'portability': {}, 'rephrase_acc': [1.0]}}
Metrics Summary:  {'pre': {'rewrite_acc': 0.06666666666666667, 'rephrase_acc': 0.15}, 'post': {'rewrite_acc': 0.9166666666666666, 'rephrase_acc': 0.6666666666666666, 'locality': {'neighborhood_acc': 1.0}}}
prompt is 我叫什么名字
predict is 我叫什么名字？我是一个由多个单词组成的字符串，每个单词之间用空格分

[pengzju]

evaluation是采用teacher forcing的形式，和generate还不太一样。这个问题你可以尝试下别的方法，结果应该都不太好。

我倾向于这并不是bug，而是当前模型编辑方法的缺陷

[pengzju]

这个问题WISE确实存在。如果你要测generate类似的指标，我建议可以使用类似bertscore（算生成文本和target_new的相似度），或者用一些简单的字符串匹配/gpt4打分等，也很期待您这边可以从这个视角再重新审视下当前所有的模型编辑方法

[zxlzr]

请问您还有其他问题吗？

[wpwpwpyo]

请问WISE目前没有generate方法，是否就在opencompass上测试不了？

[pengzju]

可以的吧，如果你是为了测通用能力，可以直接跑opencompass的