A Generation Error

[tomato996]

My model encountered a strange problem. After loading the model, the first few inferences could output content normally. However, after a few inferences, the model's inference time suddenly increased and the output of meaningless tokens was like this.

I think this is not the data problem , because sometimes the same problem can get a normal response, and sometimes it can't .

Here is my code to load and inference

ckpt_path = "/ceph_data/szy/internlm-xcomposer2d5-7B"
self.model = AutoModelForCausalLM.from_pretrained(ckpt_path, torch_dtype=torch.bfloat16, trust_remote_code=True).cuda()
self.tokenizer = AutoTokenizer.from_pretrained(ckpt_path, trust_remote_code=True)

with torch.autocast(device_type='cuda', dtype=torch.float16):
     response, _ = self.model.chat(self.tokenizer, query, image, do_sample=False, num_beams=3, use_meta=True)