LLAMA 3

[tiger2014]

Hi Team,

LLAMA 3 can be used in this frame?

Thanks

[KirillianAmu]

it does function right now, but in my experience rather poorly, likely due to the weird prompt formatting that llama 3 has.

[adammikulis]

If llama3 is being overly verbose, add "<|eot_id|>" to the AntiPrompts. I've had good results with it so far and it seems to have more personality than Mistral Instruct v0.2.

[imxcstar]

If llama3 is being overly verbose, add "<|eot_id|>" to the AntiPrompts. I've had good results with it so far and it seems to have more personality than Mistral Instruct v0.2.

I added <|eot_id|>, but it seems to have no effect

[martindevans]

I'm not sure if our antiprompt detection will properly handle special tokens like that. I know there's special case handling for EOS in some places. That could be a good improvement to the antiprompt processing for someone to make.

[AsakusaRinne]

Hi, #708 has added an example of LLaMA3 chat session. I'll appreciate it if you would like to try it and report problem to us if any. To run it, please pull the latest code of master branch and run the example project. :)

[adammikulis]

Will PR #6920 from llama.cpp resolve this issue?

[AsakusaRinne]

@adammikulis I'll update the native library binaries following ggerganov/llama.cpp#6920. Besides you could also try the current master branch, which has already provided an example of LLaMA3.

[dlyz]

I think x == EOS should be replaced everywhere with a llama_token_is_eog(x) that should be true for all end-of-something tokens. Antiprompt for <|eot_id|> won't help cause special tokens are not rendered.

ggerganov/llama.cpp#6745 (comment)

[AsakusaRinne]

I think x == EOS should be replaced everywhere with a llama_token_is_eog(x) that should be true for all end-of-something tokens. Antiprompt for <|eot_id|> won't help cause special tokens are not rendered.

That's right, we'll add such things soon. For now you could use "�" as anti-prompt as a temporary resolution.

[martindevans]

I think x == EOS should be replaced everywhere with a llama_token_is_eog(x)

This has actually been done in PR #712 (along with updating the binaries).

[martindevans]

llama3 has been supported for a while, so I'll close this issue now.