How big are the models?

[MichelDiz]

Does anyone know what the disk size is? I'm downloading the model 7B and it's already at 12GB. I think I'll stop, to migrate to another disk because mine(used just for docker tests) is full.

[j-f1]

consolidated.00.pth for 7B is 13.48GB on my machine. (The final quantized model is 4.21GB)

[MichelDiz]

What do you mean by quantized? I haven't had time to read the whole process yet. So, "quantize" is probably an extra step in the process? a model build? Thank you!

[j-f1]

Yes. The model files Facebook provides use 16-bit floating point numbers to represent the weights of the model. Research has shown that while this level of detail is useful for training models, for inference yo can significantly decrease the amount of information without compromising quality too much. llama.cpp “quantizes” the models by converting all of the 16-bit floating point numbers to 4-bit integers. This significantly reduces memory and disk space usage (as you can see), and also increases computation speed.

[MichelDiz]

Nice! thank you!

[ShouNichi]

219GB total.
7B 12.5
13B 24.2
30B 60.5
65B 121

[MichelDiz]

qq, do I need all of them or just the 65B is enought?
btw, when I'm interacting with it, can I export it as part of training? I mean. If I interact, does it learn something from my inputs?
Does it remember the last interactions?

[j-f1]

qq, do I need all of them or just the 65B is enought?

I would recommend starting with the 7B or 13B model — those run best on consumer devices. I personally haven‘t tried running the 65B model at all, and the 30B one was painfully slow.

btw, when I'm interacting with it, can I export it as part of training? I mean. If I interact, does it learn something from my inputs?
Does it remember the last interactions?

llama.cpp does not support any kind of learning. The model will of course use the information in its context to inform future responses, but there is not any sort of permanent memory, and any time you close and restart the program it will forget everything it may have learned.

[linonetwo]

Is there a torrent link for downloading the final quantized model to save calculation time?

[linonetwo]

https://github.com/ggerganov/llama.cpp/discussions/277 here we go

[anpin]

meta-llama/llama@56de950

[william8000]

Here is a directory listing after removing the 65B model and quantizing some of the other models.
My laptop has 64 GB RAM (and an nvidia gpu with unfortunately only 4 GB VRAM, so it can't load the torch gpu version).
My laptop can run the 13B model unquantized 13B/ggml-model-f16.bin and the 30B model quantized 30B/ggml-model-q4_0.bin
$ du --hum --sum --tot *B
57G 13B
141G 30B
30G 7B
226G total
$ ls -lR
.:
total 512
drwxrwxrwx 2 william william 4096 Mar 16 23:02 13B
drwxrwxrwx 2 william william 4096 Mar 16 23:35 30B
drwxrwxrwx 2 william william 4096 Mar 16 22:48 7B
-rwxrwxrwx 1 william william 2127 Mar 16 00:20 getllama.sh
-rw-rw-rw- 1 william william 499723 Mar 5 10:36 tokenizer.model
-rw-rw-rw- 1 william william 50 Mar 5 10:34 tokenizer_checklist.chk

./13B:
total 58795116
-rw-rw-rw- 1 william william 154 Mar 5 10:36 checklist.chk
-rw-rw-rw- 1 william william 13016334699 Mar 5 10:54 consolidated.00.pth
-rw-rw-rw- 1 william william 13016334699 Mar 5 10:42 consolidated.01.pth
-rw-r--r-- 1 william william 13017430697 Mar 16 22:59 ggml-model-f16.bin
-rw-r--r-- 1 william william 13017430697 Mar 16 22:59 ggml-model-f16.bin.1
-rw-r--r-- 1 william william 4069309097 Mar 16 23:02 ggml-model-q4_0.bin
-rw-r--r-- 1 william william 4069309097 Mar 16 23:03 ggml-model-q4_0.bin.1
-rw-rw-rw- 1 william william 101 Mar 5 10:36 params.json

./30B:
total 146951120
-rw-rw-rw- 1 william william 262 Mar 5 10:36 checklist.chk
-rw-rw-rw- 1 william william 16265763099 Mar 5 10:52 consolidated.00.pth
-rw-rw-rw- 1 william william 16265763099 Mar 5 10:52 consolidated.01.pth
-rw-rw-rw- 1 william william 16265763099 Mar 5 10:53 consolidated.02.pth
-rw-rw-rw- 1 william william 16265763099 Mar 5 10:52 consolidated.03.pth
-rw-r--r-- 1 william william 16267621641 Mar 16 23:29 ggml-model-f16.bin
-rw-r--r-- 1 william william 16267621641 Mar 16 23:29 ggml-model-f16.bin.1
-rw-r--r-- 1 william william 16267621641 Mar 16 23:30 ggml-model-f16.bin.2
-rw-r--r-- 1 william william 16267621641 Mar 16 23:30 ggml-model-f16.bin.3
-rw-r--r-- 1 william william 5086074121 Mar 16 23:33 ggml-model-q4_0.bin
-rw-r--r-- 1 william william 5086074121 Mar 16 23:34 ggml-model-q4_0.bin.1
-rw-r--r-- 1 william william 5086074121 Mar 16 23:35 ggml-model-q4_0.bin.2
-rw-r--r-- 1 william william 5086074121 Mar 16 23:36 ggml-model-q4_0.bin.3
-rw-rw-rw- 1 william william 101 Mar 5 10:52 params.json

./7B:
total 30436888
-rw-rw-rw- 1 william william 100 Mar 5 10:36 checklist.chk
-rw-rw-rw- 1 william william 13476939516 Mar 5 10:42 consolidated.00.pth
-rw-r--r-- 1 william william 13477682409 Mar 16 22:47 ggml-model-f16.bin
-rw-r--r-- 1 william william 4212727017 Mar 16 22:49 ggml-model-q4_0.bin
-rw-rw-rw- 1 william william 101 Mar 5 10:36 params.json