Embedding parameters #7458

YannFollet · 2024-05-22T09:55:13Z

extra parameters

--embd-normalize $integer$

$integer$	description	formula
$-1$	none
$0$	max absolute int16	$\Large{{32760 * x_i} \over\max \lvert x_i\rvert}$
$1$	taxicab	$\Large{x_i \over\sum \lvert x_i\rvert}$
$2$	euclidean (default)	$\Large{x_i \over\sqrt{\sum x_i^2}}$
$>2$	p-norm	$\Large{x_i \over\sqrt[p]{\sum \lvert x_i\rvert^p}}$

--embd-output-format $'string'$

$'string'$	description
''	same as before	(default)
'array'	single embeddings	$[[x_1,...,x_n]]$
	multiple embeddings	$[_0[x_1,...,x_n],1[x_1,...,x_n],...,{n-1}[x_1,...,x_n]]$
'json'	openai style
'json+'	add cosine similarity matrix

--embd-separator $"string"$

$"string"$
"\n"	(default)
"<#embSep#>"	for exemple
"<#sep#>"	other exemple

examples

Unix-based systems (Linux, macOS, etc.):

./embedding -p 'Castle<#sep#>Stronghold<#sep#>Dog<#sep#>Cat' --embd-separator '<#sep#>' --embd-normalize 2  --embd-output-format '' -m './path/to/model.gguf' --n-gpu-layers 99 --log-disable 2>/dev/null

Windows:

embedding.exe -p 'Castle<#sep#>Stronghold<#sep#>Dog<#sep#>Cat' --embd-separator '<#sep#>' --embd-normalize 2  --embd-output-format '' -m './path/to/model.gguf' --n-gpu-layers 99 --log-disable 2>/dev/null

--embd-normalize --embd-output-format --embd-separator description in the README.md

fix tipo

mofosyne · 2024-05-28T10:42:01Z

@YannFollet is this PR all good? Have a quick lookover. Had to do some merging to keep this in sync.

Marking as merging soon and will revisit later to see if anyone want to still comment about it.

mofosyne · 2024-06-09T02:38:27Z

@YannFollet this PR has gone out of sync. Can you synchronize this with the main branch? It appears there is no other objection and GG is happy with it. But it's out of sync so cannot easily merge.

YannFollet · 2024-06-18T10:42:33Z

@mofosyne Sorry haven't seen your message, I have done the merge with master

ggerganov · 2024-06-21T05:42:50Z

examples/embedding/embedding.cpp


-    // print cosine similarity matrix
-    if (n_prompts > 1) {
+    if (params.embd_out=="") {


Suggested change

if (params.embd_out=="") {

if (params.embd_out.empty()) {

ggerganov · 2024-06-21T05:45:30Z

common/common.h

+    int32_t embd_normalize = 2;     // normalisation for embendings (-1=none, 0=max absolute, 1=taxicab, 2=euclidean, >2=p-norm)
+    std::string embd_out   = "";    // empty = default, "array" = [] or [[],[]...], "json" = openai style, "json+" = same "json" + cosine similarity matrix
+    std::string embd_sep   = "\n";  // separator of embendings


These group of parameters should be moved below in their own section, similar to what we do for other examples:

// embedding int32_t embd_normalize = 2; // normalisation for embedings (-1=none, 0=max absolute, 1=taxicab, 2=euclidean, >2=p-norm) std::string embd_out = ""; // empty = default, "array" = [] or [[],[]...], "json" = openai style, "json+" = same "json" + cosine similarity matrix std::string embd_sep = "\n"; // separator of embedings

ggerganov · 2024-06-21T05:46:37Z

common/common.cpp

+        CHECK_ARG
+        params.embd_sep = argv[i];
+        return true;
+    }


Expand gpt_params_print_usage for these parameters

ggerganov · 2024-06-21T05:46:50Z

common/common.h

@@ -377,7 +380,7 @@ void llama_kv_cache_dump_view_seqs(const llama_kv_cache_view & view, int row_siz
 // Embedding utils
 //

-void llama_embd_normalize(const float * inp, float * out, int n);
+void llama_embd_normalize(const float * inp, float * out, int n, int embd_norm  = 2);


Suggested change

void llama_embd_normalize(const float * inp, float * out, int n, int embd_norm = 2);

void llama_embd_normalize(const float * inp, float * out, int n, int embd_norm = 2);

ggerganov · 2024-06-21T05:47:40Z

examples/embedding/embedding.cpp

+                if (params.embd_normalize==0)
+                    fprintf(stdout, "%6.0f ", emb[j * n_embd + i]);
+                else
+                    fprintf(stdout, "%9.6f ", emb[j * n_embd + i]);


Suggested change

if (params.embd_normalize==0)

fprintf(stdout, "%6.0f ", emb[j * n_embd + i]);

else

fprintf(stdout, "%9.6f ", emb[j * n_embd + i]);

if (params.embd_normalize == 0) {

fprintf(stdout, "%6.0f ", emb[j * n_embd + i]);

} else {

fprintf(stdout, "%9.6f ", emb[j * n_embd + i]);

}

ggerganov · 2024-06-21T05:47:59Z

examples/embedding/embedding.cpp

+                    float sim = llama_embd_similarity_cos(emb + i * n_embd, emb + j * n_embd, n_embd);
+                    fprintf(stdout, "%6.2f ", sim);
+                }
+                fprintf(stdout, "%1.10s",prompts[i].c_str());


Suggested change

fprintf(stdout, "%1.10s",prompts[i].c_str());

fprintf(stdout, "%1.10s", prompts[i].c_str());

ggerganov · 2024-06-21T05:49:41Z

examples/embedding/embedding.cpp

+    if (params.embd_out=="json" || params.embd_out=="json+" || params.embd_out=="array") {
+        const bool notArray = params.embd_out!="array";
+
+        fprintf(stdout, notArray?"{\n  \"object\": \"list\",\n  \"data\": [\n":"[");
+        for (int j = 0;;) { // at least one iteration (one prompt)
+            if (notArray) fprintf(stdout, "    {\n      \"object\": \"embedding\",\n      \"index\": %d,\n      \"embedding\": ",j);
+            fprintf(stdout, "[");
+            for (int i = 0;;) { // at least one iteration (n_embd > 0)
+                fprintf(stdout, params.embd_normalize==0?"%1.0f":"%1.7f", emb[j * n_embd + i]);
+                i++;
+                if (i < n_embd) fprintf(stdout, ","); else break;
+            }
+            fprintf(stdout, notArray?"]\n    }":"]");
+            j++;
+            if (j < n_prompts) fprintf(stdout, notArray?",\n":","); else break;
        }
+        fprintf(stdout, notArray?"\n  ]":"]\n");
+
+        if (params.embd_out=="json+" && n_prompts > 1) {
+            fprintf(stdout, ",\n  \"cosineSimilarity\": [\n");
+            for (int i = 0;;) { // at least two iteration (n_prompts > 1)
+                fprintf(stdout, "    [");
+                for (int j = 0;;) { // at least two iteration (n_prompts > 1)
+                    float sim = llama_embd_similarity_cos(emb + i * n_embd, emb + j * n_embd, n_embd);
+                    fprintf(stdout, "%6.2f", sim);
+                    j++;
+                    if (j < n_prompts) fprintf(stdout, ", "); else break;
+                }
+                fprintf(stdout, " ]");
+                i++;
+                if (i < n_prompts) fprintf(stdout, ",\n"); else break;
+            }
+            fprintf(stdout, "\n  ]");
+        }
+
+        if (notArray) fprintf(stdout, "\n}\n");


Suggested change

if (params.embd_out=="json" || params.embd_out=="json+" || params.embd_out=="array") {

const bool notArray = params.embd_out!="array";

fprintf(stdout, notArray?"{\n \"object\": \"list\",\n \"data\": [\n":"[");

for (int j = 0;;) { // at least one iteration (one prompt)

if (notArray) fprintf(stdout, " {\n \"object\": \"embedding\",\n \"index\": %d,\n \"embedding\": ",j);

fprintf(stdout, "[");

for (int i = 0;;) { // at least one iteration (n_embd > 0)

fprintf(stdout, params.embd_normalize==0?"%1.0f":"%1.7f", emb[j * n_embd + i]);

i++;

if (i < n_embd) fprintf(stdout, ","); else break;

}

fprintf(stdout, notArray?"]\n }":"]");

j++;

if (j < n_prompts) fprintf(stdout, notArray?",\n":","); else break;

}

fprintf(stdout, notArray?"\n ]":"]\n");

if (params.embd_out=="json+" && n_prompts > 1) {

fprintf(stdout, ",\n \"cosineSimilarity\": [\n");

for (int i = 0;;) { // at least two iteration (n_prompts > 1)

fprintf(stdout, " [");

for (int j = 0;;) { // at least two iteration (n_prompts > 1)

float sim = llama_embd_similarity_cos(emb + i * n_embd, emb + j * n_embd, n_embd);

fprintf(stdout, "%6.2f", sim);

j++;

if (j < n_prompts) fprintf(stdout, ", "); else break;

}

fprintf(stdout, " ]");

i++;

if (i < n_prompts) fprintf(stdout, ",\n"); else break;

}

fprintf(stdout, "\n ]");

}

if (notArray) fprintf(stdout, "\n}\n");

if (params.embd_out == "json" || params.embd_out == "json+" || params.embd_out == "array") {

const bool notArray = params.embd_out != "array";

fprintf(stdout, notArray ? "{\n \"object\": \"list\",\n \"data\": [\n":"[");

for (int j = 0;;) { // at least one iteration (one prompt)

if (notArray) fprintf(stdout, " {\n \"object\": \"embedding\",\n \"index\": %d,\n \"embedding\": ",j);

fprintf(stdout, "[");

for (int i = 0;;) { // at least one iteration (n_embd > 0)

fprintf(stdout, params.embd_normalize == 0 ? "%1.0f" : "%1.7f", emb[j * n_embd + i]);

i++;

if (i < n_embd) fprintf(stdout, ","); else break;

}

fprintf(stdout, notArray ? "]\n }" : "]");

j++;

if (j < n_prompts) fprintf(stdout, notArray ? ",\n" : ","); else break;

}

fprintf(stdout, notArray ? "\n ]" : "]\n");

if (params.embd_out == "json+" && n_prompts > 1) {

fprintf(stdout, ",\n \"cosineSimilarity\": [\n");

for (int i = 0;;) { // at least two iteration (n_prompts > 1)

fprintf(stdout, " [");

for (int j = 0;;) { // at least two iteration (n_prompts > 1)

float sim = llama_embd_similarity_cos(emb + i * n_embd, emb + j * n_embd, n_embd);

fprintf(stdout, "%6.2f", sim);

j++;

if (j < n_prompts) fprintf(stdout, ", "); else break;

}

fprintf(stdout, " ]");

i++;

if (i < n_prompts) fprintf(stdout, ",\n"); else break;

}

fprintf(stdout, "\n ]");

}

if (notArray) fprintf(stdout, "\n}\n");

group of parameters // embedding print usage for embedding parameters

This reverts commit 646ef4a.

* add parameters for embeddings --embd-normalize --embd-output-format --embd-separator description in the README.md * Update README.md fix tipo * Trailing whitespace * fix json generation, use " not ' * fix merge master * fix code formating group of parameters // embedding print usage for embedding parameters --------- Co-authored-by: Brian <[email protected]>

YannFollet and others added 2 commits May 22, 2024 09:29

add parameters for embeddings

625bdb5

--embd-normalize --embd-output-format --embd-separator description in the README.md

Update README.md

749b803

fix tipo

github-actions bot added the examples label May 22, 2024

Trailing whitespace

6816522

ggerganov approved these changes May 22, 2024

View reviewed changes

mofosyne added Review Complexity : Medium Generally require more time to grok but manageable by beginner to medium expertise level embeddings embedding related topics labels May 22, 2024

YannFollet and others added 2 commits May 23, 2024 01:42

fix json generation, use " not '

578a382

Merge branch 'master' into embedding-parameters

590720f

mofosyne added the merge ready indicates that this may be ready to merge soon and is just holding out in case of objections label May 28, 2024

YannFollet added 2 commits June 18, 2024 10:29

merge master

3fc2a81

fix merge master

3b1ae2c

ggerganov reviewed Jun 21, 2024

View reviewed changes

fix code formating

25fc226

group of parameters // embedding print usage for embedding parameters

ggerganov merged commit 646ef4a into ggerganov:master Jun 24, 2024
64 checks passed

Nexesenex added a commit to Nexesenex/croco.cpp that referenced this pull request Jun 25, 2024

Revert "embedding : more cli arguments (ggerganov#7458)"

0fa76c1

This reverts commit 646ef4a.

YannFollet deleted the embedding-parameters branch December 10, 2024 01:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Embedding parameters #7458

Embedding parameters #7458

YannFollet commented May 22, 2024

mofosyne commented May 28, 2024

mofosyne commented Jun 9, 2024

YannFollet commented Jun 18, 2024 •

edited

Loading

ggerganov Jun 21, 2024

ggerganov Jun 21, 2024

ggerganov Jun 21, 2024

ggerganov Jun 21, 2024

ggerganov Jun 21, 2024

ggerganov Jun 21, 2024

ggerganov Jun 21, 2024

	void llama_embd_normalize(const float * inp, float * out, int n, int embd_norm = 2);
	void llama_embd_normalize(const float * inp, float * out, int n, int embd_norm = 2);

	fprintf(stdout, "%1.10s",prompts[i].c_str());
	fprintf(stdout, "%1.10s", prompts[i].c_str());

Embedding parameters #7458

Embedding parameters #7458

Conversation

YannFollet commented May 22, 2024

extra parameters

--embd-normalize $integer$

--embd-output-format $'string'$

--embd-separator $"string"$

examples

Unix-based systems (Linux, macOS, etc.):

Windows:

mofosyne commented May 28, 2024

mofosyne commented Jun 9, 2024

YannFollet commented Jun 18, 2024 • edited Loading

ggerganov Jun 21, 2024

Choose a reason for hiding this comment

ggerganov Jun 21, 2024

Choose a reason for hiding this comment

ggerganov Jun 21, 2024

Choose a reason for hiding this comment

ggerganov Jun 21, 2024

Choose a reason for hiding this comment

ggerganov Jun 21, 2024

Choose a reason for hiding this comment

ggerganov Jun 21, 2024

Choose a reason for hiding this comment

ggerganov Jun 21, 2024

Choose a reason for hiding this comment

YannFollet commented Jun 18, 2024 •

edited

Loading