Creating endpoint with CRUD for documents and chunks #97

bafaurazan · 2024-02-15T00:57:42Z

I added file views.py that have declared types of response for client's CRUD request like( GET,POST,PUT,DEL) for chunks and documents, other files exclude .env.example have declared model schemas, url and routing. I think document's response for CRUD is incorect, because i didn't use foreignkey in document's schemas.

Code based on this site: https://medium.com/@hbakri/introducing-django-ninja-crud-ee937bd2ea65

pgronkievitz · 2024-02-15T07:31:22Z

@bafaurazan next time - please remember commits are formatted like this

title

some description, remember the empty space above

please, fix the commit message, here's how: https://stackoverflow.com/a/41987851

api/documents/views.py

api/documents/schemas.py

api/documents/api.py

pgronkievitz · 2024-02-15T07:43:03Z

P.S. when you're ready, remember to select reviewers - I've seen this PR only because I watch all the events in the repo

I added file views.py that have declared types of response for client's CRUD request like( GET,POST,PUT,DEL) for chunks and documents, other files exclude .env.example have declared model shemas, url and routing. I think document's response for CRUD is incorect, because i didn't use foreignkey.

api/documents/views.py

I additionally renamed folder containing models for code readability from documents into MyModels

In new version of PUT method, client can update objects declaring in request only variables that he want change (no all variables needed).

pgronkievitz

will CR this properly on saturday/sunday, currently naming is kinda wonky & I think chunks and documents should be separate "django apps"

I separated models creating similar chunks folder to documents folder and connect them using routers in api.api file

pgronkievitz · 2024-02-24T08:06:23Z

api/Dockerfile

-RUN manage.py collectstatic
-


question: why?

i get error message because of this:

i'm propably wrong but in my opinion command should looks like: " RUN python manage.py collectstatic", but then i get another error, because (i think) i can't get data contained in .env file

so i cant currently use this command and temporarily deleted them

if you can, correct me if i'am wrong

it doesn't work, because

you didn't change workdir in the target stage

you have to use relative path, unlike on windows, linux treats SOME_COMMAND as thing present in PATH and current working directory IS NOT automatically added to path, hence you have to use ./manage.py instead

api/api/api.py

pgronkievitz · 2024-02-24T08:09:15Z

api/chunks/models.py

+    text = models.CharField(max_length=100)
+    embedding = VectorField(dimensions=10)
+    chunk_idx = models.IntegerField()
+    start_char = models.IntegerField()
+    end_char = models.IntegerField()


issue: please add typing, it looks something like this

text = models.CharField[str, str](max_length=100)

note that you have to add type twice, because it's input/output

api/chunks/schemas.py

api/chunks/views.py

api/documents/views.py

In this commit i add: - annotations (i don't know if this is correct) - simple tests (i know that i can't do tests on main database, but i don't know how to do it work with test database (how to add extension with pgvector to test database)) P.S. I type following command to run tests: docker compose exec api python manage.py test --keepdb - i create files: documents/controllers.py and chunks/controllers.py and defined logic in them that is used by views.py.

pgronkievitz

I didn't duplicate issues, so search the rest of the code for similar cases when resolving

pgronkievitz · 2024-02-29T23:17:11Z

api/api/settings.py

+        "TEST": {
+            "NAME": env("POSTGRES_DB"),
+        },
+    },


question: What is this for?

without declaring this lines, tests run by default on the test database: "test_postgres", where pgvector extension is missing. I don't know how to fix this, so i temporarily run tests in the same database (i know this is incorrect), where extension is available.

pgronkievitz · 2024-02-29T23:19:17Z

api/chunks/models.py

+    text: Tuple[str, str] = models.CharField(max_length=100)
+    embedding: Tuple[list[float], VectorField] = VectorField(dimensions=10)
+    chunk_idx: Tuple[int, int] = models.IntegerField()
+    start_char: Tuple[int, int] = models.IntegerField()
+    end_char: Tuple[int, int] = models.IntegerField()


issue: This is not correct, see my previous comment. It may help you to set pyright to strict

but your suggestion, didn't work for models, that's why i changed it. The only way i can see is somethinkg like this text: str = models.CharField(max_length=100)

api/chunks/tests.py

api/documents/controllers.py

pgronkievitz · 2024-02-29T23:40:27Z

api/documents/models.py

 class Document(models.Model):
-    text = models.TextField()
-    embedding = VectorField(dimensions=10)
-    chunks = models.ForeignKey(Chunk, on_delete=models.CASCADE, null=True, blank=True)
+    text: Tuple[str, str] = models.TextField()
+    embedding: Tuple[list[float], VectorField] = VectorField(dimensions=10)
+    chunks: Tuple[int, int] = models.ForeignKey(Chunk, on_delete=models.CASCADE, null=True, blank=True)


Incorrect typing

api/documents/views.py

…int_for_documents_and_chunks_CRUD

bafaurazan · 2024-03-10T19:05:36Z

and Patryk if you could send me some links, that could help me solve the problems with the code, it would help me a lot

bafaurazan · 2024-03-10T22:06:52Z

should I add pyrightconfig.json file to the project to define how pyright should work?

api/chunks/controllers.py

api/chunks/tests.py

api/documents/tests.py

api/chunks/tests.py

api/documents/tests.py

api/chunks/tests.py

api/documents/tests.py

Co-authored-by: Patryk Gronkiewicz <[email protected]>

pgronkievitz requested changes Feb 15, 2024

View reviewed changes

bafaurazan force-pushed the Endpoint_for_documents_and_chunks_CRUD branch from 4a277eb to 3ed9e55 Compare February 15, 2024 15:00

bafaurazan force-pushed the Endpoint_for_documents_and_chunks_CRUD branch from 3ed9e55 to dd41b75 Compare February 15, 2024 15:01

added second router and small changes

91884fb

pgronkievitz reviewed Feb 15, 2024

View reviewed changes

api/documents/views.py Show resolved Hide resolved

bafaurazan added 2 commits February 18, 2024 23:37

separating documents and chunks

07986bb

I additionally renamed folder containing models for code readability from documents into MyModels

changing document get type

d09b09e

bafaurazan requested a review from TheJimmyNowak February 18, 2024 22:47

PUT method changes and minor adjustments

f2f95cd

In new version of PUT method, client can update objects declaring in request only variables that he want change (no all variables needed).

bafaurazan requested a review from pgronkievitz February 19, 2024 13:30

pgronkievitz requested changes Feb 21, 2024

View reviewed changes

bafaurazan and others added 4 commits February 23, 2024 21:10

Merge branch 'main' into Endpoint_for_documents_and_chunks_CRUD

16e226f

separating models into apps

0ac6cdb

I separated models creating similar chunks folder to documents folder and connect them using routers in api.api file

something to update

8298ee6

bug to fix

b34e50f

pgronkievitz requested changes Feb 24, 2024

View reviewed changes

bafaurazan requested review from pgronkievitz and removed request for TheJimmyNowak February 29, 2024 22:19

pgronkievitz requested changes Feb 29, 2024

View reviewed changes

bafaurazan added 5 commits March 4, 2024 13:47

lint and small changes

92818a7

Merge branch 'main' of https://github.com/knmlprz/ChatKNML into Endpo…

1850b46

…int_for_documents_and_chunks_CRUD

changing put controller

765f8ec

deleting test's environment

22afbde

fix api dockerfile

b64f6b4

bafaurazan requested a review from pgronkievitz March 10, 2024 19:02

changing annotations

5807907

changing environment variable get type

1f2e35c

pgronkievitz approved these changes Mar 13, 2024

View reviewed changes

bafaurazan and others added 5 commits March 13, 2024 19:46

Apply suggestions from code review

df2309e

Co-authored-by: Patryk Gronkiewicz <[email protected]>

Merge branch 'main' into Endpoint_for_documents_and_chunks_CRUD

f2c7fd6

fix

7638db8

fix2

59f3866

fix3

6eca621

bafaurazan merged commit 45af074 into main Mar 13, 2024
3 checks passed

bafaurazan deleted the Endpoint_for_documents_and_chunks_CRUD branch March 13, 2024 19:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Creating endpoint with CRUD for documents and chunks #97

Creating endpoint with CRUD for documents and chunks #97

bafaurazan commented Feb 15, 2024 •

edited

Loading

pgronkievitz commented Feb 15, 2024

pgronkievitz commented Feb 15, 2024

pgronkievitz left a comment

pgronkievitz Feb 24, 2024

bafaurazan Feb 24, 2024

pgronkievitz Feb 25, 2024

pgronkievitz Feb 24, 2024

pgronkievitz left a comment

pgronkievitz Feb 29, 2024

bafaurazan Mar 1, 2024

pgronkievitz Feb 29, 2024

bafaurazan Mar 1, 2024 •

edited

Loading

pgronkievitz Feb 29, 2024

bafaurazan commented Mar 10, 2024

bafaurazan commented Mar 10, 2024

Creating endpoint with CRUD for documents and chunks #97

Creating endpoint with CRUD for documents and chunks #97

Conversation

bafaurazan commented Feb 15, 2024 • edited Loading

pgronkievitz commented Feb 15, 2024

pgronkievitz commented Feb 15, 2024

pgronkievitz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pgronkievitz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bafaurazan Mar 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bafaurazan commented Mar 10, 2024

bafaurazan commented Mar 10, 2024

bafaurazan commented Feb 15, 2024 •

edited

Loading

bafaurazan Mar 1, 2024 •

edited

Loading