CHANGELOG

v0.8.0 (2024-09-12)

Chore

chore: updating vid vec rep resenet packages (9f212c2)
chore: updating audio vec embedding pacakges (a41f996)
chore: updating audio vec embedding pacakges (6e50e06)
chore: fix conflict in updating requirements packages (acf1a56)
chore: updating requirements packages (5390073)
chore: fix ruff lint warnings (5f73898)
chore: add error handling to clustering and reduction operations (ba01287)
chore: hardcode perplexity value for t-SNE (60315b5)
chore: change operator name to be more verbose

Signed-off-by: Snehil Shah <snehilshah.989@gmail.com> (45bef72)

chore: remove obsolete print statement

Signed-off-by: Snehil Shah <snehilshah.989@gmail.com> (92d02e2)

chore: remove the file after the operation has ended

Signed-off-by: Snehil Shah <snehilshah.989@gmail.com> (4a384f5)

chore: removing docs gatsby website (03f5ac0)

Ci

ci: fix trivy github action (a5d4df8)
ci: update dependabot yml file (8a6b007)

Feature

feat: add config

Signed-off-by: Snehil Shah <snehilshah.989@gmail.com> (73731ec)

feat: add dimensionality reduction operator (2caf4a4)
feat: improve exception handling and logic

Co-authored-by: Chaithanya512 <chay5522kalyan@gmail.com> Signed-off-by: Snehil Shah <snehilshah.989@gmail.com> (5756a59)

feat: add cluster_embeddings operator

Signed-off-by: Snehil Shah <snehilshah.989@gmail.com> (4e08e35)

feat: add audio embedding operator using CLAP model (233cfd4)
feat: add classify_video operator

Signed-off-by: Snehil Shah <snehilshah.989@gmail.com> (2001dab)

feat: add vid_vec_rep_clip operator

Signed-off-by: Snehil Shah <snehilshah.989@gmail.com> (8874ce5)

Fix

fix: es_vec test (08b3085)
fix: enable worker support for dimension reductionn (89df059)
fix: update logic as per clustering spec

Signed-off-by: Snehil Shah <snehilshah.989@gmail.com> (0bea260)

fix: correct indentation logic (d208d17)
fix: correct naming and add dimension reduction requirements (0334a4b)

Refactor

refactor: migrate CLAP operator to Hugging Face Transformers (9e5195a)
refactor: align dimension reduction with feluda interface (0acfdb7)
refactor: change modality to video

Signed-off-by: Snehil Shah <snehilshah.989@gmail.com> (a1506ba)

Style

style: prepend newline for clearer output in logs

Signed-off-by: Snehil Shah <snehilshah.989@gmail.com> (bacafdd)

Test

test: add payload writer for the worker

Signed-off-by: Snehil Shah <snehilshah.989@gmail.com> (57803e0)

Unknown

Merge pull request #381 from tattle-made/development

merge dev to main (48bfc87)

Merge pull request #380 from aatmanvaidya/update-fel

Update Packages and Fix minor Issues (cf64f4d)

Merge pull request #379 from Snehil-Shah/worker

worker for clustering media (4abae4d)

nit: add logs

Signed-off-by: Snehil Shah <snehilshah.989@gmail.com> (1ba6de1)

draft: implement initial clustering worker (7129ac3)
init: add Dockerfile for worker

Signed-off-by: Snehil Shah <snehilshah.989@gmail.com> (777994b)

Merge pull request #375 from Snehil-Shah/clustering

[81] - add operator to cluster embeddings (1da1c27)

Merge pull request #376 from Chaithanya512/dim-reduction-operator

feat: add dimensionality reduction operator (cb76d85)

Merge pull request #372 from Chaithanya512/audio_vec_emb_clap

feat: audio embedding operator using CLAP model (faa9727)

Merge pull request #370 from Snehil-Shah/classify-video

[81] - add operator to classify videos using a zero-shot approach (91f604b)

Merge pull request #369 from Snehil-Shah/vid_vec_rep_clip

[81] - add operator to encode videos into vector embeddings using CLIP-ViT-B-32 (c5f09de)

Merge pull request #367 from aatmanvaidya/remove-docs

chore: removing docs gatsby website (2b9275a)

v0.7.2 (2024-05-28)

Fix

fix: Updated pr-security workflow fix: Removed ignored vulnerabilities that were fixed (fe9c26a)

Unknown

Merge pull request #344 from tattle-made/hotfix

Hotfix (10a277c)

Merge pull request #343 from duggalsu/update_pr_security_workflow

Update pr security workflow (b26a03c)

v0.7.1 (2024-05-28)

Fix

fix: Updated github action versions to latest (44b46ad)

Unknown

Merge pull request #342 from tattle-made/hotfix

Hotfix (eecc55f)

Merge pull request #341 from duggalsu/update_scorecard_workflow

Update scorecard workflow (b2e2aa8)

v0.7.0 (2024-05-28)

Chore

chore: update Dockerfile (8ca692e)
chore: removing file size limit from vid_vec_rep_resnet (22b9f2a)

Ci

ci: prod github action for media worker (d4bb47f)

Unknown

Merge pull request #339 from tattle-made/development

Merge Dev to Main (a1c22d3)

Merge pull request #340 from tattle-made/aatmanvaidya-patch-1

chore: update Dockerfile (eb018a8)

Merge pull request #332 from aatmanvaidya/media-worker-for-es

fix: updating media worker to index videos only (e6a8510)

Merge pull request #324 from Snehil-Shah/language-detection

feat: add operator to detect language in an audio file (acfa3a0)

Merge remote-tracking branch 'upstream/main' into media-worker-for-es (157508b)

v0.6.6 (2024-05-24)

Chore

chore: fixing ruff lint errors (b655e96)
chore: affix version & generate hashed requirements.txt (7305560)

Documentation

docs: update module doc (14de2fa)

Feature

feat: improve speech extraction's perfomance time (d105009)
feat: add operator for language detection in audio

Signed-off-by: Snehil Shah <snehilshah.989@gmail.com> (2744035)

Fix

fix: Updated vulnerable requests package (4b97299)
fix: updating media worker to index vidoes only (926044d)
fix: securely handle tmp for storage safety (8f3fa52)

Test

test: add case for speech extraction from heterogeneous audio (b448a5b)

Unknown

Merge pull request #334 from tattle-made/hotfix

Hotfix (b9f921a)

Merge pull request #333 from duggalsu/fix_dependabot_20240524

Fix dependabot 20240524 (e7704e1)

Regenerated src/requirements.txt (e25bd43)

v0.6.5 (2024-05-10)

Chore

chore: removing store from hash worker config (9f71faa)

Fix

fix: adding store check in hash worker (73157e1)

Unknown

Merge pull request #309 from tattle-made/hotfix

Hotfix (f21e2e6)

Merge pull request #308 from aatmanvaidya/hash-fix

fix: adding store check in hash worker (4af10d9)

Merge pull request #307 from tattle-made/hotfix

Hotfix (8723337)

Merge pull request #306 from aatmanvaidya/hash-config-hotfix

chore: removing store from hash worker config (75c4a78)

v0.6.4 (2024-05-08)

Fix

fix: Update packages to fix vulnerabilities (d222f9c)

Unknown

Merge pull request #305 from tattle-made/hotfix

Hotfix (2d1bb77)

Merge pull request #304 from duggalsu/fix_dependabot_20240507

Fix dependabot issues (97cc70d)

v0.6.3 (2024-05-07)

Fix

fix: config-server structure (831cf54)

Unknown

Merge pull request #303 from tattle-made/hotfix

Hotfix (ce0f0e1)

Merge pull request #302 from aatmanvaidya/config-fix

fix: config-server structure (ed1c62b)

v0.6.2 (2024-05-06)

Fix

fix: adding base requirements to media worker dockerfiles (bbddf70)

Unknown

Merge pull request #293 from tattle-made/hotfix

Hotfix (31abc9d)

Merge pull request #292 from aatmanvaidya/pinned-dep-fix

fix: adding base requirements to media worker dockerfiles (bd95984)

v0.6.1 (2024-05-06)

Fix

fix: tqdm vulnerable version update (841a748)

Unknown

Merge pull request #291 from tattle-made/hotfix

Hotfix (4ed81ab)

Merge pull request #290 from duggalsu/fix_dependabot_20240506

Fix dependabot issues (444a06f)

v0.6.0 (2024-05-01)

Chore

chore: fixing merge conflicts (4ce050e)

Unknown

Merge pull request #286 from tattle-made/development

chore: merging development to main (bba03e4)

Merge pull request #285 from aatmanvaidya/dev-2

chore: fixing merge conflicts (6239cef)

v0.5.4 (2024-04-30)

Chore

chore: sending ack on exception instead of nack (f6d9258)
chore: if statement checks for store in config inside media worker (435cf80)
chore: updating hash worker config (059d878)
chore: adding init files for hash and media worker (a508c5a)
chore: fixing ruff lint error (82f85a6)
chore: deleting empty query.py file in store (8cf33fc)
chore: fixing ruff lint error (8b920d5)
chore: deleting media test file (9ad98f7)

Ci

ci: media worker staging workflow (a4280f4)
ci: Added npm ci for pinned package installation

Added setup node version action (60c47e8)

ci: Added using ruff action for CI linting (246c6cd)
ci: Fix bandit to run from single workflow

Added bandit.yml to run on PR
Disabled bandit from pr-security.yml (692503e)

Fix

fix: Fixed dependabot issues

Updated idna package
Updated transformers package
Updated torch and torchvision for python 3.12 compatibility (d027dfe)

fix: hash worker relies more on core feluda (d8aed50)
fix: hash payload writer can send audio/video both (502ad4e)
fix: sending crc value to report queue (a89990e)
fix: making store component init more lean (3a63883)
fix: media worker relies more on core feluda (0389a3a)
fix: store can init all components properly (9b5a6d9)
fix: store init can start postgresql (d9d1cb1)
fix: feluda core supports amazom mq (91783af)
fix: Updated vulnerable pillow dependency in requirements (2c60f67)
fix: Add pinned dev requirements

Added dev requirements for feluda core, video and audio benchmark
Fixed video benchmark not working without new boto3 dependency
Fixed audio benchmark not working without new wget dependency (293d970)

fix: Added pinned pip package

Added base requirements.in with pip
Created base requirements.txt with pip pinned by hash
Recreated requirements.txt with --allow-unsafe to get correct setuptools
Updated feluda core dockerfile to install pip using base requirements
Updated ci test dockerfile to install pip using base requirements
Updated all benchmark dockerfiles to install pip using base requirements
Updated all worker dockerfiles to install pip using base requirements
Updated readme pip install and pip-compile commands (05e23b2)

fix: adding contextmanager for hash worker (312ab3f)
fix: audio factory supports s3 download (9834490)

Refactor

refactor: updating config structure for store (a2252f6)
refactor: s3 download to a new file (83ed5cf)

Unknown

Merge pull request #284 from tattle-made/hotfix

Hotfix (9d679c0)

Merge pull request #283 from duggalsu/fix_dependabot_20240430

Fixed dependabot issues (1319f89)

- Added ignore vuln until fix issued (9da6bc8)
Merge pull request #275 from aatmanvaidya/ack-fix

chore: sending ack on exception instead of nack (0b93759)

Merge pull request #261 from aatmanvaidya/hash-worker-fix

fix: hash worker relies more on core feluda (cbf1349)

Merge pull request #250 from aatmanvaidya/media-worker

feat: media worker (0d97313)

- Test inverted ternary operator (c7e3e05)
- Modify env settings (0e2a13b)
- Test switch conditional values (f05fbea)
- Set conditional falsy and truthy values (6a1e908)
- Fix env var boolean setting (f80d81e)
- Added conditional exit_zero for bandit (41036ad)

v0.5.3 (2024-04-04)

Chore

chore: amazon mq send_message lint fix (063096c)
chore: adding else condition of media_type (4e6e4eb)
chore: media_type is command line arg in payload writer (9d0fa4e)
chore: store check (2df0db6)

Feature

feat: media worker supports amazon mq (8b92b7f)
feat: Amazon MQ (766b805)
feat: media worker supports audio (b92a4d4)
feat: media worker supports video (ffba9cd)

Fix

fix: Updated vulnerable pillow dependency in requirements (efd1be8)
fix: setting up payload for audio (3a87ce0)
fix: postgres media worker table (dd974d2)
fix: media factory audio function update & setting up media worker (2e0fda5)

Refactor

refactor: config supports postgres & feat: setting up media payload writer (f43f205)

Unknown

Merge pull request #257 from tattle-made/hotfix

Hotfix (67ac03a)

Merge pull request #256 from duggalsu/update_pillow

Update pillow (44b0f88)

v0.5.2 (2024-03-23)

Ci

ci: Added npm ci for pinned package installation

Added setup node version action (fd4a8f1)

ci: Added using ruff action for CI linting (ed89995)

Fix

fix: Add pinned dev requirements

Added dev requirements for feluda core, video and audio benchmark
Fixed video benchmark not working without new boto3 dependency
Fixed audio benchmark not working without new wget dependency (d974e7e)

Unknown

Merge pull request #244 from tattle-made/hotfix

Hotfix (7bae56d)

Merge pull request #243 from duggalsu/add_dev_requirements

Add dev requirements (aa2c09f)

Merge pull request #242 from tattle-made/hotfix

Hotfix (efb06e5)

Merge pull request #241 from duggalsu/use_npm_pinned

Use npm pinned (52a4bbf)

Merge pull request #240 from duggalsu/use_ci_ruff_action

ci: Added using ruff action for CI linting (5d7137b)

v0.5.1 (2024-03-23)

Chore

chore: Added security policy (2628ddb)

Ci

ci: Fix bandit to run from single workflow

Added bandit.yml to run on PR
Disabled bandit from pr-security.yml (e441f2e)

ci: Added ci test for media file hash operator (e5f6414)

Fix

fix: Added pinned pip package

Added base requirements.in with pip
Created base requirements.txt with pip pinned by hash
Recreated requirements.txt with --allow-unsafe to get correct setuptools
Updated feluda core dockerfile to install pip using base requirements
Updated ci test dockerfile to install pip using base requirements
Updated all benchmark dockerfiles to install pip using base requirements
Updated all worker dockerfiles to install pip using base requirements
Updated readme pip install and pip-compile commands (810a45e)

Unknown

Merge pull request #239 from tattle-made/hotfix

Hotfix (3113756)

Merge pull request #238 from duggalsu/add_base_requirements

Add base requirements (0391f2a)

Merge pull request #237 from tattle-made/hotfix

Hotfix (83e2efd)

Merge pull request #236 from duggalsu/fix_bandit_codeql_warning

Fix bandit codeql warning (a7c05bb)

- Test inverted ternary operator (fe03cb5)
- Modify env settings (2d48897)
- Test switch conditional values (7f1315a)
- Set conditional falsy and truthy values (f85d164)
- Fix env var boolean setting (68923a6)
- Added conditional exit_zero for bandit (764b8bb)
Merge pull request #235 from tattle-made/hotfix

Hotfix (650a379)

Merge pull request #234 from duggalsu/add_ci_test_media_file_hash

Add ci test media file hash (88f1d30)

Merge pull request #233 from tattle-made/hotfix

Hotfix (13829a6)

Merge pull request #232 from duggalsu/add_security_policy

Add security policy (f0f6d70)

v0.5.0 (2024-03-21)

Chore

chore: adding context manager to audio operator (38f610f)
chore: updating table name (c015d30)

Ci

ci: Added PR checks to run on main branch (efc63e7)

Feature

feat: Added audio CRC write to postgres

Modified video worker column value (17ca1a7)

feat: Add video CRC to postgres

Created trigger function for updating timestamp
Modified create table function with conditional for table
Added create table for storing CRC
Added function to create trigger for table
Modified store and update functions to store worker_column
Added postgres init and call to storing video CRC
Pinned images in docker compose (f11e1c8)

feat: hash worker can add data to postgresql (eae1993)
feat: Added creation of CRC for video and audio vec for media matching (ca52993)
feat: Add calculating video vec CRC for media matching (9d57125)

Fix

fix(security): SQL issues

Replaced all dynamic SQL with prepared statements
Removed all f-strings
Added str conversion to hash function on db store as failsafe (8e86b2a)

fix(security): SQL issues

Conflicts:

src/core/store/postgresql.py (`3a5f003`)

fix: store and update postgres functions to support hash worker (1511228)
fix: postgres store and update func in hash worker (f7cb056)
fix: Update urllib3 package and fix feluda core dockerfile

Updated urllib3 manually for botocore compatibility
Updated groupadd, useradd and mkdir commands to handle failure
Added setting python user to debug image
Added venv volume to feluda api in docker compose file
Added build args for feluda indexer and reporter
Added volumes in docker compose file (317c2a0)

Refactor

refactor: commenting init func (bbdffbe)

Unknown

Merge pull request #230 from tattle-made/development

merge dev to main (ed75278)

Merge pull request #229 from duggalsu/test_sql

Fix SQL queries (81a99ed)

Merge pull request #228 from aatmanvaidya/hash-worker-update

fix: postgres store and update func in hash worker (27f6715)

Merge pull request #227 from aatmanvaidya/audio-delete

chore: adding context manager to audio operator (bd6202f)

Merge pull request #226 from duggalsu/add_crc_postgres_audio

feat: Added audio CRC write to postgres (ddc008a)

Merge pull request #225 from duggalsu/add_crc_postgres

feat: Add video CRC to postgres (6d173bb)

Merge pull request #224 from tattle-made/hotfix

Hotfix (34f3ed0)

Merge pull request #223 from duggalsu/add_pr_checks_main

Add pr checks main (aceb907)

Merge pull request #222 from duggalsu/update_urllib3_audio_video_requirements

fix: Update urllib3 package and fix feluda core dockerfile (e5e5f4e)

Merge pull request #221 from aatmanvaidya/postgres-hash

feat: hash worker can add data to postgresql (c34a6b3)

Merge pull request #220 from duggalsu/impl_media_crc

Impl media crc (f4018b0)

v0.4.10 (2024-03-18)

Fix

fix(ci): Fixed github workflow docker build args format (b6a3d14)

Unknown

Merge pull request #219 from tattle-made/hotfix

Hotfix (4b4a3ff)

Merge pull request #218 from duggalsu/fix_docker_build_args_format

Fix docker build args format (fe1adfd)

v0.4.9 (2024-03-18)

Fix

fix(ci): Added docker build args to github workflows (496d600)

Unknown

Merge pull request #217 from tattle-made/hotfix

Hotfix (c039f63)

Merge pull request #216 from duggalsu/fix_docker_build_args

Fix docker build args (ef9c159)

v0.4.8 (2024-03-18)

Fix

fix: Fixed volume sync and non-root user permission issues (5fd4258)

Unknown

Merge pull request #215 from tattle-made/hotfix

Hotfix (20ed58a)

Merge pull request #214 from duggalsu/fix_volume_sync

Fix volume sync (772d71d)

- Added UID and GID args to ci-sut docker compose (3bdb16a)

v0.4.7 (2024-03-18)

Ci

ci: remove intermediate cache deletion step (ade1e67)
ci: updated dockerfile base image platform (d0542bf)
ci: Add github cache action for local key-based caching (7b8ed07)
ci: Added dependabot config (8a57943)

Fix

fix: Remove github caching and modify base image to use TARGETPLATFORM (0dfb172)

Unknown

Merge pull request #213 from tattle-made/hotfix

Hotfix (4541139)

Merge pull request #212 from duggalsu/remove_github_caching

Remove GitHub caching (c490cea)

Merge pull request #211 from tattle-made/hotfix

Hotfix (cdb081a)

Merge pull request #210 from duggalsu/rm_intermediate_cache_del

Rm intermediate cache del (caae5a8)

Merge pull request #209 from tattle-made/hotfix

Hotfix (decd119)

Merge pull request #208 from duggalsu/modify_base_img_platform

Modify base img platform (c48efa4)

Merge pull request #201 from tattle-made/hotfix

Hotfix (d91149a)

Merge pull request #200 from duggalsu/fix_github_caching

Fix GitHub caching (389f021)

Merge pull request #190 from tattle-made/hotfix

Hotfix (5d13eea)

Merge pull request #189 from duggalsu/add_dependabot_yml

Add dependabot yml (74945f2)

v0.4.6 (2024-03-17)

Ci

ci: removed require hashes to allow no-deps to work (e634622)
ci: added no deps and require hashes for pip audit (e5f110c)

Fix

fix: Security enhancements and performance optimizations

security: Removed exposed port in all workers
security: Pinned docker platform in all dockerfile images
security: Pinned docker images digest in all dockerfile images
security: Pinned python packages by hash digest in all dockerfile images
perf: Optimized workers and test image for smaller size and build times
perf: Enabled github docker cache for all workflows
docs: Updated readme with generate hash instructions for requirements (7b181af)

Unknown

Merge pull request #188 from tattle-made/hotfix

Hotfix (f92ef82)

Merge pull request #187 from duggalsu/opt_vid_worker

Opt vid worker (217d60d)

- setup python version (2d1863c)

v0.4.5 (2024-03-15)

Ci

ci: removed explict checkout in docker build (2068476)
ci: fix context in test workflow (201c900)
ci: Fix issues with docker push vidvec benchmark test workflow (1d4a31e)

Fix

fix: Fix workflows (c356dc9)

Unknown

Merge pull request #186 from tattle-made/hotfix

Hotfix (2c19559)

Merge pull request #185 from duggalsu/fix_docker_workflow

Fix docker workflow (acc4000)

Merge pull request #184 from tattle-made/hotfix

Hotfix (3d6c84f)

Merge pull request #183 from duggalsu/fix_vidvec_benchmark_test_workflow_2

Fix vidvec benchmark test workflow 2 (7d5cc01)

Merge pull request #182 from tattle-made/hotfix

Hotfix (f6d87f8)

Merge pull request #181 from duggalsu/fix_vidvec_benchmark_test_workflow

Fix vidvec benchmark test workflow (6f2a32a)

Merge pull request #180 from tattle-made/hotfix

Hotfix (0b30a78)

Merge pull request #179 from tattle-made/scorecard_workflow

Scorecard workflow (d1f468d)

Create scorecard.yml (7bf1e20)

v0.4.4 (2024-03-14)

Chore

chore: removed global import (d02366c)

Fix

fix: tempfile path for make from file in memory function (831a700)
fix: tempfile impl (2f9cc07)
fix: Removed use of tempfile downloads (eb97f09)
fix: Created tempfile safely (badc043)

Unknown

Merge pull request #178 from tattle-made/hotfix

Hotfix (a85423a)

Merge pull request #164 from duggalsu/fix_tempfile_issues

Fix tempfile issues (c846df5)

Merge remote-tracking branch 'refs/remotes/origin/fix_tempfile_issues' into fix_tempfile_issues

Conflicts:

src/core/models/media_factory.py (`36ec52b`)

Fix tempfile issues

Optimized dockerfile
Modified tempfile tests (d082d93)

- Fixed video and audio file return path (bc43d19)

v0.4.3 (2024-03-14)

Fix

fix: flask issues

Added config vars to env template
Disabled flask production docker image
Enabled flask dev server and debugpy to run on localhost
Added debug cmd flag as recommended approach
Modified flask app run to load host and debug values from env (cfc0601)

Unknown

Merge pull request #177 from tattle-made/hotfix

Hotfix (f2a2b3d)

Merge pull request #176 from duggalsu/fix_flask_issues

Fix flask issues (c68596f)

v0.4.2 (2024-03-13)

Fix

fix: wget issues (ff70726)

Unknown

Merge pull request #175 from tattle-made/hotfix

Hotfix (fc508fa)

Merge pull request #174 from duggalsu/fix_wget_issues

Fix wget issues (6b98b8f)

v0.4.1 (2024-03-13)

Fix

fix: Renamed hash worker graviton file ci: Added publishing media hash worker docker images ci: Added release hash worker dockerfiles (33fb2d8)

Unknown

Merge pull request #173 from tattle-made/hotfix

Hotfix (ec7b922)

Merge pull request #172 from duggalsu/add_hash_worker_workflow

Add hash worker workflow (b58c11f)

v0.4.0 (2024-03-13)

Ci

ci: updating hash worker github workflow (73e5634)

Fix

fix: subprocess call issue (058cee5)

Unknown

Merge pull request #165 from tattle-made/development

merge dev to main (55204f9)

Merge pull request #166 from aatmanvaidya/temp-dev-2

resolving merge conflicts (1dea270)

Merge remote-tracking branch 'upstream/main' into temp-dev-2 (3c28a9f)
Merge pull request #171 from duggalsu/fix_subprocess_call

Fix subprocess call (acf4974)

v0.3.4 (2024-03-13)

Chore

chore: fixing module imports (4097614)
chore: resolving merge conflicts (5e3da48)

Ci

ci: fixing hash worker dockerfiles (46173da)

Fix

fix: Fixed yaml load (0582882)
fix: Fixed assert issues (33f5fd2)
fix: linting issues (39d90be)
fix: Removed secrets (11f4186)
fix: Created tempfile safely (fc0d3a2)
fix: hash operator and worker (64d0797)

Unknown

Merge pull request #170 from tattle-made/hotfix

Hotfix (8f4e730)

Merge pull request #169 from duggalsu/fix_yaml_load

fix: Fixed yaml load (ab3851f)

Merge pull request #168 from duggalsu/fix_assert_issues

fix: Fixed assert issues (1290ef9)

Merge pull request #167 from duggalsu/fix_secrets

Fix secrets (3670bc9)

- Fixed video and audio file return path (a251873)
Merge pull request #163 from aatmanvaidya/hash-op

fix: hash operator and worker (d51aeb3)

v0.3.3 (2024-03-12)

Chore

chore: updated numpy version (a299171)

Ci

ci: Add bandit cron workflow to trigger on push to main for resolved issues (90864bd)
ci: Updated cron time to test trigger (170c313)
ci: Enabled github token and modified cron UTC time (89c773f)
ci: Updated path settings (b02ed5b)
ci: Fix bandit cron workflow (faf897f)
ci: Disabled bandit level (4802bea)
ci: Added bandit SAST scanning (5ecd5da)
ci: fixed scanners option (ae1ceb4)
ci: Added exit code with limit sarif severities and always upload results (b105ac5)
ci: Updated codeql-action version (03e48ea)
ci: Removed failure condition for trivy scan to allow sarif upload (7b17253)
ci: Added githbu codeql sarif upload permissions (1bcd94b)
ci: Enabled trivy result upload to github codeql (e9afff0)
ci: separate pip audit (e93e002)
ci: add pip audit security workflow (b7a4db2)

Fix

fix: (security) Added timeout to requests (f5c0645)

Unknown

Merge pull request #162 from tattle-made/hotfix

Hotfix (9c7d06b)

Merge pull request #161 from duggalsu/add_requests_timeout

Add requests timeout (805e426)

Merge pull request #160 from tattle-made/hotfix

Hotfix (76d900e)

Merge pull request #159 from duggalsu/add_on_push_bandit_cron_workflow

Add on push bandit cron workflow (bb97334)

Merge pull request #158 from tattle-made/hotfix

Hotfix (7b29e51)

Merge pull request #157 from duggalsu/test_bandit_cron_workflow_2

Test bandit cron workflow 2 (5ef5baf)

Merge pull request #156 from tattle-made/hotfix

Hotfix (dc387bc)

Merge pull request #155 from duggalsu/test_bandit_cron_workflow

Test bandit cron workflow (bba6af8)

Merge pull request #154 from tattle-made/hotfix

Hotfix (9fca369)

Merge pull request #153 from duggalsu/fix_bandit_cron_workflow

Fix bandit cron workflow (d4f7e01)

Merge pull request #152 from tattle-made/hotfix

Hotfix (0605f12)

Merge pull request #151 from tattle-made/bandit_cron_job

Bandit cron job (453093d)

Create bandit.yml (f9f5e2b)
Merge pull request #150 from tattle-made/hotfix

Hotfix (5355a37)

Merge pull request #149 from duggalsu/add_ci_bandit

Add ci bandit (c899ab4)

Merge pull request #148 from tattle-made/hotfix

Hotfix (1001e92)

Merge pull request #147 from duggalsu/add_ci_trivy

Add ci trivy (0f7e440)

ci (security): Added IaC scan with Trivy (ac487f9)
fix (security): Renamed graviton dockerfiles for detection by trivy (bc2b01c)
fix (security): Harden dockerfiles

Added unprivileged python user
Created venv for all pip install
Added chown for all files and dirs
Added --no-install-recommends to apt-get install cmd
Fixed feluda core server.py import issue (a760d2e)

Merge pull request #146 from tattle-made/hotfix

Hotfix (adf12d0)

Merge pull request #145 from duggalsu/add_ci_pip_audit

Add ci pip audit (40f3ff6)

v0.3.2 (2024-03-07)

Chore

chore: fixing docker-compose.yml (3fe5de6)

Ci

ci: github action to push md5hash worker to staging (b5df260)
ci: add security checks on PR (ca7de73)

Feature

feat: worker for md5hash operator (008cc8d)

Fix

fix: linter formatting (4758315)
fix: linter issues (11f3683)

Unknown

Merge pull request #144 from tattle-made/hotfix

Hotfix (dc64745)

Merge pull request #143 from aatmanvaidya/md5-ci

ci: github action to push md5hash worker to staging (c896d29)

Merge pull request #142 from aatmanvaidya/md5-hash-worker

feat: worker for md5hash operator (da40ef9)

Merge pull request #135 from duggalsu/test_official_docker_push_action

Test official docker push action (1dbe184)

Merge pull request #141 from duggalsu/fix_linter_issues

Fix linter issues (7dd3979)

v0.3.1 (2024-03-07)

Fix

fix: Dockerfile arm build (cce1ebe)

Unknown

Merge pull request #140 from tattle-made/hotfix

Hotfix (68469b1)

Merge pull request #139 from aatmanvaidya/fix-dockerfiles

fix: Dockerfile arm build (a031c5d)

v0.3.0 (2024-03-07)

Ci

ci: Test fix versioning and modify docker images (455b77c)
ci: Fix docker yml (d0a48f8)
ci: Fix yaml issue (2f47f19)
ci: Fix yaml issue (82bfeb4)
ci: Test fix versioning and modify docker images (ae54bcd)
ci: Add worker docker push (97abfb0)
ci: Fix docker yml (93f1e51)
ci: Fix yaml issue (a320c49)
ci: Fix yaml issue (148fd8d)

Feature

feat: workers to search audio and video files (2301c98)
feat: workers to search audio and video files (cd94344)
feat: workers can report to queue (3c59999)

Style

style: fixing logger (dc86b15)
style: fixing logger (933f6db)

Test

test: official docker build and push workflow (4ad6685)

Unknown

Merge pull request #138 from tattle-made/development

merge development to main (df59805)

Merge pull request #136 from aatmanvaidya/worker-search-2

feat: workers to search audio and video files (d9c0d1b)

Merge branch 'worker-search-2' of https://github.com/aatmanvaidya/feluda into worker-search-2 (5a55ea7)
Merge pull request #137 from aatmanvaidya/temp-main

rebasing development with main (29b36b7)

ci (fix): Modify pr testing

Removed integration tests and required components
Used unittest for unit tests
Disabled audio from disk test (828b1e4)

ci (fix): Increased es java mem limit (b4bdee4)
ci (fix): version output (16add46)
Merge pull request #134 from tattle-made/hotfix

Hotfix (96eab93)

Merge pull request #133 from duggalsu/fix_ci_merge

ci: Test fix versioning and modify docker images (09d6941)

Merge branch 'hotfix' into fix_ci_merge (5775824)
Merge pull request #132 from duggalsu/ci_merge_versioning

Ci merge versioning (a49c6f5)

Merge pull request #131 from aatmanvaidya/worker-report

feat: workers can report to queue (cf801f1)

Merge pull request #130 from tattle-made/hotfix

Hotfix (b74f0c7)

Merge pull request #129 from duggalsu/fix_ci_merge_versioning

Fix ci merge versioning (4903a49)

ci (fix): Modify pr testing

Removed integration tests and required components
Used unittest for unit tests
Disabled audio from disk test (0a596e0)

ci (fix): Increased es java mem limit (d39fed4)
ci (fix): version output (d3aa1e4)

v0.2.0 (2024-03-05)

Ci

ci: allocating more RAM to elasticsearch (c0e3f15)
ci: Delete workflow

Deleted merge dev hotfix workflow
Disabled conditional build on merge main workflow (1ba4481)

ci: Fix secrets access

Removed conditional on merge main workflow
Removed environment label from merge dev hotfix workflow
Modified docker secrets name (0881176)

ci: Fix merge workflows

Disabled conditional checks on merge main workflow
Disabled environment setting from merge dev hotfix (d094b78)

Refactor

refactor: merging development to main (5959f74)

Test

test: fixing video es vec (bf7dede)

Unknown

Merge pull request #128 from tattle-made/development

refactor: merge development to main (73cfa5d)

Merge pull request #127 from aatmanvaidya/temp-main

refactor: updating dev from main (b4dca72)

Merge remote-tracking branch 'upstream/development' into temp-main (d4415c6)
Merge pull request #125 from tattle-made/hotfix

Hotfix (06eb74e)

Merge pull request #124 from duggalsu/del_merge_dev_hotfix_workflow

ci: Delete workflow (0744842)

Merge remote-tracking branch 'upstream/hotfix' into del_merge_dev_hotfix_workflow

Conflicts:

.github/workflows/merge-dev-hotfix.yml (`3fb216c`)

Merge pull request #122 from duggalsu/fix_ci_merge_secrets_access

ci: Fix secrets access (969261c)

Merge pull request #121 from duggalsu/fix_ci_merge_workflow

Fix ci merge workflow (0e36a80)

Merge pull request #120 from aatmanvaidya/main-merge

refactor: merging development to main (22bb325)

v0.1.0 (2024-03-04)

Chore

chore: fixing spelling in Dockerfile (a3e2fbe)
chore: renaming video worker files (10b60b2)
chore: updating library to local folder location (daa91f9)
chore: renaming audio cnn model (cbdbac9)
chore: renaming worker files (6059355)
chore: renaming operator (191abd6)
chore: deleting docker-compose-standalone and renaming privacy policy (#89) (bf6fb1e)
chore: adding .env-template (#88) (c556f8b)
chore: skipping unmaintained tests (c48b724)
chore: removing video files (bce9759)
chore: add DMP issue template (63aa375)
chore: adding audio file (#73) (52d0fd6)
chore: not skipping image search test (#57) (9f42558)
chore: revert to commit 8479c38 (98a76de)
chore: testing ci (e909448)
chore: text vector debugging (5097840)
chore: change batchsize & num_workers to handle memory issue, improve exception logging (7cd003e)

Ci

ci: Add automated semantic versioning

Renamed and modified docker push vidvec benchmark yml
Modified tags and dockerfile name for vidvec worker staging yml
Added init python file with version number
Added github workflow on merge on main branch
Added github workflow on merge on development or hotfix branch (972b6a6)

ci: audiovec worker staging dockerhub tag change (a79620d)
ci: audiovec benchmark graviton dockerfile (81b02b5)
ci: updating vidvec gravition Dockerfile (b1402b5)
ci: removing duplicate run commands in Dockerfile (f967496)
ci: audio worker staging github workflow (eec380a)
ci: video worker dockerfile for graviton (8554f3b)
ci: workflow dockerfile location change (d0826b8)
ci: fix location for video requirment file (96e0b87)
ci: workflow to push video worker to docker hub (594fc6c)
ci: workflow to push video worker to docker hub (730b808)
ci: workflow to push video worker to docker hub (0d517c7)
ci: updating vid Dockerfiles (8c32315)
ci: python version change in video dockerfile (#66) (aafde1c)
ci: giving access to tests for video vec (#61) (5945004)

Documentation

docs: adding reference of audio cnn model (9d499be)
docs: adding comments to the config-indexer (7bcbfca)
docs: tests documentation for YOLO and Tesseract operators (#47) (8eefb63)
docs: YOLO segementation operator (#46) (1a3da78)
docs: documentation for tesseract ocr operator (37e3e30)
docs: add Usage Case Studies (31400a6)

Feature

feat: worker for audio operator (becf40c)
feat: md5 operator (9e2d1be)
feat: feluda store supports audio (#78)
feat: feluda store supports audio
fix: delete and refresh for ES
dhore: profiling audio operator (f6987a6)
feat: add poc multiprocess test (f43646b)
feat: audio operator to extract embedding vectors (#59)
feat: audio emebddings
chore: deleting music files
chore: renaming files
docs: documentation for audio embedding operator
docs: adding work to be done for the operator (484d5ae)
feat: c-profiling test for video vec (#60)
feat: c-profiling test for video vec
feat: test to find time taken for video vec (247f5db)
feat: add workflow to push vidvec specific operator to dockerhub (17e0d57)
feat: operator to detect objects using YOLO (#44)
feat: operator to detect objects using YOLO
test file comment main function
chore: moving ultralytics install to opreator (17b9d10)
feat: operator to extract text in images using tesseract (#40)
feat: opreator to detect text in images using tesseract
chore: adding test images and making test multilingual (edec4a9)
feat: add license (a44e233)
feat: update docs (19a9217)
feat: add NER, add text search, add Not Implemented http response for raw query

Added an operator to do Named Entity Recognition on non-english text
Created a test and handler to respond to text queries
Deferred the raw query search for later. It returns an HTTP response 501 now to denote that it hasnot been implemented (d57fb90)

feat: add pathways for /index depending on request content type (2f9f11f)
feat: wip workers ready. testing pending (84f9f33)
feat: indexing via worker in place (612593e)
feat: test end to end indexing for text, image and video (af6f144)
feat: create generator for video vectors and store it in es (0208259)
feat: index all 3 mediatypes via URL (bcb0237)
feat: remove stray files (55c39cf)

Fix

fix: updating vidvec benchmark scripts (9aa4a50)
fix: context manager handles vidoes (e8f36ce)
fix: model is downloaded from github release (0c142df)
fix: video operator deletes file (8f97a95)
fix: video operator locaiton in Dockerfile (6d0e25d)
fix: worker handles disconnection to MQ (b6a48f3)
fix: cnn models works when saved on local machine (c86623d)
fix: worker supports url and optimising on connection lost (80c122d)
fix: model fetched from local folder (ef512ef)
fix: audio vec test supports media factory (41be124)
fix: video vec test supports media factory (bf727fd)
fix: image vec test supports media factory (6e33270)
fix: Dockerfile location (a8ce31e)
fix: server operator setup (c8a5bf8)
fix: video es test (#87) (2670490)
fix: url media link for index api test (089412a)
fix: index and search tests (ffc8efc)
fix: search as client test (#83) (9855e96)
fix: graviton supported github runner (1ac59c9)
fix: based image for graviton (e51ab2f)
fix: workflow file (8e74e9e)
fix: video search (#52)
chore: moving test files to a folder
fix: video search
docs: commenting TODO in search.py (af54ac0)
fix: search api as client (2573490)
fix: vid_vec_rep_res operator (0efc971)
fix: index image api (bc1a56a)
fix: image_vec_rep_resnet operator (7b3b419)
fix: elastic search test (0326407)
fix: uncommenting ES_HOST code (144c828)
fix: server is up and running (29b6100)
fix: removed trailing comma (a687377)
fix: add separator between filenames (184022f)

Refactor

refactor: config file for video worker (66cd139)
refactor: moving tests to sub-folders (6837b89)
refactor: rebasing worker branch to master (1656ece)
refactor: core module import (07cbb11)
refactor: removing api folder (91c797b)
refactor: vidvec locust es files (6d90ded)
refactor: vidvec benchmark files (4daa789)
refactor: moving test files (6e62215)
refactor: moving tests files (2eb932d)
refactor: benchmark test sh file (#64)
refactor: benchmark test sh file
ci: dockerfile udpate for benchmark.sh
chore: echo statements for benchmark file (37e768a)
refactor: cleanup deprecated thigns. (4c67853)
refactor: added comment (a8053ba)
refactor: cleaned up (e7515c0)
refactor: debugging statements (fddb01a)
refactor: source id field (6dfc8d3)

Test

test: benchmarking scripts for audiovec operator (12685bd)
test: md5 hash supports media factory (a75751c)
test: media factory unit test (d2914d1)
test: worker to queue and index video files (#84)
refactor: small improvements
test: worker to queue and index video vec (6eaf19b)

Unknown

Merge pull request #119 from tattle-made/hotfix

Hotfix (aa0d296)

Merge pull request #116 from aatmanvaidya/audio-worker-ci-tag-change

ci: audiovec worker staging dockerhub tag change (71c43c8)

Merge pull request #115 from aatmanvaidya/bench-graviton

ci: audiovec benchmark graviton dockerfile (3ca919c)

Merge pull request #114 from aatmanvaidya/video-worker-config

refactor: config file for video worker (31cfe77)

Merge pull request #118 from tattle-made/hotfix

Hotfix (6cdeb25)

Merge pull request #117 from duggalsu/github_workflow_merge_dev

ci: Add automated semantic versioning (4a12794)

Merge pull request #113 from tattle-made/hotfix

Hotfix (9de3594)

Merge pull request #112 from aatmanvaidya/video-bench-update

fix: updating vidvec benchmark scripts (c766038)

Merge pull request #111 from aatmanvaidya/audio-benchmark

test: benchmarking scripts for audiovec operator (4bf9dab)

Merge pull request #110 from aatmanvaidya/audio-worker-github-workflow

ci: audio worker staging github workflow (9e11c03)

Merge pull request #109 from aatmanvaidya/audio-worker

feat: worker for audio operator (23ff0f0)

Merge pull request #108 from aatmanvaidya/video-graviton

ci: video worker dockerfile for graviton (fed8e23)

Merge pull request #93 from aatmanvaidya/md5-test-fix

test: md5 hash supports media factory (b0685a6)

Merge pull request #106 from aatmanvaidya/audio-cnn-fix

fix: audio operators uses function from a local folder (da3d37d)

Merge pull request #107 from aatmanvaidya/video-op-fix

fix: context manager handles vidoes (586b0c0)

Merge pull request #105 from tattle-made/hotfix

Hotfix (4ae4219)

Merge pull request #104 from duggalsu/github_workflow_pr

GitHub workflow pr (cda6bbb)

- Disabled failing tests (b7ace62)
- Enabled failing tests (e7df52c)
- Modified failure condition (6c22a62)
- Added echo output (4e3d04e)
- Skipped index api tests (87ba7bb)
- Added sut output and failure condition (7f2022e)
- Disabled failure check (d7e2aa4)
- Added log outputs and failure condition (ceea228)
- Enabled failing tests for testing PR workflow (cd953a4)
- Modified pr workflow to use test.env

Added test.env
Added test.env inclusion in gitignore (cd37c9d)

- Skipped search and index api tests (469d95e)
- Added nose to requirements (282e7f6)
- Disabled image vec operator in config-server.yml

Disabled audio testing (026e92f)

Add github PR testing workflow

Added github pr workflow yaml
Added ci pr test dockerfile
Added ci pr docker compose file
Added ci dockerfiles to gitignore (c857181)

Merge pull request #102 from aatmanvaidya/vid-ci-5

fix: video operator locaiton in Dockerfile (d6ed2a3)

Merge pull request #103 from tattle-made/hotfix

Hotfix (3f41df9)

Merge pull request #101 from aatmanvaidya/vid-ci-4

ci: workflow dockerfile location change (b7d9ab2)

Merge pull request #100 from aatmanvaidya/vid-ci-2

ci: fix location for video requirment file (92a103e)

Merge pull request #99 from dennyabrain/local-hotfix

Local hotfix reconciling with main (79659eb)

Merge pull request #98 from tattle-made/hotfix

Hotfix (579b7ea)

Merge pull request #96 from aatmanvaidya/worker-ci

ci: workflow to push video worker to docker hub (07b609c)

Merge pull request #95 from aatmanvaidya/worker-improve

fix: video worker handles disconnection to RabbitMQ (9ce22a2)

Merge pull request #94 from aatmanvaidya/tests-move

refactor: moving tests to sub-folders (bf24df2)

Merge pull request #80 from duggalsu/enable_rabbitmq

Enable RabbitMQ (e055a24)

Enable RabbitMQ

Enabled rabbitmq disabled in PR https://github.com/tattle-made/feluda/pull/74/files
Updated rabbitmq version (8fb6d1f)

Merge pull request #79 from duggalsu/benchmark_es

Add ElasticSearch benchmarking (03915d3)

Add ElasticSearch benchmarking

Added locust out files to gitignore
Fixed import issues in video operator
Fixed file size limit to 10mb in video operator
Optimized video operator to run with constant memory usage
Fixed UnboundLocalError in es_vec.py
Updated flask_cors package for locust compatibility
Added locust package to requirements.in
Regenerated feluda core requirements.txt
Added tests for indexing and searching videos in elastic search
Added video operator ES search benchmark locust file
Added python file to index videos as init for ES load testing
Added bash script to perform video load testing (1e6b470)

[WIP] test: evaluating audio vec ES index and search (#77)
test: evaluating audio vec ES index and search
docs: delete stored documents (ad94ad7)
Merge pull request #76 from duggalsu/add_arch_to_docker_tag

Added architecture to docker tag name (812cd1d)

- Added architecture to docker tag name (17c24d5)
Merge pull request #75 from duggalsu/fix_docker_tag_issue

Fix tag name issue (f4905bf)

Fix tag name issue

Modified tag naming syntax in github actions (54fea21)

Merge pull request #74 from duggalsu/config_changes

Update configs (7ca754d)

Update configs

Disabled feluda reporter from docker compose file
Disabled rabbitmq
Removed operator specific packages from feluda core
Regenerated feluda core requirements
Enabled debian bullseye in core feluda for AWS Graviton support
Removed apt-get packages from feluda core dockerfile
Updated torch version in vid vec requirements.in
Added graviton dockerfile
Updated vid vec github action to create dockerfile per arch (e005e32)

Merge pull request #72 from duggalsu/add_pytorch_arch_conditional

Add pytorch arch conditional (136894a)

- Revert base image for Graviton compatibility (2985432)
Add arch conditional pytorch install

Added conditional for building multi-arch cpu pytorch (1ecc864)

Merge pull request #71 from duggalsu/fix_vid_vec_graviton_dockerfile

Fix Graviton commands (72bd171)

Fix Graviton commands

Fixed AWS Graviton opt commands in dockerfile (f3893aa)

Merge pull request #70 from duggalsu/fix_github_actions_docker_platforms

Fix Unexpected input(s) 'platforms' (09c320d)

Fix Unexpected input(s) 'platforms'

Fixed platforms issue (6fb2c89)

Merge pull request #69 from duggalsu/fix_github_actions_docker

Fix docker github action (e74f388)

Fix docker github action

Added custom node version
Fixed node12 and node16 deprecation warnings by upgrading actions
Fixed set-output deprecation warning (8dac771)

Merge pull request #68 from duggalsu/opt_pytorch_graviton

Optimize docker for multi-arch builds (575b6c4)

- Fix num_processes flag (c22bc58)
Optimize docker for multi-arch builds

Added pytorch optimization for AWS graviton in dockerfile
Modified requirements.txt to work with multi-arch support
Modified docker vid vec github action with multi-arch build support (3e2707f)

fix (9a10d9a)
revert: python version (#67) (4950ee7)
Merge pull request #63 from duggalsu/fix_operator_issue

Fix modular operator issues (04c92f4)

Fix modular operator issues

Updated shebang in shell scripts
Updated operator dockerfiles
Updated requirements (d850c05)

Merge pull request #62 from duggalsu/benchmark_scripts

Benchmark scripts (b4c4eca)

- Added shebang to all scripts (2ea4d1b)
- Added chmod executable for all scripts in image and vid dockerfiles (fcc8a68)
Add benchmarking scripts

Added image vec python script
Added image memray benchmarking shell script
Added image pyinstrument benchmarking shell script
Added video vec python script
Added video memray benchmarking shell script
Added video pyinstrument benchmarking shell script (ee7d338)

Merge pull request #58 from duggalsu/docker_opt

Create and optimize Dockerfiles (7ece59c)

- fixed typo in readme (267d13e)
- Updated readme

Added --no-cache-dir for pip install in dockerfiles
Removed vim curl single install command from core dockerfile
Removed torch, torchvision as core feluda dependency
Added numpy as core feluda dependency
Recreated core requirements.txt
Modified test urls (7b0cf99)

Create and optimize Dockerfiles

Optimized feluda core Dockerfile
Added image_vec_rep_resnet operator Dockerfile
Added vid_vec_rep_resnet operator Dockerfile
Updated boto3 to fix dependency incompatibility
Fixed deprecation warning for resnet18
Updated image_vec_rep_resnet requirements.in
Recreated image_vec_rep_resnet requirements.txt
Updated nltk version in text_vec_rep_paraphrase_lxml for compatibility
Recreated text_vec_rep_paraphrase_lxml requirements.txt
Removed unused packages for video operator
Fixed os as global import in video operator
Removed ffmpeg dependency in video operator and feluda core
Recreated video operator requirements.txt
Removed unused packages from feluda core
Recreated feluda core requirements.txt (85a7e6d)

Merge pull request #56 from duggalsu/cpu_profiling

Add cpu profiling and optimize operator (c350792)

Add cpu profiling and optimize operator

Added pyinstrument for cpu profiling
Added gitignore requirement for pyinstrument
Disabled compress function for operator
Removed ffmpy dependency as it is not required now (95a2959)

Merge pull request #55 from duggalsu/mem_profiling

Add memory profiling (9d7fb86)

Add memory profiling

Added memray package for memory profiling
Recreated requirements.txt
Fixed typo in operator
Added memray output bin and html files to gitignore (69ab511)

Merge pull request #54 from duggalsu/test_documentation

Update documentation (a29e7a7)

- Updated readme

Updated gitignore with sonarqube exclusion (375c1b9)

Merge pull request #50 from duggalsu/deprecate_ner_extraction_operator

Deprecate NER Extraction operator (8e55226)

Deprecate NER Extraction operator

Deleted the ner extraction python file as packages are not maintained
Deleted operator requirements.in
Deleted operator requirements.txt (7b27aec)

- Updated readme

Updated required operators in config-server.yml
Downgraded package in vid_vec_rep_resnet_requirements for compatibility
Modified handler.py for compatibility with current operators
Updated packages in core requirements to match operator versions (77f867c)

- Updated readme

Removed package install scripts from operators
Created operator-specific requirements.in
Generated operator-specific requirements.txt (f276855)

- Updated readme

Moved all operator package installs to requirements.in
Deleted operator/installer.py script for harm reduction
Recreated requirements.txt (acf8b73)

- Removed extra newline from docker-compose file

Updated flask debug settings for new version in Dockerfile
Updated package sentencepiece to work with cp311 (c60cf42)

- Updated python docker image (c05de12)
Update Feluda

Updated readme instructions
Fixed docker-compose file issues
Updated and pinned packages to work with cp311, fixes from pip-audit
Recreated requirements.txt (6d91094)

Merge pull request #43 from aatmanvaidya/docs-ocr

docs: documentation for tesseract ocr operator (79d67ee)

Create privacy_policy.md (ac254d1)
Update README.md (2aad47e)
fix : renamed env variable for kosh api (f6bb56c)
chore : testing ci (72c6c4a)
feat : added deploy script to docs (3a8187b)
chore : testing ci (7e626c1)
chore : testing ci (c6424cf)
chore : testing ci (7e3d836)
feat : misc

add support for raw query of es.
add ci workflow. (88a6607)

fix : string to dict conversation in reporter. (8f1433a)
fix : resolve issues arising from using e_kosh_id as the key in store as opposed to id or post_id (d4aabba)
feat : add debug support for store and queue. enable shared networking with kosh. (3c30e6e)
Merge branch 'master' of github.com:tattle-made/tattle-api (3919dbd)
feat : add ner support, add debug cli and its documentation (8757c60)
GP | added CORS support, fixed image search (caace21)
fix : make datasource_id and client_id optional (8873c9e)
fix : docker compose debug changes (bdceed1)
fix : merge conflict (4fe2143)
tmp (2a37de0)
doc: clean up feluda and document config module. (edeb72e)
doc : add todo to convert an image search model into an operator (4266717)
feat : test end to end index endpoint (b3e5b12)
refactor : separate feluda core code and user code (0918507)
doc : add overview, operators, architecture, etc. (0961e59)
doc : add gatsby site for documentation. (b5c519a)
fix : rename operator variable (839adfa)
doc : add endpoint and overview refactor : rename feature to endpoint (68f9d5c)
fix: (0a02845)
doc: add caveate about environment variables (0da5097)
feat : add type safety for config (be08274)
feat : add logger and documentation. add queue (3a80079)
feat : add frontmatter to documentation pages (2a08202)
dmp (a198747)
feat : add test for /represent image (1e3d8d1)
doc : add documentation for operators (a4bceda)
refactor : move operator config up as standalone in config.yml. remove index and search realted config. (170f838)
feat : debugging ready (38525c5)
feat : set up 3 endpoints for indexing media (1efc4fd)
testing API endpoints with json and file sent in one request (08204b8)
dmp (14b7845)
fix : update README (56f42f0)
wip (a8409ab)
temp (2fcb3b0)
dmp (157f7ed)
dmp (931850f)
change package versions to resolve conflicts (e13cb92)
add video indexing example & docker build info (c439ccc)
increased number of search results returned by API (5594278)
added bulk indexing info (9252902)
add more elasticsearch info (47f2104)
updated (c9bd498)
add explanatory comment for query size (d764fe1)
expand docstring for sentencer transformer function (3122078)
comment out index deletion & add warning (3887661)
deprecate mongo, remove its dependencies (f3e4e28)
deprecate mongo, remove its dependencies (69d3970)
return top 3 search hits in more readable format (c943828)
changed text vector size (edaeead)
added sentence transformers library (6615f95)
replaced word2vec with sentence transformer embeddings (0a92b67)
word2vec db install files with lang ids fixed (ae5fb01)
ignore word2vec db & vecs (32ee2c4)
allow word2vec installation files (321b8e7)
debugging chore: create fresh indices on server start (947fd52)
prettify logs' ' (32fbacb)
added error handling (1c598c5)
added error handling (224b44e)
removed old code (c14325a)
enabled indexing via singleton rabbitmq (8d3dd55)
enabled indexing via singleton rabbitmq (28e1535)
copied api server side helper (e8e1a5d)
add singleton es instance (e4d7215)
enabled simple text search (4ecddb4)
use wrapper func for indexing to avoid instantiating es here (e42692b)
return avg vid vec's index id & define wrapper func for indexing (4d92228)
defined separate funcs for generatings vec (9615b59)
import helper funcs for generating vecs (70e8c67)
enabled image & video search (e996e9b)
add combined_vec field for images (8377adc)
helper for queue-less indexing (e34e201)
update index fields (72071d6)
move import indexing code from helper (22bfef2)
enabled text vector based similarity search (87b7af1)
removed unnecessary print statements (38fad38)
enabled queue-less text indexing (e4758e9)
refactor (b76070b)
changed local ES host for development (b9f7ed3)
renamed (9b664e3)
renamed (e7aec50)
moved index check/creation to server (bb39768)
helper for creating the indices (0c856a5)
check if index exists on server start (092b264)
fixed mongo local host (4a34440)
add fields to mappings & ensure text vecs are searchable (db37c20)
explicitly specify which text analyzer (e9a6637)
ensure index with proper mapping exists before indexing (0cb9c85)
index text vec in elasticsearch (6158cf4)
(feat) add CD github action for development branch (a7236f4)
Merge changes to elasticsearch host (35f0726)
(chore) wip (6504f59)
removed (78955df)
ignore docker logs, allow word2vec installation files (ba34c2e)
update word2vec path (ebad682)
update word2vec path (ed44081)
set up only word2vec.db & include currently scraped langs (b8d826c)
word2vec & alignedvec db setup scripts (73955a0)
pass mongo db, collection names as env variables (d9a3ccd)
ignore google creds (282de50)
add creds for s3 bucket containing google application creds (22c75a5)
enable restart when receive.py is modified (cda7fdd)
reduced batch size & refactored (234ffab)
refactored (f3aea6b)
download google creds from s3 (92a0f34)
added boto3 (bebb6a2)
hstack instead of vstack with small batch sizes (ed02b10)
revert to original batchsize (cb0e9bf)
try with batchsize=1 (4338661)
updated .env instructions (c9276a1)
fixed formatting (27d43b1)
updated (a703fb9)
templates for env files (a6aa9a3)
don't import langdetect (deb28e7)
time execution (b6fb235)
pre-compiled requirements for copying in dockerfile (e1c8aa1)
add ffmpy, remove unused packages (9386cb4)
enable multistage builds & leaner images (274805e)
enable multistage build & increase shared memory (9413527)
increased heartbeat interval & added publisher confirms (f5e215d)
enable video compression with ffmpy (4deb006)
return mongo id after indexing (88af964)
allow queue-less indexing via original endpoints (d667e4a)
added queueing, updated requirements, refactored code (49d5f51)
compute time taken (126b952)
downloads & unzips wordvecs with error handling (3417361)
added wget (2dc98f3)
create single endpoint for all media uploads index videos via file urls make request data keys consistent with simple search (c28a342)
Merge branch 'feature/dockerization' (ad31800)
cherry picked changes from master (37b1584)
added debugging statements (475b36e)
sample cli commands for elasticsearch (555a10b)
fixes, new generator for bulkdata upload for vidsearch index (9129f87)
Merge branch 'master' of github.com:tattle-made/tattle-api (4ef99b5)
use new es video index (de9e2c5)
extract vid attrs, do sanity check (a2a1216)
create_indices for vid, txt, img (9049ef0)
video search todo lis (a4092cf)
(fix) (99a7791)
(feat) dockerize app (1a986cc)
streamlit app to testout word embeddings (4fdccc6)
working upload_video, get mean_feature, upload to es (7a1e783)
support passing vid as a param to run VideoAnlayzer as a script (0ecfb24)
don't overwrite feature_matrix while computing QR (57fc28c)
VideoAnalyzer updates: extract features, find keyframes via QR (972e57a)
updated es_test script (a26b210)
v0.1 upload_video api (53306a9)
VideoAnalyzer class (329fb5b)
Merge pull request #9 from tattle-made/dev/ngmuley/fix-install-for-mac

update install script for mac OS, update requirements.txt with latest… (3789a6b)

elastic search support, es_flag, and query_es method (17c8d6a)
setup local es on aws (fdd5198)
Merge branch 'master' of github.com:tattle-made/tattle-api (8f15776)
[fix] text len < 3 error for detect_lang (3599d2c)
es_test script (4b1a9df)
update install script for mac OS, update requirements.txt with latest pip-compiled from requirements.in (6dfaa74)
db setup for aligned wordvecs (270220a)
return lang=None if text is empty (ae696d5)
[fix] shorted uuid to fix mongodb 8-byte int issue (60f70f8)
remove monitoring (c201313)
only fetch top 10 docs from mongo (9113bbe)
Merge branch 'master' of github.com:tattle-made/tattle-api (12b632d)
monitor running time for a few functions (f638aa4)
monitor -> timeit decorator (f393141)
Merge pull request #7 from dennyabrain/feature/public-docs

(chore) add contributing and CoC readmes (892bc81)

(chore) add contributing and CoC readmes (d247215)
fix the missing words in word2vec db bug (75d74be)
limit duplicate docs at 10 (aa8b29a)
user textblob lib to detect language (0340cbd)
fix order of source issue in find_duplicate (04ce18e)
return list of docs in order of match in find_duplicates (94ec663)
default doc_id int instead of string (c38dc41)
source for each doc, default tattle-admin (b39f7d7)
search_tags API, pass list of tags, list of sources (ac2c7a1)
support for flexible threshold passing in find_duplicate API (7e4f1f5)
remove_tags API and support for source in tags (a9f3568)
delete_doc API (0074efc)
ebextension packages (8ce9655)
torch to requirements (f341f13)
remove torch stuff from requirements, to be installed separately (58f06e9)
video analysis notebook changes (0d649e0)
word2vec script change (c905ebd)
eb config (bf6c156)
fix, docsearch -> textsearch (f615d79)
remove single quotes to avoid messing up the sql query (0b99cbf)
handle en,gu,hi language ids (1f83db6)
fix another png issue (3b7170c)
separate image search, text search, doc search(text+image) (274d3e1)
take care of new lines in text (644fad0)
closes #2, convert RGBA to RGB (089a049)
error checking for detect_text (d958744)
find_text API end point for finding text inside images (382c086)
fix detect_text function (86e3cbf)
Merge branch 'master' of github.com:tattle-made/tattle-api (1806859)
video analysis experiments (e5d398f)
support for searching document vectors (c0f07f8)
search threshold to 0.6, handle None vec (1d0fe7b)
doc vectors search support (0350d69)
[fix] imports (099988e)
[fix] import requests (c9bc592)
full index on wordvecs (6215c3b)
doc2vec method (95e9456)
ignore .vec and .db files from git (1c48ecc)
scripts to generate word2vec database (b288ddb)
install_torch script (0da5955)
detect_lang function (b82e1e8)
move image_from_url to analyzer (ca36f7d)
fasttext, langdetect depedencies (a29cb1d)
update_tags api (51e1ba4)
fix has_text field (c885ea4)
tags field for documents (1ad64a8)
update imagesearch when a new image is uploaded (0be7859)
support to search for images in find_duplicate (89347cb)
add tqdm updated requirements.txt (6b994e8)
add image fingerprint with each upload_image call (ac7db71)
support for doc_id for upload_text and upload_image apis (409d7f0)
text detection in uploaded image (02c0d1f)
fix image_upload api (bfd59e4)
some tests (489093f)
upload_image api (67be7d8)
find_duplicate API, add support for text (13cce36)
upload_text API (d4dbbf2)
ignore etc, share (e698a2a)
dnspython requirement (249875c)
test for text docs (96269a5)
pymongo requirement (a099621)
application loop (1691203)
fix Dockerfile pip install (5d6c87b)
try to fix Dockerfile (9d67658)
requirements.txt, to avoid installing pip-tools on server (2f744a0)
ignore eb files (c9df037)
add dockerfile (09919f7)
torch, skimage dependencies (a4bf37e)
api name change (0eb9df7)
tests (181127a)
feature extraction for images (3987959)
format readme (b6fc431)
ignore data dir (09f2210)
analyze.py, try google vision api (5ec1e03)
gitignore (320b19b)
init (ca74ede)

Files

CHANGELOG.md

Latest commit

History

CHANGELOG.md

File metadata and controls

CHANGELOG

v0.8.0 (2024-09-12)

Chore

Ci

Feature

Fix

Refactor

Style

Test

Unknown

v0.7.2 (2024-05-28)

Fix

Unknown

v0.7.1 (2024-05-28)

Fix

Unknown

v0.7.0 (2024-05-28)

Chore

Ci

Unknown

v0.6.6 (2024-05-24)

Chore

Documentation

Feature

Fix

Test

Unknown

v0.6.5 (2024-05-10)

Chore

Fix

Unknown

v0.6.4 (2024-05-08)

Fix

Unknown

v0.6.3 (2024-05-07)

Fix

Unknown

v0.6.2 (2024-05-06)

Fix

Unknown

v0.6.1 (2024-05-06)

Fix

Unknown

v0.6.0 (2024-05-01)

Chore

Unknown

v0.5.4 (2024-04-30)

Chore

Ci

Fix

Refactor

Unknown

v0.5.3 (2024-04-04)

Chore

Feature

Fix

Refactor

Unknown

v0.5.2 (2024-03-23)

Ci

Fix

Unknown

v0.5.1 (2024-03-23)

Chore

Ci

Fix

Unknown

v0.5.0 (2024-03-21)

Chore

Ci

Feature

Fix

Conflicts:

src/core/store/postgresql.py (3a5f003)

src/core/store/postgresql.py (`3a5f003`)

src/core/models/media_factory.py (`36ec52b`)