Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(task_db): sqlite and in memory abstraction #301

Merged
merged 12 commits into from
Jul 2, 2024
Merged

feat(task_db): sqlite and in memory abstraction #301

merged 12 commits into from
Jul 2, 2024

Conversation

petarvujovic98
Copy link
Contributor

@smtmfft Here are some changes I propose, also a raiko branch so both of us can collaborate

@petarvujovic98 petarvujovic98 changed the base branch from main to tasks_db July 1, 2024 09:14
@petarvujovic98
Copy link
Contributor Author

We can close #300 and continue working on it here

@petarvujovic98
Copy link
Contributor Author

@smtmfft I have come up with a wrapper implementation that uses the cfg! macro to choose which implementation to use

@smtmfft
Update Cargo.toml
800b837 
use in-mem as default.
@smtmfft smtmfft temporarily deployed to test-environment July 2, 2024 08:30 — with GitHub Actions Inactive
smtmfft
smtmfft approved these changes Jul 2, 2024
View reviewed changes
Copy link
Contributor

@smtmfft smtmfft left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! I changed default to in-mem impl, see if we want more unittest. If no more, let's merge this soon and start to integrate the ZK user scenario.
Also list some tasks related to ZK user case: #306

@petarvujovic98 petarvujovic98 merged commit 9e905ed into tasks_db Jul 2, 2024
12 checks passed
@petarvujovic98 petarvujovic98 deleted the tasks-db-sqlite branch July 2, 2024 09:00
github-merge-queue bot pushed a commit that referenced this pull request Jul 2, 2024
@mratsim @petarvujovic98 @smtmfft
feat(task db): implement a task DB (#208)
48ea079 
* task-manager: dump fight vs sqlite

* task-manager: SQL tables+views+triggers success - but arguments passed in execute are 'NULL'

* task-manager: passing enqueue_task tests

* task-manager: cleanup - ease copy pasting to SQL script, Registered status, persistent views, remove debug print

* task-manager: add DB size query

* task-manager: id_proof is unneeded + prettify queries

* task-manager: change DB schema - allow multiple provers and status for same task in case of failures-retry

* task-manager: allow task updates

* task-manager: retrieve cached proofs from DB

* task-manager: add status check

* task-manager: add progress reports

* chore(task_manager): Run cargo fmt

* feat: address small lints

* feat(task-manager): use result type with thiserror

* chore(task-db): fix typos

* refactor(task-manager): clean up tests

* fix(docker): unignore task manager

* [WIP](task_manager): write initial task handler stubs

* chore(task_manager): run cargo fmt

* [WIP](task_manager): write status and proof get handlers

* refactor(host): use merge instead of nest

* chore(format): format workflow files

* chore(deps): use consistent dependency style

* chore(host): rename tx to task_channel

* [WIP](task_manager): add initial submit logic

* chore(clippy): remove unused parameter

* chore(clippy): remove unused imports

* refactor(core): add copy trait to proof types

* feat(task_manager): simplify db and adapt tests

* fix(clippy): fix dereference issue

* [WIP]: handle proof request by worker and update task status

* [WIP]: add block fetching and initial blockhash getting for submit

* [WIP]: handle task creation, status and proof retrieval

* fix(host): fix route sub-path

* feat(raiko): abstract task manager and impl a mem db for easy integration (#296)

* impl a mem db for easy integration

Signed-off-by: smtmfft <[email protected]>

* fix clippy and unit test

Signed-off-by: smtmfft <[email protected]>

* fix fmt

Signed-off-by: smtmfft <[email protected]>

---------

Signed-off-by: smtmfft <[email protected]>

* fix: throw error instead of panicing on runtime checks

* fix(core,task_manager): add custom ensure and require fns

* feat(task_db): sqlite and in memory abstraction (#301)

* enable sqlite db by feature

Signed-off-by: smtmfft <[email protected]>

* debug lifetime

Signed-off-by: smtmfft <[email protected]>

* resolve lifetime issue and make all tests pass

Signed-off-by: smtmfft <[email protected]>

* refactor(task_db): simplify structure for sqlite and use cached statements

* feat(task_db): abstract task db implementation into wrapper

* fix(task_db): add await to test call

* fix(task_db): fix import declaration

* fix(task_db): add async and mutable variables

* fix(host): fix task manager usage

* fix(task_db): fix test for async

* Update Cargo.toml

use in-mem as default.

---------

Signed-off-by: smtmfft <[email protected]>
Co-authored-by: smtmfft <[email protected]>
Co-authored-by: smtmfft <[email protected]>

* feat(task_manager): return empty list on key not found

* feat(host,task_manager): add tracing and handle workers

* feat(host): fix response structure

* chore(clippy): remove unused imports

* fix(ci): remove git merge added lines

* fix(task_manager): add blob proof type field

---------

Signed-off-by: smtmfft <[email protected]>
Co-authored-by: Petar Vujović <[email protected]>
Co-authored-by: smtmfft <[email protected]>
Co-authored-by: smtmfft <[email protected]>
Champii pushed a commit to Champii/raiko that referenced this pull request Jul 2, 2024
@mratsim @petarvujovic98
@smtmfft @Champii
feat(task db): implement a task DB (taikoxyz#208)
5d63e7e 
* task-manager: dump fight vs sqlite

* task-manager: SQL tables+views+triggers success - but arguments passed in execute are 'NULL'

* task-manager: passing enqueue_task tests

* task-manager: cleanup - ease copy pasting to SQL script, Registered status, persistent views, remove debug print

* task-manager: add DB size query

* task-manager: id_proof is unneeded + prettify queries

* task-manager: change DB schema - allow multiple provers and status for same task in case of failures-retry

* task-manager: allow task updates

* task-manager: retrieve cached proofs from DB

* task-manager: add status check

* task-manager: add progress reports

* chore(task_manager): Run cargo fmt

* feat: address small lints

* feat(task-manager): use result type with thiserror

* chore(task-db): fix typos

* refactor(task-manager): clean up tests

* fix(docker): unignore task manager

* [WIP](task_manager): write initial task handler stubs

* chore(task_manager): run cargo fmt

* [WIP](task_manager): write status and proof get handlers

* refactor(host): use merge instead of nest

* chore(format): format workflow files

* chore(deps): use consistent dependency style

* chore(host): rename tx to task_channel

* [WIP](task_manager): add initial submit logic

* chore(clippy): remove unused parameter

* chore(clippy): remove unused imports

* refactor(core): add copy trait to proof types

* feat(task_manager): simplify db and adapt tests

* fix(clippy): fix dereference issue

* [WIP]: handle proof request by worker and update task status

* [WIP]: add block fetching and initial blockhash getting for submit

* [WIP]: handle task creation, status and proof retrieval

* fix(host): fix route sub-path

* feat(raiko): abstract task manager and impl a mem db for easy integration (taikoxyz#296)

* impl a mem db for easy integration

Signed-off-by: smtmfft <[email protected]>

* fix clippy and unit test

Signed-off-by: smtmfft <[email protected]>

* fix fmt

Signed-off-by: smtmfft <[email protected]>

---------

Signed-off-by: smtmfft <[email protected]>

* fix: throw error instead of panicing on runtime checks

* fix(core,task_manager): add custom ensure and require fns

* feat(task_db): sqlite and in memory abstraction (taikoxyz#301)

* enable sqlite db by feature

Signed-off-by: smtmfft <[email protected]>

* debug lifetime

Signed-off-by: smtmfft <[email protected]>

* resolve lifetime issue and make all tests pass

Signed-off-by: smtmfft <[email protected]>

* refactor(task_db): simplify structure for sqlite and use cached statements

* feat(task_db): abstract task db implementation into wrapper

* fix(task_db): add await to test call

* fix(task_db): fix import declaration

* fix(task_db): add async and mutable variables

* fix(host): fix task manager usage

* fix(task_db): fix test for async

* Update Cargo.toml

use in-mem as default.

---------

Signed-off-by: smtmfft <[email protected]>
Co-authored-by: smtmfft <[email protected]>
Co-authored-by: smtmfft <[email protected]>

* feat(task_manager): return empty list on key not found

* feat(host,task_manager): add tracing and handle workers

* feat(host): fix response structure

* chore(clippy): remove unused imports

* fix(ci): remove git merge added lines

* fix(task_manager): add blob proof type field

---------

Signed-off-by: smtmfft <[email protected]>
Co-authored-by: Petar Vujović <[email protected]>
Co-authored-by: smtmfft <[email protected]>
Co-authored-by: smtmfft <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@smtmfft smtmfft smtmfft approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@petarvujovic98 @smtmfft