Skip to content

Conversation

@mergify
Copy link

@mergify mergify bot commented Nov 12, 2025

Fixes issues in the storage system by guaranteeing immediate durability for responses and ensuring background writers stay alive. Three related fixes:

  • Responses to the OpenAI-compatible API now write directly to Postgres/SQLite inside the request instead of detouring through an async queue that might never drain; this restores the expected read-after-write behavior and removes the "response not found" races reported by users.

  • The access-control shim was stamping owner_principal/access_attributes as SQL NULL, which Postgres interprets as non-public rows; fixing it to use the empty-string/JSON-null pattern means conversations and responses stored without an authenticated user stay queryable (matching SQLite).

  • The inference-store queue remains for batching, but its worker tasks now start lazily on the live event loop so server startup doesn't cancel them—writes keep flowing even when the stack is launched via llama stack run.

Closes #4115

Test Plan

Added a matrix entry to test our "base" suite against Postgres as the store.


This is an automatic backport of pull request #4118 done by Mergify.

@mergify mergify bot added the conflicts label Nov 12, 2025
@mergify
Copy link
Author

mergify bot commented Nov 12, 2025

Cherry-pick of 492f79c has failed:

On branch mergify/bp/release-0.3.x/pr-4118
Your branch is up to date with 'origin/release-0.3.x'.

You are currently cherry-picking commit 492f79ca.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   .github/actions/setup-test-environment/action.yml
	modified:   llama_stack/providers/utils/sqlstore/authorized_sqlstore.py
	modified:   tests/integration/fixtures/common.py
	modified:   tests/integration/suites.py

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	both modified:   .github/workflows/integration-tests.yml
	added by them:   llama_stack/distributions/ci-tests/run-with-postgres-store.yaml
	both modified:   llama_stack/distributions/starter-gpu/run.yaml
	both modified:   llama_stack/distributions/starter/run.yaml
	both modified:   llama_stack/distributions/starter/starter.py
	both modified:   llama_stack/providers/utils/inference/inference_store.py
	both modified:   llama_stack/providers/utils/responses/responses_store.py
	deleted by us:   tests/integration/ci_matrix.json

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

@mergify mergify bot mentioned this pull request Nov 12, 2025
@meta-cla meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Nov 12, 2025
Fixes issues in the storage system by guaranteeing immediate durability
for responses and ensuring background writers stay alive. Three related
fixes:

* Responses to the OpenAI-compatible API now write directly to
Postgres/SQLite inside the request instead of detouring through an async
queue that might never drain; this restores the expected
read-after-write behavior and removes the "response not found" races
reported by users.

* The access-control shim was stamping owner_principal/access_attributes
as SQL NULL, which Postgres interprets as non-public rows; fixing it to
use the empty-string/JSON-null pattern means conversations and responses
stored without an authenticated user stay queryable (matching SQLite).

* The inference-store queue remains for batching, but its worker tasks
now start lazily on the live event loop so server startup doesn't cancel
them—writes keep flowing even when the stack is launched via llama stack
run.

Closes #4115

Added a matrix entry to test our "base" suite against Postgres as the
store.
@ashwinb ashwinb force-pushed the mergify/bp/release-0.3.x/pr-4118 branch from d13fd2b to 81e44b0 Compare November 12, 2025 20:40
@ashwinb ashwinb force-pushed the mergify/bp/release-0.3.x/pr-4118 branch from e72f380 to 283a998 Compare November 12, 2025 20:55
@ashwinb ashwinb merged commit a6c3a9c into release-0.3.x Nov 12, 2025
88 checks passed
@ashwinb ashwinb deleted the mergify/bp/release-0.3.x/pr-4118 branch November 12, 2025 21:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot. conflicts

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants