Ability to run langfuse in tests automatically without manual involvement

[SinghCoder]

Hey team,

Thanks for the wonderful project and hard work you folks are putting in.

I am trying to run langfuse in my docker compose setup for e2e tests debuggability. Running it via docker compose was easy part (thanks to amazing docs).

The hard part is allowing my service to connect to this langfuse instance.. It needs API keys to connect, but that seems like a manual step for now.. i.e. no way to provide mock project and api keys in docker file?

I saw this pending issue #517 with a PR linked even, but no updates there for ~5 months now..

Is there another way to do this at the moment?
I saw your .github pipeline file also, but couldn't figure out how can I plug environment "examples" into the docker compose file.. Do you think that would help? Or any other suggestions on achieving this?

[marcklingen]

Hi @SinghCoder, agree, this would be super useful and we moved to slow on this. Would the seed command solve for your goal already?

I intended to make a change on this which allows to set a default project via env variables alone but did not get to it yet

[marcklingen]

ok great, thanks for the update! we hopefully get to a proper setup for this soon

[SinghCoder]

Thank you! Really appreciate the great tool you folks have built.

[marcklingen]

soon = might take another 2-3 weeks at least as I want to get some foundational things shipped to get this right and reliable long-term tbh

[marcklingen]

thanks for the kind words! let us know if you have any additional feedback

[SinghCoder]

Sure, thanks, will do.

[marcklingen]

Hi @SinghCoder,

I'm happy to share an update. We've just released headless initialization via the LANGFUSE_INIT_* environment variables. You can learn more about this change on our changelog: https://langfuse.com/changelog/2024-09-04-headless-initialization-or-self-hosted-deployments

Regarding running Langfuse in CI, we also migrated our CI tests to use this initialization. See here for an example of our GitHub action (needs to be passed to the application at runtime):

langfuse/.github/workflows/pipeline.yml

Lines 121 to 132 in 898cca5

	- name: Start Langfuse
	run: (pnpm run start&)
	env:
	LANGFUSE_INIT_ORG_ID: "seed-org-id"
	LANGFUSE_INIT_ORG_NAME: "Seed Org"
	LANGFUSE_INIT_PROJECT_ID: "7a88fb47-b4e2-43b8-a06c-a5ce950dc53a"
	LANGFUSE_INIT_PROJECT_NAME: "Seed Project"
	LANGFUSE_INIT_PROJECT_PUBLIC_KEY: "pk-lf-1234567890"
	LANGFUSE_INIT_PROJECT_SECRET_KEY: "sk-lf-1234567890"
	LANGFUSE_INIT_USER_EMAIL: "[email protected]"
	LANGFUSE_INIT_USER_NAME: "Demo User"
	LANGFUSE_INIT_USER_PASSWORD: "password"

Thank you for your patience. We wanted to add this feature much sooner, but it was blocked by a recent organizational change that could have otherwise caused the configuration to break.

Please let me know if this solves your use case and if you have any feedback.

[marcklingen]

You could just use the langfuse python sdk to fetch them and then add them to another instance (need to loop over all observations within a trace as well), thereby you do not have to rely on interfaces that might break

[SinghCoder]

yeah, I meant, some kind of intermediate state so I can store it in a common storage in s3 or somewhere along with other metrics artifacts etc.
And then, when someone else from the team wants to visualize it locally, they run the langfuse locally, and import from s3 to here.
I understand your solution as well, and that makes sense to me also, but our use case is a bit different as I would like to keep the traces of that test run dumped along with other artifacts too.
Excited to hear if that makes sense to you, and I can contribute to it in the codebase if the export/import functionality for a session makes sense for example.
In the product I understand it will be a big decision, because you will need to finalize the intermediate storage format, will it be OTEL etc format, something custom, and shoudl we do only traces, etc etc, and might not be a small change in itself.

[marcklingen]

Unfortunately I agree on this, this is not a small change. Thereby we will need to maintain a storage format that is forwards/backwards compatible. I think this is not worth the maintenance effort right now unless more teams are interested in this. Can you open another thread for this and reference it here?

[SinghCoder]

Sure thing!
For now, our internal format export/import works.
I will create a thread.

[SinghCoder]

Opened https://github.com/orgs/langfuse/discussions/3240
Thanks!!