Traces Disapeared on my webui, but are still available through api (python sdk)

[RAPHCVR]

Hey, on my self hosted langfuse, traces are available through the api it seems:

from langfuse import Langfuse

langfuse = Langfuse(
    secret_key="sk...",
    public_key="pk..",
    host="https://langfuse..."
)

# Fetch list of traces, supports filters and pagination
traces = langfuse.fetch_traces()

# Fetch list of observations, supports filters and pagination
observations = langfuse.fetch_observations()

# Fetch list of sessions
sessions = langfuse.fetch_sessions()

Indeed there is plenty of elements here.

But, on the webui, only 3 traces are available (all trace were there this morning, I haven't touched anything), I tried to update to the latest version, but it did not solve the issue.

Finally, I am using Kubernetes with the provided helm chart from the docs.

Thanks in advance for your help !

[dosubot[bot]]

Hey @RAPHCVR! 👋 I'm here to help you with any bugs, questions, or contributions. Let's get to the bottom of this!

If traces are visible through the API but not in the web UI, there could be several reasons for this discrepancy:

1. Data Mismatches: There might be mismatches between the data stored in Postgres and Clickhouse. You can use the provided script to verify records between these two databases and log any mismatches ^[1].

2. Access Control: Ensure that the user accessing the web UI has the necessary permissions and is a member of the relevant project. Access control mechanisms can affect trace visibility ^[1].

3. Caching or Synchronization Issues: The Langfuse backend processes events asynchronously, which might introduce delays in their availability in the UI. This could lead to temporary discrepancies until the data is fully processed and synchronized ^[2].

4. Query Parameters: Check if any query parameters or filters are applied in the web UI that might limit the number of traces displayed. Parameters such as limit, fromTimestamp, toTimestamp, and tags can affect the results ^[3].

5. Helm Chart Configurations: Verify that the Helm chart configurations, especially those related to database connections and migrations, are correctly set. Incorrect settings could affect data synchronization between Postgres and Clickhouse ^[4].

Review these areas to identify and resolve the issue with trace visibility in the web UI.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[RAPHCVR]

It seems it has to do with clickhouse pod, does someone have a hint ?

[marcklingen]

Did you just update to v3? Do you see that any background migrations are still running?

[marcklingen]

Have you selected a long enough timeframe here?

[RAPHCVR]

Thanks for your reply @marcklingen, I tried to release langfuse again from scratch, and the issue is not there from now it seems (as with my first deployment), but maybe the traces will disappear again, I have this error in clickhouse that was also present when the bug was occuring:

[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	2024.12.17 08:57:14.616728 [ 620 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f0df80d7-cf7e-429d-8ff0-9c24d467fb58::202412_1_7952_270}: Code: 241. DB::Exception: (total) memory limit exceeded: would use 2.70 GiB (attempt to allocate chunk of 4361936 bytes), current RSS 1.79 GiB, maximum: 2.70 GiB. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000d1d323b
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000817ab6c
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	2. DB::Exception::Exception<char const*, char const*, String, long&, String, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000000d1f36d2
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	3. MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000000d1f279e
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	4. MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000000d1f2205
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	5. MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000000d1f2205
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	6. MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000000d1f2205
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	7. Allocator<false, false>::alloc(unsigned long, unsigned long) @ 0x000000000d1a4140
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	8. DB::Memory<Allocator<false, false>>::alloc(unsigned long) @ 0x000000000d238ead
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	9. DB::MergeTreeDataPartWriterOnDisk::Stream<false>::Stream(String const&, std::shared_ptr<DB::IDataPartStorage> const&, String const&, String const&, String const&, String const&, std::shared_ptr<DB::ICompressionCodec> const&, unsigned long, std::shared_ptr<DB::ICompressionCodec> const&, unsigned long, DB::WriteSettings const&) @ 0x0000000012939b90
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	10. void std::__function::__policy_invoker<void (DB::ISerialization::SubstreamPath const&)>::__call_impl<std::__function::__default_alloc_func<DB::MergeTreeDataPartWriterWide::addStreams(DB::NameAndTypePair const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&, std::shared_ptr<DB::IAST> const&)::$_0, void (DB::ISerialization::SubstreamPath const&)>>(std::__function::__policy_storage const*, DB::ISerialization::SubstreamPath const&) @ 0x000000001294ace1
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	11. DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000010c72a30
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	12. DB::SerializationSparse::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000010ce4894
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	13. DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000010c72ce8
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	14. DB::MergeTreeDataPartWriterWide::addStreams(DB::NameAndTypePair const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&, std::shared_ptr<DB::IAST> const&) @ 0x0000000012942bbf
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	15. DB::MergeTreeDataPartWriterWide::MergeTreeDataPartWriterWide(String const&, String const&, std::unordered_map<String, std::shared_ptr<DB::ISerialization const>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, std::shared_ptr<DB::ISerialization const>>>> const&, std::shared_ptr<DB::IDataPartStorage>, DB::MergeTreeIndexGranularityInfo const&, std::shared_ptr<DB::MergeTreeSettings const> const&, DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::VirtualColumnsDescription const> const&, std::vector<std::shared_ptr<DB::IMergeTreeIndex const>, std::allocator<std::shared_ptr<DB::IMergeTreeIndex const>>> const&, std::vector<std::shared_ptr<DB::ColumnPartStatistics>, std::allocator<std::shared_ptr<DB::ColumnPartStatistics>>> const&, String const&, std::shared_ptr<DB::ICompressionCodec> const&, DB::MergeTreeWriterSettings const&, std::shared_ptr<DB::MergeTreeIndexGranularity>) @ 0x0000000012942925
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	16. DB::createMergeTreeDataPartWideWriter(String const&, String const&, std::unordered_map<String, std::shared_ptr<DB::ISerialization const>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, std::shared_ptr<DB::ISerialization const>>>> const&, std::shared_ptr<DB::IDataPartStorage>, DB::MergeTreeIndexGranularityInfo const&, std::shared_ptr<DB::MergeTreeSettings const> const&, DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::VirtualColumnsDescription const> const&, std::vector<std::shared_ptr<DB::IMergeTreeIndex const>, std::allocator<std::shared_ptr<DB::IMergeTreeIndex const>>> const&, std::vector<std::shared_ptr<DB::ColumnPartStatistics>, std::allocator<std::shared_ptr<DB::ColumnPartStatistics>>> const&, String const&, std::shared_ptr<DB::ICompressionCodec> const&, DB::MergeTreeWriterSettings const&, std::shared_ptr<DB::MergeTreeIndexGranularity>) @ 0x0000000012929cc2
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	17. DB::createMergeTreeDataPartWriter(DB::MergeTreeDataPartType, String const&, String const&, std::unordered_map<String, std::shared_ptr<DB::ISerialization const>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, std::shared_ptr<DB::ISerialization const>>>> const&, std::shared_ptr<DB::IDataPartStorage>, DB::MergeTreeIndexGranularityInfo const&, std::shared_ptr<DB::MergeTreeSettings const> const&, DB::NamesAndTypesList const&, std::unordered_map<String, unsigned long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, unsigned long>>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::VirtualColumnsDescription const> const&, std::vector<std::shared_ptr<DB::IMergeTreeIndex const>, std::allocator<std::shared_ptr<DB::IMergeTreeIndex const>>> const&, std::vector<std::shared_ptr<DB::ColumnPartStatistics>, std::allocator<std::shared_ptr<DB::ColumnPartStatistics>>> const&, String const&, std::shared_ptr<DB::ICompressionCodec> const&, DB::MergeTreeWriterSettings const&, std::shared_ptr<DB::MergeTreeIndexGranularity>) @ 0x0000000012833da8
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	18. DB::MergedBlockOutputStream::MergedBlockOutputStream(std::shared_ptr<DB::IMergeTreeDataPart> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::NamesAndTypesList const&, std::vector<std::shared_ptr<DB::IMergeTreeIndex const>, std::allocator<std::shared_ptr<DB::IMergeTreeIndex const>>> const&, std::vector<std::shared_ptr<DB::ColumnPartStatistics>, std::allocator<std::shared_ptr<DB::ColumnPartStatistics>>> const&, std::shared_ptr<DB::ICompressionCodec>, std::shared_ptr<DB::MergeTreeIndexGranularity>, DB::TransactionID, bool, bool, bool, DB::WriteSettings const&) @ 0x0000000012a975b3
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	19. DB::MergeTask::ExecuteAndFinalizeHorizontalPart::prepare() const @ 0x0000000012809e77
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	20. DB::MergeTask::ExecuteAndFinalizeHorizontalPart::execute() @ 0x000000001281160e
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	21. DB::MergeTask::execute() @ 0x000000001281a3e2
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	22. DB::MergePlainMergeTreeTask::executeStep() @ 0x0000000012c38b68
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	23. DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001283e10d
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	24. ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::worker() @ 0x000000000d2a997b
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	25. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>(void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&&)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000d2b0442
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	26. ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x000000000d2a6c22
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	27. void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x000000000d2adfda
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	28. ? @ 0x00007ccca41211c4
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	29. ? @ 0x00007ccca41a0ac0
[2024-12-17  09:57:14]	(langfuse-clickhouse-shard0-0 clickhouse)	 (version 24.11.1.2557 (official build))

I have this error with the clickhouse pods.

[RAPHCVR]

Hey @marcklingen, I still have this error in my Clickhouse pod (I am using Kubernetes for deployment).
It does not seem to affect the displaying of the traces, but it's still a bit concerning.
No matter if I redeploy all from zero, the error ends to appear each time.