Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zilla stops sending sse updates after a couple days #1340

Open
pnikhil610 opened this issue Nov 28, 2024 · 0 comments
Open

zilla stops sending sse updates after a couple days #1340

pnikhil610 opened this issue Nov 28, 2024 · 0 comments
Labels
bug Something isn't working needs_triage Unknowns need to be addressed

Comments

@pnikhil610
Copy link

Describe the bug
My kafka topic receives about 10 million+ events per day. Each message has key and there are about 50k unique keys. I have zilla sse endpoint which filters based on key. After about a day, it makes only about sse 20k sse connections. Then sometime in day 2 or later, it stops sending live updates to sse client. That is, it freezes somehow. It just show one entries. Even though additional entries are pushed, I don't get them. I'm not sure if it's cache or resource issue.

To Reproduce
Steps to reproduce the behavior:

  1. Use similar config as mine
  2. Send lots of events in kafka topic
  3. Create a several thousand sse connection
  4. At some point, you'll see zilla stops sending updates

Expected behavior
Zilla should repeatedly clean cache to outcome if it's memory issue.

Screenshots
If applicable, add screenshots to help explain your problem.

Zilla Environment:
container and resources look like

      containers:
       - name: zilla
         image: ghcr.io/aklivity/zilla:0.9.101
         args:
           - start
           - '-v'
           - '-e'
           - '-Pzilla.engine.worker.capacity=128'
         env:
           - name: ZILLA_VERSION
             value: 0.9.101
         resources:
           limits:
             cpu: '2'
             memory: 2G
           requests:
             cpu: '2'
             memory: 2G
             ```

**Attach the `zilla.yaml` config file:**

My config looks like
```yaml
name: zilla-ohlc-prod-1440min
   telemetry:
     exporters:
       stdout_log_exporter:
         type: stdout
   bindings:
     north_tcp_server:
       type: tcp
       kind: server
       options:
         host: 0.0.0.0
         port:
           - 7114
       routes:
           - when:
               - port: 7114
             exit: north_http_server
     north_http_server:
       type: http
       kind: server
       options:
         access-control:
           policy: cross-origin
       routes:
         - when:
             - headers:
                 :scheme: http
                 :path: /events/*
             - headers:
                 :scheme: http
                 :path: /events
           exit: north_sse_server
     north_sse_server:
       type: sse
       kind: server
       exit: north_sse_kafka_mapping
     north_sse_kafka_mapping:
       type: sse-kafka
       kind: proxy
       routes:
         - when:
             - path: /events/key/ohlc_1440/{id}
           with:
             topic: ohlc_1440
             filters:
               - key: ${params.id}
           exit: north_kafka_cache_client
     north_kafka_cache_client:
       type: kafka
       kind: cache_client
       exit: south_kafka_cache_server
     south_kafka_cache_server:
       type: kafka
       kind: cache_server
       options:
         bootstrap: 
           - ohlc_1440
         topics:
           - name: ohlc_1440
             defaultOffset: live
       exit: south_kafka_client
     south_kafka_client:
       type: kafka
       kind: client
       exit: south_tcp_client
     south_tcp_client:
       type: tcp
       kind: client
       options:
         host: broker
         port: 9092
       routes:
         - when:
             - cidr: 0.0.0.0/0

Kafka Environment:
Confluent Kafka

Client Environment:
Chrome browser latest or any other browser

@pnikhil610 pnikhil610 added the bug Something isn't working label Nov 28, 2024
@jfallows jfallows added the needs_triage Unknowns need to be addressed label Dec 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working needs_triage Unknowns need to be addressed
Projects
None yet
Development

No branches or pull requests

2 participants