Skip to content

Commit

Permalink
refine
Browse files Browse the repository at this point in the history
  • Loading branch information
waitinfuture committed Jul 24, 2023
1 parent 07bcd56 commit 3bc7595
Showing 1 changed file with 8 additions and 2 deletions.
10 changes: 8 additions & 2 deletions docs/developers/trafficcontrol.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ From the `Worker`'s perspective, the income data flow comes from two sources:
The buffered memory can be released when the following conditions are satisfied:

- Data is flushed to file
- If replication is on, after primary data is sent to replica `Worker`
- If replication is on, after primary data is written to wire

The basic idea is that, when `Worker` is under high memory pressure, slow down or stop income data, and at same
time force flush to release memory.
Expand Down Expand Up @@ -68,7 +68,7 @@ last time window. It defines two watermarks:
- `Low Watermark`, under which everything goes OK
- `High Watermark`, when exceeds this, top users will be Congestion Controlled

Celeborn uses `UserIdentifier` to identify users. `Worker` collects bytes pushed for each user in the last time
Celeborn uses `UserIdentifier` to identify users. `Worker` collects bytes pushed from each user in the last time
window. When used direct memory exceeds `High Watermark`, users who occupied more resources than the average
occupation will receive `Congestion Control` message.

Expand All @@ -77,4 +77,10 @@ occupation will receive `Congestion Control` message.
`Congestion Avoidance` phase, which slowly increases push rate. Upon receiving `Congestion Control`, it goes back
to `Slow Start` phase.

`Congestion Control` can be enabled and tuned by the following configurations:

- `celeborn.worker.congestionControl.enabled`
- `celeborn.worker.congestionControl.low.watermark`
- `celeborn.worker.congestionControl.high.watermark`
- `celeborn.worker.congestionControl.sample.time.window`
- `celeborn.worker.congestionControl.user.inactive.interval`

0 comments on commit 3bc7595

Please sign in to comment.