[Core] Support for Scheduling-defined Prefill-Decode Disaggregation feature #15

Xinyi-ECNU · 2024-08-23T08:54:44Z

Design for introducing cluster-level prefill-decode disaggregation design to Llumnix. Based on dynamic rescheduling of requests in Llumnix, this design allows Llumnix to manage prefill/decoding instances and the scheduling of requests on these instances. Specifically, this PR designs broader scheduling semantics, enabling the rules for PD disaggregation to be expressed as customized policies within Llumnix.

llumnix/arg_utils.py

llumnix/backends/backend_interface.py

llumnix/backends/vllm/llm_engine.py

llumnix/backends/vllm/scheduler.py

llumnix/backends/vllm/llm_engine.py

llumnix/backends/vllm/scheduler.py

llumnix/global_scheduler/global_scheduler.py

llumnix/global_scheduler/migration_scheduler.py

llumnix/llumlet/local_migration_scheduler.py

zhypku

I'd like to call this feature as: scheduling-defined pdd :)

llumnix/arg_utils.py

llumnix/backends/backend_interface.py

llumnix/backends/vllm/llm_engine.py

llumnix/global_scheduler/global_scheduler.py

llumnix/backends/vllm/scheduler.py

llumnix/config.py

llumnix/global_scheduler/migration_scheduler.py

llumnix/llm_engine_manager.py

llumnix/global_scheduler/migration_scheduler.py

CLAassistant · 2024-08-28T03:20:16Z

All committers have signed the CLA.

llumnix/backends/backend_interface.py

llumnix/config.py

llumnix/global_scheduler/global_scheduler.py

llumnix/global_scheduler/migration_scheduler.py

llumnix/llm_engine_manager.py

llumnix/llumlet/llumlet.py

llumnix/llumlet/migration_coordinator.py

llumnix/backends/vllm/scheduler.py

llumnix/llumlet/request.py

llumnix/llumlet/llumlet.py

llumnix/llumlet/local_migration_scheduler.py

llumnix/global_scheduler/migration_scheduler.py

llumnix/backends/backend_interface.py

llumnix/llumlet/request.py

llumnix/llumlet/llumlet.py

llumnix/global_scheduler/migration_scheduler.py

github-actions · 2024-09-24T13:31:39Z

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	184.00 MB	192.00 MB	200.00 MB	208.00 MB	224.00 MB	232.00 MB	280.00 MB	288.00 MB	312.00 MB	344.00 MB	368.00 MB	416.00 MB	424.00 MB	432.00 MB	440.00 MB	496.00 MB	720.00 MB	912.00 MB
rpc_speed(GB/s)	1.05	1.53	1.79	1.95	2.04	2.18	2.18	2.22	2.24	2.34	2.34	2.41	2.43	2.40	2.49	2.51	2.62	2.34	2.34	2.59	2.52	2.50	2.52	2.46	2.64	2.78	2.18	2.49	2.80	2.59	2.76	2.92	3.18	3.05	3.13	3.17	2.93	3.45	3.32	3.06

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	192.00 MB	200.00 MB	240.00 MB	280.00 MB	312.00 MB	384.00 MB	416.00 MB	464.00 MB	480.00 MB	488.00 MB	536.00 MB	544.00 MB	696.00 MB
gloo_speed(GB/s)	0.92	1.60	1.98	2.28	2.44	2.78	2.72	2.99	2.64	2.91	2.90	2.57	2.62	2.68	3.41	2.62	2.36	2.25	2.19	2.13	2.18	2.50	2.92	2.90	3.40	2.56	2.38	2.97	2.51	2.68	2.84	2.63	2.73	2.64	2.81

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	184.00 MB	192.00 MB	208.00 MB	232.00 MB	240.00 MB	256.00 MB	280.00 MB	312.00 MB	320.00 MB	416.00 MB	424.00 MB	448.00 MB	464.00 MB	488.00 MB	528.00 MB	536.00 MB	752.00 MB
nccl_speed(GB/s)	0.19	0.45	0.68	0.85	1.14	1.33	1.50	1.52	2.00	1.76	1.76	2.24	2.38	2.63	2.26	2.63	2.86	2.58	3.38	3.44	4.23	3.41	3.62	3.31	3.22	4.43	5.22	4.46	2.65	2.94	4.44	5.66	5.35	5.49	3.73	3.54	5.44	3.99	3.96

github-actions · 2024-09-24T13:49:27Z

prefill	p25	p50	p75	p95	p99	mean
latency(ms)	32545.25	99194.50	192927.00	260004.45	292360.40	114263.90

decode	p25	p50	p75	p95	p99	mean
latency(ms)	54.14	59.66	73.90	135.57	231.50	72.70

github-actions · 2024-09-25T04:05:25Z

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	184.00 MB	192.00 MB	200.00 MB	216.00 MB	232.00 MB	240.00 MB	248.00 MB	272.00 MB	312.00 MB	352.00 MB	376.00 MB	424.00 MB	464.00 MB	480.00 MB	536.00 MB	544.00 MB	560.00 MB
rpc_speed(GB/s)	1.03	1.52	1.75	1.90	1.99	2.11	2.03	2.13	2.16	2.21	2.28	2.35	2.32	2.36	2.44	2.40	2.42	2.37	2.42	2.39	2.46	2.39	2.64	2.45	2.32	2.61	2.58	2.27	2.63	2.48	2.76	2.72	2.91	2.78	2.93	2.96	3.01	3.08	2.99

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	184.00 MB	192.00 MB	200.00 MB	224.00 MB	264.00 MB	312.00 MB	320.00 MB	416.00 MB	424.00 MB	480.00 MB	536.00 MB
gloo_speed(GB/s)	0.94	1.56	2.08	2.27	2.47	2.53	2.69	2.91	3.05	2.95	2.82	3.05	3.31	3.30	2.61	2.37	2.73	3.03	2.74	2.42	2.36	2.61	3.02	3.19	3.02	2.83	2.57	2.53	2.40	2.85	2.29	2.60	2.61

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	184.00 MB	192.00 MB	208.00 MB	232.00 MB	280.00 MB	312.00 MB	336.00 MB	416.00 MB	424.00 MB	432.00 MB	472.00 MB	480.00 MB	536.00 MB	560.00 MB	568.00 MB
nccl_speed(GB/s)	0.19	0.45	0.68	0.87	1.04	1.20	1.44	1.67	1.92	1.93	1.96	2.17	2.38	2.23	2.33	2.68	2.90	3.11	2.72	3.08	2.83	4.17	2.83	3.27	5.00	4.47	2.73	3.18	3.45	5.82	6.39	3.52	5.66	7.48	6.22	5.66	4.20

github-actions · 2024-09-25T04:25:56Z

prefill	p25	p50	p75	p95	p99	mean
latency(ms)	28162.75	101213.50	206765.75	237777.95	250164.08	111382.05

decode	p25	p50	p75	p95	p99	mean
latency(ms)	53.26	59.09	72.60	130.21	401.89	75.28

llumnix/arg_utils.py

llumnix/global_scheduler/migration_scheduler.py

github-actions · 2024-10-08T07:29:29Z

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	184.00 MB	192.00 MB	200.00 MB	208.00 MB	216.00 MB	224.00 MB	232.00 MB	280.00 MB	312.00 MB	416.00 MB	424.00 MB	432.00 MB	464.00 MB	472.00 MB	536.00 MB	552.00 MB	912.00 MB
rpc_speed(GB/s)	1.08	1.59	1.86	2.01	2.11	2.18	2.23	2.29	2.30	2.32	2.46	2.43	2.47	2.34	2.63	2.54	2.53	2.49	2.59	2.64	2.71	2.43	2.52	2.51	2.64	2.73	2.43	2.59	2.78	2.88	2.71	3.00	3.11	3.07	3.31	3.12	3.18	3.26	3.73

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	184.00 MB	192.00 MB	200.00 MB	208.00 MB	216.00 MB	224.00 MB	240.00 MB	256.00 MB	264.00 MB	312.00 MB	416.00 MB	464.00 MB	480.00 MB	496.00 MB	560.00 MB	568.00 MB
gloo_speed(GB/s)	0.95	1.55	2.02	2.23	2.31	2.59	2.67	2.72	2.81	3.20	3.21	2.81	3.45	3.11	3.12	2.81	2.83	2.16	1.85	2.66	2.14	2.46	2.20	2.66	2.06	3.30	2.52	3.27	3.69	1.55	2.86	2.35	2.72	2.47	2.67	2.77	2.75	2.83

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	184.00 MB	192.00 MB	200.00 MB	240.00 MB	280.00 MB	312.00 MB	400.00 MB	416.00 MB	424.00 MB	432.00 MB	464.00 MB	480.00 MB	536.00 MB	560.00 MB
nccl_speed(GB/s)	0.18	0.43	0.67	0.99	1.03	1.24	1.43	1.48	1.92	1.81	2.48	2.22	2.59	2.40	2.32	2.59	3.00	3.18	3.41	3.38	3.40	3.40	3.49	3.38	3.49	3.83	2.55	3.04	5.52	5.22	5.93	4.58	4.94	5.32	3.97	3.49

github-actions · 2024-10-08T07:46:24Z

prefill	p25	p50	p75	p95	p99	mean
latency(ms)	28845.75	126632.50	179699.25	235444.65	245677.13	110102.76

decode	p25	p50	p75	p95	p99	mean
latency(ms)	53.66	60.27	74.86	141.30	329.02	74.69

llumnix/arg_utils.py

github-actions · 2024-10-08T11:18:55Z

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	184.00 MB	192.00 MB	200.00 MB	208.00 MB	224.00 MB	240.00 MB	264.00 MB	280.00 MB	312.00 MB	320.00 MB	416.00 MB	480.00 MB	560.00 MB	704.00 MB
rpc_speed(GB/s)	1.05	1.55	1.78	1.91	2.03	2.06	2.10	2.18	2.25	2.24	2.30	2.35	2.31	2.44	2.41	2.44	2.46	2.51	2.45	2.52	2.55	2.42	2.37	2.64	2.46	2.42	2.69	2.52	2.63	2.82	2.83	2.65	2.99	2.94	3.08	3.32

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	192.00 MB	200.00 MB	232.00 MB	280.00 MB	312.00 MB	320.00 MB	416.00 MB	448.00 MB	456.00 MB	464.00 MB	480.00 MB	536.00 MB	696.00 MB
gloo_speed(GB/s)	0.91	1.55	1.98	2.18	2.22	2.53	2.60	2.78	2.84	2.99	2.94	2.91	2.83	2.47	2.98	2.54	2.82	2.35	2.94	2.30	1.72	2.23	3.34	1.50	2.63	2.60	2.29	2.41	2.70	2.77	2.70	2.47	2.41	2.62	2.84

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	184.00 MB	192.00 MB	200.00 MB	232.00 MB	312.00 MB	320.00 MB	384.00 MB	416.00 MB	424.00 MB	464.00 MB	480.00 MB	536.00 MB	544.00 MB	576.00 MB
nccl_speed(GB/s)	0.19	0.46	0.73	0.98	1.09	1.30	1.63	1.70	1.95	1.78	2.14	2.15	2.98	2.60	2.42	2.67	2.93	3.16	2.95	3.59	3.64	3.52	2.62	3.97	5.17	3.01	4.23	4.01	5.04	5.64	6.17	5.41	5.80	4.02	3.99	4.35

github-actions · 2024-10-08T11:37:46Z

prefill	p25	p50	p75	p95	p99	mean
latency(ms)	25627.75	106297.00	172871.00	245538.05	281652.55	109409.63

decode	p25	p50	p75	p95	p99	mean
latency(ms)	54.52	61.77	80.36	155.31	401.34	78.39

github-actions · 2024-10-08T11:52:05Z

prefill	p25	p50	p75	p95	p99	mean
latency(ms)	31386.00	79725.00	174082.25	252759.10	307512.08	104876.12

decode	p25	p50	p75	p95	p99	mean
latency(ms)	54.12	60.87	77.45	144.30	440.16	79.42

github-actions · 2024-10-08T12:22:09Z

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	184.00 MB	192.00 MB	200.00 MB	208.00 MB	216.00 MB	232.00 MB	240.00 MB	248.00 MB	312.00 MB	320.00 MB	416.00 MB	448.00 MB	464.00 MB	480.00 MB	528.00 MB
rpc_speed(GB/s)	1.05	1.55	1.75	1.95	2.04	2.14	2.16	2.15	2.27	2.24	2.36	2.40	2.36	2.38	2.44	2.46	2.52	2.46	2.52	2.49	2.58	2.45	2.49	2.36	2.45	2.45	2.61	2.50	2.59	2.62	2.78	2.85	2.90	3.12	2.88	2.81	3.21

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	184.00 MB	192.00 MB	200.00 MB	232.00 MB	240.00 MB	272.00 MB	280.00 MB	312.00 MB	416.00 MB	472.00 MB	480.00 MB	536.00 MB	552.00 MB
gloo_speed(GB/s)	0.94	1.59	2.09	2.22	2.47	2.63	2.87	2.90	2.83	2.90	2.78	3.21	3.27	3.20	3.07	3.00	2.76	1.63	2.64	2.70	1.79	2.50	2.77	1.57	2.84	2.87	2.70	2.77	2.36	2.08	2.98	2.29	2.44	2.46	2.47

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	184.00 MB	192.00 MB	200.00 MB	248.00 MB	280.00 MB	312.00 MB	320.00 MB	416.00 MB	424.00 MB	464.00 MB	480.00 MB	536.00 MB	552.00 MB	560.00 MB	912.00 MB
nccl_speed(GB/s)	0.20	0.56	0.73	0.81	1.24	1.29	1.39	1.68	1.80	1.78	1.90	1.92	1.75	2.70	2.92	2.72	2.91	3.44	3.72	3.98	2.29	3.32	3.22	3.68	3.43	3.82	2.62	3.43	4.49	5.75	6.05	5.84	6.35	4.66	4.13	5.22	4.81

llumnix/global_scheduler/global_scheduler.py

llumnix/global_scheduler/dispatch_scheduler.py

llumnix/llumlet/llumlet.py

github-actions · 2024-10-14T03:45:35Z

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	176.00 MB	192.00 MB	200.00 MB	312.00 MB	368.00 MB	472.00 MB	560.00 MB	584.00 MB	736.00 MB
rpc_speed(GB/s)	1.04	1.53	1.75	1.92	1.97	2.09	2.13	2.13	2.18	2.26	2.27	2.23	2.38	2.30	2.36	2.43	2.34	2.40	2.58	2.42	2.37	2.42	2.69	2.67	2.75	2.99	3.09	3.28	3.30

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	184.00 MB	312.00 MB	320.00 MB	336.00 MB	472.00 MB	560.00 MB	688.00 MB
gloo_speed(GB/s)	0.95	1.52	1.93	2.02	2.37	2.41	2.41	2.44	2.55	3.01	3.01	3.53	3.08	3.17	3.56	2.75	1.82	2.67	2.61	2.84	2.36	2.57	2.89	2.37	2.36	2.58	2.50	1.84	2.39

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	184.00 MB	192.00 MB	280.00 MB	288.00 MB	312.00 MB	480.00 MB	536.00 MB	544.00 MB	576.00 MB	696.00 MB	736.00 MB	960.00 MB
nccl_speed(GB/s)	0.19	0.44	0.67	0.90	0.95	1.24	1.24	1.60	1.69	1.89	2.19	1.87	2.36	2.46	2.61	2.53	2.80	2.71	3.57	3.25	3.34	4.82	4.98	3.63	2.60	2.98	5.51	5.19	3.99	4.83	4.42	4.70	4.95	4.68

github-actions · 2024-10-14T03:55:52Z

prefill	p25	p50	p75	p95	p99	mean
latency(ms)	26800.25	87265.37	157870.81	294178.67	322501.43	111403.61

decode	p25	p50	p75	p95	p99	mean
latency(ms)	52.59	57.49	68.09	108.79	165.82	64.83

zhypku reviewed Aug 23, 2024

View reviewed changes

Xinyi-ECNU requested a review from zhypku August 26, 2024 06:02

zhypku reviewed Aug 26, 2024

View reviewed changes

zhypku requested review from s5u13b, KuilongCui and ZeldaHuang August 26, 2024 07:05

zhypku reviewed Aug 26, 2024

View reviewed changes

KuilongCui reviewed Aug 26, 2024

View reviewed changes

KuilongCui reviewed Aug 27, 2024

View reviewed changes

Xinyi-ECNU changed the title ~~[Core] Support for Prefill-Decode Disaggregation feature~~ [Core] Support for Scheduling-defined Prefill-Decode Disaggregation feature Aug 27, 2024

Xinyi-ECNU force-pushed the pd_disagg branch from 598f08b to c28584b Compare August 28, 2024 03:28

Xinyi-ECNU force-pushed the pd_disagg branch from c28584b to 16a05d1 Compare September 5, 2024 03:26

s5u13b reviewed Sep 6, 2024

View reviewed changes

ZeldaHuang reviewed Sep 6, 2024

View reviewed changes

llumnix/llumlet/local_migration_scheduler.py Show resolved Hide resolved

Xinyi-ECNU force-pushed the pd_disagg branch from ab199a4 to f50fa8d Compare September 9, 2024 08:57

s5u13b reviewed Sep 9, 2024

View reviewed changes

llumnix/global_scheduler/migration_scheduler.py Outdated Show resolved Hide resolved

zhypku reviewed Sep 10, 2024

View reviewed changes

llumnix/global_scheduler/migration_scheduler.py Show resolved Hide resolved

Xinyi-ECNU force-pushed the pd_disagg branch from 937afce to c921ef9 Compare September 11, 2024 03:06

KuilongCui reviewed Sep 11, 2024

View reviewed changes

Xinyi-ECNU force-pushed the pd_disagg branch from 3c6166e to 749a93f Compare September 24, 2024 01:59

KuilongCui reviewed Sep 25, 2024

View reviewed changes

llumnix/arg_utils.py Show resolved Hide resolved

llumnix/global_scheduler/migration_scheduler.py Outdated Show resolved Hide resolved

Xinyi-ECNU force-pushed the pd_disagg branch from 3e1623b to 2e2401b Compare October 8, 2024 07:08

KuilongCui approved these changes Oct 8, 2024

View reviewed changes

llumnix/arg_utils.py Show resolved Hide resolved

s5u13b reviewed Oct 8, 2024

View reviewed changes

llumnix/arg_utils.py Outdated Show resolved Hide resolved

s5u13b approved these changes Oct 8, 2024

View reviewed changes

Xinyi-ECNU force-pushed the pd_disagg branch from 14655be to 466351a Compare October 8, 2024 10:56

ZeldaHuang reviewed Oct 9, 2024

View reviewed changes

llumnix/global_scheduler/global_scheduler.py Outdated Show resolved Hide resolved

llumnix/global_scheduler/dispatch_scheduler.py Show resolved Hide resolved

llumnix/llumlet/llumlet.py Outdated Show resolved Hide resolved

llumnix/llumlet/llumlet.py Show resolved Hide resolved

Xinyi-ECNU and others added 13 commits October 14, 2024 10:13

rebase main

964d7c0

fix

12581b8

fix

0c4b8ca

fix

371de1d

fix

043da6f

resolve conflict

3e95ded

fix

8aa7d3a

fix

50b2717

fix

2a4502c

fix

6303527

rebase

2f781cf

fix

dea6812

rebase main

d2def02

Xinyi-ECNU force-pushed the pd_disagg branch from 466351a to d2def02 Compare October 14, 2024 03:31

Xinyi-ECNU merged commit ce45945 into main Oct 14, 2024
14 checks passed

Xinyi-ECNU deleted the pd_disagg branch October 18, 2024 02:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Core] Support for Scheduling-defined Prefill-Decode Disaggregation feature #15

[Core] Support for Scheduling-defined Prefill-Decode Disaggregation feature #15

Xinyi-ECNU commented Aug 23, 2024 •

edited

Loading

zhypku left a comment

CLAassistant commented Aug 28, 2024 •

edited

Loading

github-actions bot commented Sep 24, 2024

github-actions bot commented Sep 24, 2024

github-actions bot commented Sep 25, 2024

github-actions bot commented Sep 25, 2024

github-actions bot commented Oct 8, 2024

github-actions bot commented Oct 8, 2024

github-actions bot commented Oct 8, 2024

github-actions bot commented Oct 8, 2024

github-actions bot commented Oct 8, 2024

github-actions bot commented Oct 8, 2024

github-actions bot commented Oct 14, 2024

github-actions bot commented Oct 14, 2024

[Core] Support for Scheduling-defined Prefill-Decode Disaggregation feature #15

[Core] Support for Scheduling-defined Prefill-Decode Disaggregation feature #15

Conversation

Xinyi-ECNU commented Aug 23, 2024 • edited Loading

zhypku left a comment

Choose a reason for hiding this comment

CLAassistant commented Aug 28, 2024 • edited Loading

github-actions bot commented Sep 24, 2024

github-actions bot commented Sep 24, 2024

github-actions bot commented Sep 25, 2024

github-actions bot commented Sep 25, 2024

github-actions bot commented Oct 8, 2024

github-actions bot commented Oct 8, 2024

github-actions bot commented Oct 8, 2024

github-actions bot commented Oct 8, 2024

github-actions bot commented Oct 8, 2024

github-actions bot commented Oct 8, 2024

github-actions bot commented Oct 14, 2024

github-actions bot commented Oct 14, 2024

Xinyi-ECNU commented Aug 23, 2024 •

edited

Loading

CLAassistant commented Aug 28, 2024 •

edited

Loading