[doc][dpapp] Initial bmv2 dpapp HLD #606

jimmyzhai · 2024-07-23T17:00:03Z

The document is a high level design doc of BMv2 data plane app. As a proof of concept, it describes the design of a data-plane app example, how it cooperates with DASH pipeline BMv2 to implement DASH data plane. The app will be based on VPP.

DPAPP is another packet processing engine running on CPUs. It adds extra capacities onto DASH capable ASIC:

It serves the slow (exception) path of packet processing to implement complicated pipeline logic, which is hard to be done in ASIC, for example adding/updating/removing table entry inline.
It optionally implements some DASH-SAI APIs, which are not proper/easy to be done upon ASIC SDK.

KrisNey-MSFT · 2024-07-30T21:29:34Z

Looks like this check is failing @jimmyzhai - let us know (or attend the Community call tomorrow at 9am Pacific) if you need help

chrispsommers · 2024-08-14T16:08:17Z

documentation/dataplane/dash-bmv2-data-plane-app.md

+- vpp workers, they serve as an exception path of packet processing, running on multi-cpus. It creates a flow in local flow table and notifies dashsai server to offload it to BMv2 flow table. The packet is temporarily queued. After workers know the success of flow offloading to BMv2, they deque the packet and send it back to P4 pipeline via VPP port. The workers also do flow age-out task with proper scheduling.
+- flow table, is a local cache of BMv2 flow table.
+- DASH SAI, is a unique interface for DASH object CRUD of DASH pipeline, implemented by DASH BMv2.
+- VPP port, is a veth interface and connects to BMv2 via veth pair. It serves as datapath channel to receive/send all packets between date-plane app and BMv2. Generally the port supports multi RSS queues, each queue binds to one vpp worker.


So this impacts the current bmv2 model and requires adding two new veth ports to it? Can you update the diagrams to show the veth names?

Yes, veth4 on bmv2 and veth5 on dpapp. I think the veth port names are implementation details. I'll later update them in dash-pipeline README instead of this HLD.

KrisNey-MSFT · 2024-08-19T19:51:58Z

@r12f for review

jimmyzhai · 2024-08-20T17:03:37Z

Looks like this check is failing @jimmyzhai - let us know (or attend the Community call tomorrow at 9am Pacific) if you need help

Fixed

r12f · 2024-08-29T00:07:38Z

.wordlist.txt

@@ -100,6 +100,8 @@ configs
 Conntrack
 Containerlab
 CP
+cpus


it might be better to get this one removed : D

r12f · 2024-08-29T00:23:37Z

documentation/dataplane/dash-bmv2-data-plane-app.md

+## 3. Project scenario
+
+### 3.1. Stateful packet process - flow
+- Flow Creation<br>


we can use 2 enters + indent instead of
. it is preferred to not having html tags in the markdown docs.

r12f · 2024-08-29T00:25:06Z

documentation/dataplane/dash-bmv2-data-plane-app.md

+
+### 3.1. Stateful packet process - flow
+- Flow Creation<br>
+In DASH pipeline, after 5-tuple flow keys are well extracted, packet goes to flow lookup stage. It does the flow lookup. If any flow is matched, packet is marked a flow-hit flag, otherwise flow-miss flag. The packet continues to go to next stages, like ACL, (destination) NAT, routing, etc. After routing stage, if route is found and packet is flow-miss, it will bypass the rest stages and be forwarded to data-plane app. The data-plane app will use dash-sai APIs to create flow in flow table, and then re-inject the packet back to pipeline.


"after 5-tuple flow keys" -> after flow keys....

this is because flow keys might not be 5 tuples, e.g. 3 tuples based on business needs.

r12f · 2024-08-29T00:26:12Z

documentation/dataplane/dash-bmv2-data-plane-app.md

+
+### 3.1. Stateful packet process - flow
+- Flow Creation<br>
+In DASH pipeline, after 5-tuple flow keys are well extracted, packet goes to flow lookup stage. It does the flow lookup. If any flow is matched, packet is marked a flow-hit flag, otherwise flow-miss flag. The packet continues to go to next stages, like ACL, (destination) NAT, routing, etc. After routing stage, if route is found and packet is flow-miss, it will bypass the rest stages and be forwarded to data-plane app. The data-plane app will use dash-sai APIs to create flow in flow table, and then re-inject the packet back to pipeline.


When flow is hit, the packet should by pass the policy matching stages, such as ACL, routing and etc.

r12f · 2024-08-29T00:27:17Z

documentation/dataplane/dash-bmv2-data-plane-app.md

+- Flow Creation<br>
+In DASH pipeline, after 5-tuple flow keys are well extracted, packet goes to flow lookup stage. It does the flow lookup. If any flow is matched, packet is marked a flow-hit flag, otherwise flow-miss flag. The packet continues to go to next stages, like ACL, (destination) NAT, routing, etc. After routing stage, if route is found and packet is flow-miss, it will bypass the rest stages and be forwarded to data-plane app. The data-plane app will use dash-sai APIs to create flow in flow table, and then re-inject the packet back to pipeline.
+- Flow Deletion<br>
+In flow lookup stage, TCP FIN/RST packet is always marked flow-miss and later forwarded to data-plane app. 


tcp FIN and RST packet should not mark as flow miss, as they should not go through the policy matching stages. however, besides applying the SDN transformation, we should notify the DPAPP to get the flow removed by forwarding.

r12f · 2024-08-29T00:29:56Z

documentation/dataplane/dash-bmv2-data-plane-app.md

+
+### 3.2. HA
+- Inline flow replication<br>
+In HA context, Active data-plane app creates flow, replicates the flow in


"replicates the flow in metadata"

will be better to rephrase this part.

r12f · 2024-08-29T00:30:22Z

documentation/dataplane/dash-bmv2-data-plane-app.md

+- Inline flow replication<br>
+In HA context, Active data-plane app creates flow, replicates the flow in
+metadata, glues it with original packet, and sends the packet to Standby
+data-plane app via DPU data-plane channel. Standby data-plane app recreates


"sends the packet to Standby data-plane app" -> "sends the packet to Standby"

this is because the p4 can also be involved.

r12f · 2024-08-29T01:27:29Z

documentation/dataplane/dash-bmv2-data-plane-app.md

+the flow, and acknowledges Active data-plane app to finish flow creation. The
+same logic can apply for flow deletion, flow age-out.
+- Flow bulk sync<br>
+Flow bulk sync replicates batch flows from one DPU to another to make flow table consistency on Active and Standby DPUs. When HA agents starts a bulk sync via DASH SAI, Active data-plane app will walk flow table based on sync method (perfect/range), generate batch flows and send them to Standby data-plane app with gRPC via control-plane channel. Standby date-plane app will create flows in order.


nit: "flow table consistency" -> "flow table consistent"

r12f · 2024-08-29T01:28:57Z

documentation/dataplane/dash-bmv2-data-plane-app.md

+Flow bulk sync replicates batch flows from one DPU to another to make flow table consistency on Active and Standby DPUs. When HA agents starts a bulk sync via DASH SAI, Active data-plane app will walk flow table based on sync method (perfect/range), generate batch flows and send them to Standby data-plane app with gRPC via control-plane channel. Standby date-plane app will create flows in order.
+
+### 3.3. Flow re-simulation
+When SONiC changes polices via DASH SAI, flow could be impacted. Data-plane app is raised to re-simulate flow. In HA context, Active data-plane app also needs to sync the updated flows to Standby.


"flow could be impacted"

better to be more specific. we can say "some flows might need to be updated to get the latest policy applied" or so.

r12f · 2024-08-29T02:32:45Z

documentation/dataplane/dash-bmv2-data-plane-app.md

+
+## 6. Detailed design
+
+Referring to the below figure from [HA API HLD], it greatly outlines the whole packet flow in data plane for both standalone and HA context.


"[HA API HLD]" missing link.

r12f · 2024-08-29T02:35:29Z

documentation/dataplane/dash-bmv2-data-plane-app.md

+When DASH pipeline requests DPAPP for flow creation, it encapsulates DASH metadata in an ethernet frame with EtherType DASH_METADATA and appends the original customer packet. The packet sent to DPAPP is like:
+
+<div style="text-align: center;">
+Ethernet HEADER|DASH metadata|customer packet


this style doesn't seem to work.

Removed this style.

r12f · 2024-08-29T02:42:23Z

documentation/dataplane/dash-bmv2-data-plane-app.md

+Ethernet HEADER|DASH metadata|customer packet
+</div>
+
+ The number of DASH_METADATA is 0x876D, which reuses the number of EtherType SECURE_DATA (vpp/src/vnet/ethernet/types.def at master · FDio/vpp · GitHub).


nit: initial space.

r12f · 2024-08-29T02:45:05Z

documentation/dataplane/dash-bmv2-data-plane-app.md

+   Flow data: As next figure
+```
+
+```


will be better to call out what does the following packet is

claimed in Ethernet HEADER|DASH metadata|customer packet

r12f · 2024-08-29T02:48:25Z

documentation/dataplane/dash-bmv2-data-plane-app.md

+    participant S as DASH SAI Server
+    autonumber
+    C->>+S: ENI_ATTR_FULL_FLOW_RESIMULATION_REQUESTED
+    S->>+R: Call DASH SAI set_eni_attr (epoch) 


I believe we don't have set eni attr to update epoch. we will need to call p4 runtime api directly or something to make it work.

also I believe epoch is named as flow version.

Add epoch as an internal attribute of eni and flow, which is invisible to public SAI.

r12f · 2024-08-29T02:52:18Z

documentation/dataplane/dash-bmv2-data-plane-app.md

+
+### 6.4. HA flow
+Base on basic flow, HA flow adds an extra FLOW_SYNCED state, which involves
+extra sync for request/response ping-pang between DPAPP and PEER DPAPP.


nit: ping-pong

Following dpapp HLD - #606, this is the 1st part implementation to add dpapp target in Makefile, which is implemented as a vpp plugin.

r12f · 2024-09-04T02:37:50Z

documentation/dataplane/dash-bmv2-data-plane-app.md

+- Flow Age-out
+
+  In flow lookup stage, if packet hits one flow, it will refresh flow
+  timestamp. Data-plane app periodically scans flow table and check if flow is


data-plane app can help the data plane implementing the flow age-out mechanism by bridging the gap in the current data plane engine.

"e.g. ......"

KrisNey-MSFT · 2024-09-04T20:59:48Z

Just curious, is this related to PR464 ?
BYO (Bring-Your-Own) data plane app HLD

r12f · 2024-09-10T01:53:59Z

documentation/dataplane/dash-bmv2-data-plane-app.md

+
+```
+   0                   1                   2                   3   
+   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 


we can use the mermaid graph to simplify this graph now, which is just available on github.

mermaid packet diagram is still in beta, not support varbit representation. I'll prefer to it after it has this feature.

i see. sounds good!

KrisNey-MSFT · 2024-09-10T17:15:00Z

Just curious, is this related to PR464 ? BYO (Bring-Your-Own) data plane app HLD

ping @jimmyzhai and @r12f

r12f · 2024-09-11T01:29:15Z

lgtm now! thanks Junhua!

jimmyzhai · 2024-09-11T02:05:35Z

Just curious, is this related to PR464 ? BYO (Bring-Your-Own) data plane app HLD

ping @jimmyzhai and @r12f

No. BYO (Bring-Your-Own) data plane app HLD is a design guideline for customer owned data plane app.

Following dpapp HLD - #606, this is the 2nd part implementation. It updates bmv2 pipeline to have stateful packet processing.

Following dpapp HLD - #606, this is the 3th part implementation. It implements the basic logic of inline flow creation, deletion and ageout. - Build with command `make dpapp` - Run with command `make run-dpapp`

jimmyzhai changed the title ~~[dpapp] Initial bmv2 dpapp HLD~~ [doc][dpapp] Initial bmv2 dpapp HLD Jul 23, 2024

This was referenced Jul 23, 2024

[dpapp] add data plane app target #607

Merged

[dpapp] Add flow in bmv2 pipeline #608

Merged

[dpapp] Initial dpapp implementation being a vpp plugin #609

Merged

KrisNey-MSFT added the help wanted Extra attention is needed label Jul 30, 2024

Initial bmv2 dpapp HLD

56ea32a

jimmyzhai force-pushed the dp_app_hld branch from 129833a to 56ea32a Compare August 14, 2024 14:46

jimmyzhai requested a review from r12f August 14, 2024 16:01

chrispsommers reviewed Aug 14, 2024

View reviewed changes

Fix spelling check error

fdd642c

jimmyzhai force-pushed the dp_app_hld branch from ab68678 to fdd642c Compare August 20, 2024 16:57

jimmyzhai requested a review from chrispsommers August 20, 2024 17:18

r12f reviewed Aug 29, 2024

View reviewed changes

r12f pushed a commit that referenced this pull request Aug 29, 2024

[dpapp] add data plane app target (#607)

a638ad3

Following dpapp HLD - #606, this is the 1st part implementation to add dpapp target in Makefile, which is implemented as a vpp plugin.

Fix per review comments

245ea4d

jimmyzhai force-pushed the dp_app_hld branch from af05a3f to 245ea4d Compare August 30, 2024 03:26

r12f reviewed Sep 4, 2024

View reviewed changes

Fix per comments

1f51ca3

jimmyzhai force-pushed the dp_app_hld branch from ba0ba2f to 1f51ca3 Compare September 4, 2024 13:57

KrisNey-MSFT added Ready to merge? and removed help wanted Extra attention is needed labels Sep 4, 2024

Update flow data figure

781075e

r12f reviewed Sep 10, 2024

View reviewed changes

r12f merged commit 10b5ba5 into sonic-net:main Sep 11, 2024
3 checks passed

jimmyzhai removed the Ready to merge? label Sep 11, 2024

r12f pushed a commit that referenced this pull request Oct 26, 2024

[dpapp] Add flow in bmv2 pipeline (#608)

c3fd153

Following dpapp HLD - #606, this is the 2nd part implementation. It updates bmv2 pipeline to have stateful packet processing.

jimmyzhai deleted the dp_app_hld branch November 21, 2024 02:00


		## 6. Detailed design

		Referring to the below figure from [HA API HLD], it greatly outlines the whole packet flow in data plane for both standalone and HA context.

[doc][dpapp] Initial bmv2 dpapp HLD #606

[doc][dpapp] Initial bmv2 dpapp HLD #606

Conversation

jimmyzhai commented Jul 23, 2024 • edited Loading

KrisNey-MSFT commented Jul 30, 2024

chrispsommers Aug 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KrisNey-MSFT commented Aug 19, 2024

jimmyzhai commented Aug 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KrisNey-MSFT commented Sep 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KrisNey-MSFT commented Sep 10, 2024

r12f commented Sep 11, 2024

jimmyzhai commented Sep 11, 2024

jimmyzhai commented Jul 23, 2024 •

edited

Loading

chrispsommers Aug 14, 2024 •

edited

Loading