diff --git a/docs/hw/flits.md b/docs/hw/flits.md new file mode 100644 index 00000000..7633f3ed --- /dev/null +++ b/docs/hw/flits.md @@ -0,0 +1,73 @@ +# Flits + +We start with the _flits_, also called flow control units, which is the smallest unit of data that is sent from node to another (e.g. from router to router, of network interface to router). Flits consists of two types of data: + +1. **Routing information**: Also called the header of the flit, this data is used by the routers to route the flit through the network. The header contains information such as the destination address, the source address, the flit type, etc. + +1. **Payload**: This is the actual data that is being sent from one node to another. Router do not look at the payload, they just forward it to the next hop. + +In traditional NoCs, the links have a fixed width (e.g. 32-bit). Usually, the payload and the information required for routing far exceeds this width. Therefore, the payload is split into multiple data flits and a header flit which encodes the routing information. Based on this information, routers in the network then know how to route the following data flits to the destination. + +In modern NoCs, this way of serializing payloads over the network is not very efficient anymore. With higher bandwidth requirements of the endpoints, the serialization becomes more and more a bottleneck, since the frequency of the links is limited. Further, serialization causes additional latency which is undesirable. Lastly, the overhead of the header flits is not negligible, especially when the payload is small. + +Therefore, In _FlooNoC_ and modern NoCs in general[^1], another approach is taken, which differs in two ways: + +[^1]: The [AMBA CHI](https://developer.arm.com/documentation/ihi0050/latest/) protocol also encodes the entire payload (cachelines in this case) in a single flit including all the information required for routing. + +1. **Flit width**: The width of the flits is not fixed anymore. Instead, the flits can be as wide as the payload to send the payload in a single cycle resp. in a single flit. + +1. **Parallel header**: Instead of sending the header before the payload, the header is sent in parallel to the payload. This way, the link utilization is not degraded by header flits. + +!!! tip "Wires are cheap now" + + You might wonder why this was not used in the first place. The reason is that wires were not as cheap as they are today. Modern technologies now have >10 metal layers which can fit >10000 wires/mm. A very good source on this topic, which has also influenced the design of _FlooNoC_ is the NOCS keynote [Reflections on 21 Years of NoCS](https://www.youtube.com/watch?v=Nk3oQm9NxcY) from Bill Dally, one of the pioneers in early NoC research. + +Below, we will discuss the header and the payload in more detail. + +## Header + +In _FlooNoC_ the header consists of the following fields: + +| Field | Type | Required by | Description | +| ----------- | --------------- | ---------- | --------------------- | +| `dst_id` | `dst_t` or `route_t` | Router | The destination ID of the flit. In the case of source-based routing, it encodes the entire route from source to destination. | +| `src_id` | `dst_t` | Chimney | The source ID of the flit. Used by the Chimney to set the `dst_id` for the response | +| `last` | `logic` | Router | Can be used by the Chimneys that a burst of flits is not interleaved with other flits from other endpoints. If not set the router performs wormhole routing i.e. it locks the direction until flit with the `last` bit set has been sent. | +| `axi_ch` | `axi_ch_e`[^2] | Chimney | Used by the Chimney to encode the type of packet e.g. which AXI channel | +| `rob_req` | `logic` | Chimney | Flag to signal that the flit might need to be reordered in the Chimney | +| `rob_idx` | `rob_idx_t` | Chimney | The index into the reorder buffer of the Chimney | +| `atop` | `logic` | Chimney | Flag to signal that the flit is an ATOP flit and can bypass the reodering logic. | + +!!! info "Additional fields" + + Additional fields can be added to the header if needed. The routers just ignore those fields and forward them to the next router/Chimney. + +!!! example "SystemVerilog macros" + + _FlooNoC_ provides System Verilog macros in `typedef.svh` to generate the header fields such as `FLOO_TYPEDEF_HDR_T`. + +[^2]: The type is arbitrary and depends on the type of Chimney used but it is usually an `enum`. For instance, the single-AXI chimney encodes only 5 channels in `axi_ch_e`, while the narrow-wide AXI chimney encodes 10 channels in `nw_ch_e`. This can also be extended. + +## Payload + +The payload itself can be any data that needs to be sent from one node to another. The payload is not interpreted by the routers, they just forward it to the next hop. The payload can be of any width. Usually, the payload consists of an AXI request or response of one of the 5 channels. + +## Types of flits + +The entire flit is constructed by concatenating the header and the payload. One can define many types of flits i.e. one for every payload. The only requirement is that the flit includes the following fields: + +| Field | Type | Description | +| ----------- | --------------- | --------------------- | +| `hdr` | `hdr_t` | The header of the flit, which is identical across the whole network | +| `payload` | `payload_t`[^3] | The payload of the flit, which can be of any width | +| `rsvd` | `logic[x:0]` | Optional padding bits if the flit is smaller than the link width | + +[^3]: The payload type can be anything. For instance, sending an AXI AW is done by defining a `axi_aw_chan_t` struct for the payload type. + +!!! info "Additional fields" + + Again, additional fields can be added if needed. The only requirement is that the width of the flit matches exactly the width of the link. + +!!! example "SystemVerilog macros" + + _FlooNoC_ provides System Verilog macros in `typedef.svh` to generate the flits such as `FLOO_TYPEDEF_FLIT_T` for flits with the `rsvd` field and generic flits `FLOO_TYPEDEF_GENERIC_FLIT_T` without a `rsvd` field. diff --git a/docs/hw/links.md b/docs/hw/links.md index d5cb8999..17622ae0 100644 --- a/docs/hw/links.md +++ b/docs/hw/links.md @@ -1,27 +1 @@ -# Link-Level transport - -## Flits - -We start with the _flits_, also called flow control units, which is the smallest unit of data that is sent from node to another (e.g. from router to router, of network interface to router). Flits consists of two types of data: - -1. **Routing information**: Also called the header of the flit, this data is used by the routers to route the flit through the network. The header contains information such as the destination address, the source address, the flit type, etc. - -1. **Payload**: This is the actual data that is being sent from one node to another. Router do not look at the payload, they just forward it to the next hop. - -In traditional NoCs, the links have a fixed width (e.g. 32-bit). Usually, the payload and the information required for routing far exceeds this width. Therefore, the payload is split into multiple data flits and a header flit which encodes the routing information. Based on this information, routers in the network then know how to route the following data flits to the destination. - -In modern NoCs, this way of serializing payloads over the network is not very efficient anymore. With higher bandwidth requirements of the endpoints, the serialization becomes more and more a bottleneck, since the frequency of the links is limited. Further, serialization causes additional latency which is undesirable. Lastly, the overhead of the header flits is not negligible, especially when the payload is small. - -Therefore, In _FlooNoC_ and modern NoCs in general, another approach is taken, which differs in two ways: - -1. **Flit width**: The width of the flits is not fixed anymore. Instead, the flits can be as wide as the payload to send the payload in a single cycle resp. in a single flit. - -1. **Parallel header**: Instead of sending the header before the payload, the header is sent in parallel to the payload. This way, the link utilization is not degraded by header flits. - -!!! info "AMBA CHI" - - _FlooNoC_ is not the first NoC to use this approach. For example, the [AMBA CHI](https://developer.arm.com/documentation/ihi0050/latest/) protocol also encodes the entire payload (cachelines in this case) in a single flit including all the information required for routing. - -!!! tip "Wires are cheap now" - - You might wonder why this was not used in the first place. The reason is that wires were not as cheap as they are today. Modern technologies now have >10 metal layers which can fit >10000 wires/mm. A very good source on this topic, which has also influenced the design of _FlooNoC_ is the NOCS keynote [Reflections on 21 Years of NoCS](https://www.youtube.com/watch?v=Nk3oQm9NxcY) from Bill Dally, one of the pioneers in early NoC research. +# Links diff --git a/mkdocs.yml b/mkdocs.yml index 2c66c76e..fea9145a 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -51,16 +51,18 @@ plugins: enable_creation_date: true markdown_extensions: - - pymdownx.superfences - - pymdownx.tabbed: - alternate_style: true + - admonition - attr_list + - footnotes + - md_in_html + - pymdownx.details - pymdownx.emoji: emoji_index: !!python/name:material.extensions.emoji.twemoji emoji_generator: !!python/name:material.extensions.emoji.to_svg - - md_in_html - - admonition - - pymdownx.details + - pymdownx.superfences + - pymdownx.tabbed: + alternate_style: true + - tables nav: - Home: @@ -71,6 +73,7 @@ nav: - Getting Started: getting_started.md - Hardware IPs: - Overview: hw/overview.md + - Flits: hw/flits.md - Links: hw/links.md - Routing Algorithms: hw/route_algos.md - Network Interfaces: hw/chimneys.md