Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

i40e系列网卡 dpvs fullnet不行 #433

Closed
oupeng007 opened this issue Apr 11, 2019 · 8 comments
Closed

i40e系列网卡 dpvs fullnet不行 #433

oupeng007 opened this issue Apr 11, 2019 · 8 comments

Comments

@oupeng007
Copy link

i40e网卡驱动i40e_fdir.c没地方设置tcp/udp input dst_port_mask,导致fullnet返程数据不能根据cpu 个数做掩码，不能命中flow director表项，返程数据hash到了不同的核，造成conn lookup miss
IPVS: new conn: [1] TCP 10.20.130.108/57705 10.102.135.250/80 10.102.135.251/1032 10.102.12.96/80 refs 2
IPVS: state trans: TCP in [S...] 10.20.130.108:57705->10.102.12.96:80 state NONE->SYN_RECV conn.refcnt 2
IPVS: conn lookup: [6] TCP 10.102.12.96/80 -> 10.102.135.251/1032 miss
@zeably
Copy link

zeably commented Apr 26, 2019

把详细的日志贴出来

@oupeng007
Copy link
Author

void netif_mask_fdir_filter(int af, const struct netif_port *port,
struct rte_eth_fdir_filter *filt)
{
struct rte_eth_fdir_info fdir_info;
const struct rte_eth_fdir_masks *fmask;
union rte_eth_fdir_flow *flow = &filt->input.flow;

if (rte_eth_dev_filter_ctrl(port->id, RTE_ETH_FILTER_FDIR,
            RTE_ETH_FILTER_INFO, &fdir_info) < 0) {
    RTE_LOG(WARNING, NETIF, "%s: Fail to fetch fdir info of %s !\n",
            __func__, port->name);
    return;
}
fmask = &fdir_info.mask;

/* ipv4 flow */
if (af == AF_INET) {
    flow->ip4_flow.src_ip &= fmask->ipv4_mask.src_ip;
    flow->ip4_flow.dst_ip &= fmask->ipv4_mask.dst_ip;
    flow->ip4_flow.tos &= fmask->ipv4_mask.tos;
    flow->ip4_flow.ttl &= fmask->ipv4_mask.ttl;
    flow->ip4_flow.proto &= fmask->ipv4_mask.proto;
    flow->tcp4_flow.src_port &= fmask->src_port_mask;
    flow->tcp4_flow.dst_port &= fmask->dst_port_mask;
    return;
}

主要在这段代码中 调用i40e网卡获取fdir_info信息接口，get不到rte_eth_fdir_masks相关信息
查看驱动代码发现
ixgbe_fdir.c中有对掩码的设置接口：fdir_set_input_mask

i40e_fdir.c中没有对掩码进行设置，并且没有实现该接口

@oupeng007
Copy link
Author

把详细的日志贴出来
SAPOOL: FDIR: add dpdk0 IPv4 TCP/UDP ip 10.102.135.251 port 0 (0x0000) mask 0x0007 queue 0 lcore 1 filterID 0/1
SAPOOL: FDIR: add dpdk0 IPv4 TCP/UDP ip 10.102.135.251 port 1 (0x0001) mask 0x0007 queue 1 lcore 2 filterID 0/1
SAPOOL: FDIR: add dpdk0 IPv4 TCP/UDP ip 10.102.135.251 port 2 (0x0002) mask 0x0007 queue 2 lcore 3 filterID 0/1
SAPOOL: FDIR: add dpdk0 IPv4 TCP/UDP ip 10.102.135.251 port 3 (0x0003) mask 0x0007 queue 3 lcore 4 filterID 0/1
SAPOOL: FDIR: add dpdk0 IPv4 TCP/UDP ip 10.102.135.251 port 4 (0x0004) mask 0x0007 queue 4 lcore 5 filterID 0/1
SAPOOL: FDIR: add dpdk0 IPv4 TCP/UDP ip 10.102.135.251 port 5 (0x0005) mask 0x0007 queue 5 lcore 6 filterID 0/1
SAPOOL: FDIR: add dpdk0 IPv4 TCP/UDP ip 10.102.135.251 port 6 (0x0006) mask 0x0007 queue 6 lcore 7 filterID 0/1
SAPOOL: FDIR: add dpdk0 IPv4 TCP/UDP ip 10.102.135.251 port 7 (0x0007) mask 0x0007 queue 7 lcore 8 filterID 0/1
SAPOOL: sa_pool_create: sa pool created
IPVS: conn lookup: [6] TCP 10.102.90.1/8080 -> 10.102.135.173/34302 miss
IPVS: tcp_conn_sched: [6] try sched non-SYN packet: [...R] 10.102.90.1/8080->10.102.135.173/34302
IPVS: conn lookup: [4] TCP 10.21.80.55/9092 -> 10.102.135.173/46758 miss
IPVS: tcp_conn_sched: [4] try sched non-SYN packet: [...R] 10.21.80.55/9092->10.102.135.173/46758
IPVS: conn lookup: [8] TCP 10.21.34.1/9092 -> 10.102.135.173/59050 miss
IPVS: tcp_conn_sched: [8] try sched non-SYN packet: [...R] 10.21.34.1/9092->10.102.135.173/59050
IPVS: conn lookup: [3] TCP 10.30.56.220/7002 -> 10.102.135.173/45310 miss
IPVS: tcp_conn_sched: [3] try sched non-SYN packet: [S.A.] 10.30.56.220/7002->10.102.135.173/45310
IPVS: conn lookup: [3] TCP 10.30.56.220/7002 -> 10.102.135.173/45310 miss
IPVS: tcp_conn_sched: [3] try sched non-SYN packet: [..A.] 10.30.56.220/7002->10.102.135.173/45310
IPVS: conn lookup: [3] TCP 10.30.56.220/7002 -> 10.102.135.173/45310 miss
IPVS: tcp_conn_sched: [3] try sched non-SYN packet: [..A.] 10.30.56.220/7002->10.102.135.173/45310
IPVS: conn lookup: [4] TCP 10.21.33.96/9092 -> 10.102.135.173/54980 miss
IPVS: tcp_conn_sched: [4] try sched non-SYN packet: [...R] 10.21.33.96/9092->10.102.135.173/54980
IPVS: conn lookup: [6] TCP 172.26.24.145/5574 -> 10.102.135.173/51770 miss
IPVS: tcp_conn_sched: [6] try sched non-SYN packet: [S.A.] 172.26.24.145/5574->10.102.135.173/51770
IPVS: conn lookup: [6] TCP 172.26.24.145/5574 -> 10.102.135.173/51770 miss
IPVS: tcp_conn_sched: [6] try sched non-SYN packet: [..A.] 172.26.24.145/5574->10.102.135.173/51770
IPVS: conn lookup: [6] TCP 172.26.24.145/5574 -> 10.102.135.173/51770 miss
IPVS: tcp_conn_sched: [6] try sched non-SYN packet: [..A.] 172.26.24.145/5574->10.102.135.173/51770

就是flow director表项设置问题，i40e网卡底层没有设置端口掩码的接口，造成流不能命中flow director

@oupeng007
Copy link
Author

把详细的日志贴出来

static void
i40e_fdir_info_get(struct rte_eth_dev *dev, struct rte_eth_fdir_info *fdir)
{
struct i40e_pf *pf = I40E_DEV_PRIVATE_TO_PF(dev->data->dev_private);
struct i40e_hw *hw = I40E_PF_TO_HW(pf);
uint16_t num_flex_set = 0;
uint16_t num_flex_mask = 0;

RTE_LOG(DEBUG,PMD, "mode=0x%x,dst_port_mask=0x%x\n",
              dev->data->dev_conf.fdir_conf.mode,
              dev->data->dev_conf.fdir_conf.mask.dst_port_mask);


if (dev->data->dev_conf.fdir_conf.mode == RTE_FDIR_MODE_PERFECT)
	fdir->mode = RTE_FDIR_MODE_PERFECT;
else
	fdir->mode = RTE_FDIR_MODE_NONE;

fdir->guarant_spc =
	(uint32_t)hw->func_caps.fd_filters_guaranteed;
fdir->best_spc =
	(uint32_t)hw->func_caps.fd_filters_best_effort;
fdir->max_flexpayload = I40E_FDIR_MAX_FLEX_LEN;
fdir->flow_types_mask[0] = I40E_FDIR_FLOWS;
int i;
for (i = 1; i < RTE_FLOW_MASK_ARRAY_SIZE; i++)
	fdir->flow_types_mask[i] = 0ULL;
fdir->flex_payload_unit = sizeof(uint16_t);
fdir->flex_bitmask_unit = sizeof(uint16_t);
fdir->max_flex_payload_segment_num = I40E_MAX_FLXPLD_FIED;
fdir->flex_payload_limit = I40E_MAX_FLX_SOURCE_OFF;
fdir->max_flex_bitmask_num = I40E_FDIR_BITMASK_NUM_WORD;

i40e_fdir_info_get_flex_set(pf,
			fdir->flex_conf.flex_set,
			&num_flex_set);
i40e_fdir_info_get_flex_mask(pf,
			fdir->flex_conf.flex_mask,
			&num_flex_mask);

fdir->flex_conf.nb_payloads = num_flex_set;
fdir->flex_conf.nb_flexmasks = num_flex_mask;

}

看这里没有获取fdir->mask

@oupeng007
Copy link
Author

把详细的日志贴出来

static void
ixgbe_fdir_info_get(struct rte_eth_dev *dev, struct rte_eth_fdir_info *fdir_info)
{
struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct ixgbe_hw_fdir_info *info =
IXGBE_DEV_PRIVATE_TO_FDIR_INFO(dev->data->dev_private);
uint32_t fdirctrl, max_num;
uint8_t offset;

fdirctrl = IXGBE_READ_REG(hw, IXGBE_FDIRCTRL);
offset = ((fdirctrl & IXGBE_FDIRCTRL_FLEX_MASK) >>
		IXGBE_FDIRCTRL_FLEX_SHIFT) * sizeof(uint16_t);

fdir_info->mode = dev->data->dev_conf.fdir_conf.mode;
max_num = (1 << (FDIRENTRIES_NUM_SHIFT +
		(fdirctrl & FDIRCTRL_PBALLOC_MASK)));
if (fdir_info->mode >= RTE_FDIR_MODE_PERFECT &&
    fdir_info->mode <= RTE_FDIR_MODE_PERFECT_TUNNEL)
	fdir_info->guarant_spc = max_num;
else if (fdir_info->mode == RTE_FDIR_MODE_SIGNATURE)
	fdir_info->guarant_spc = max_num * 4;

fdir_info->mask.vlan_tci_mask = info->mask.vlan_tci_mask;
fdir_info->mask.ipv4_mask.src_ip = info->mask.src_ipv4_mask;
fdir_info->mask.ipv4_mask.dst_ip = info->mask.dst_ipv4_mask;
IPV6_MASK_TO_ADDR(info->mask.src_ipv6_mask,
		fdir_info->mask.ipv6_mask.src_ip);
IPV6_MASK_TO_ADDR(info->mask.dst_ipv6_mask,
		fdir_info->mask.ipv6_mask.dst_ip);
fdir_info->mask.src_port_mask = info->mask.src_port_mask;
fdir_info->mask.dst_port_mask = info->mask.dst_port_mask;
fdir_info->mask.mac_addr_byte_mask = info->mask.mac_addr_byte_mask;
fdir_info->mask.tunnel_id_mask = info->mask.tunnel_id_mask;
fdir_info->mask.tunnel_type_mask = info->mask.tunnel_type_mask;
fdir_info->max_flexpayload = IXGBE_FDIR_MAX_FLEX_LEN;

if (fdir_info->mode == RTE_FDIR_MODE_PERFECT_MAC_VLAN ||
    fdir_info->mode == RTE_FDIR_MODE_PERFECT_TUNNEL)
	fdir_info->flow_types_mask[0] = 0;
else
	fdir_info->flow_types_mask[0] = IXGBE_FDIR_FLOW_TYPES;

fdir_info->flex_payload_unit = sizeof(uint16_t);
fdir_info->max_flex_payload_segment_num = 1;
fdir_info->flex_payload_limit = IXGBE_MAX_FLX_SOURCE_OFF;
fdir_info->flex_conf.nb_payloads = 1;
fdir_info->flex_conf.flex_set[0].type = RTE_ETH_RAW_PAYLOAD;
fdir_info->flex_conf.flex_set[0].src_offset[0] = offset;
fdir_info->flex_conf.flex_set[0].src_offset[1] = offset + 1;
fdir_info->flex_conf.nb_flexmasks = 1;
fdir_info->flex_conf.flex_mask[0].flow_type = RTE_ETH_FLOW_UNKNOWN;
fdir_info->flex_conf.flex_mask[0].mask[0] =
		(uint8_t)(info->mask.flex_bytes_mask & 0x00FF);
fdir_info->flex_conf.flex_mask[0].mask[1] =
		(uint8_t)((info->mask.flex_bytes_mask & 0xFF00) >> 8);

}

这里有获取fdir_info->mask

@ytwang0320
Copy link
Collaborator

之前发现有这个问题，查看了X710的datasheet，flow director不支持tcp的mask功能，标准协议的mask都无法用软件更改。

@ywc689
Copy link
Collaborator

ywc689 commented Jun 3, 2019

在dpvs.conf中打开 conn/redirect 配置，通过软件的方法可使回来的包命中连接表，不过性能会有所损耗。

@zhutao-jarvan
Copy link

zhutao-jarvan commented Apr 13, 2020 •
edited
Loading

参考这个patch：#440

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@ywc689 @ytwang0320 @zhutao-jarvan @zeably @oupeng007