Linux bridge开启hairpin模拟测试macvlan vepa模式(续)-联通外部网络

通过Linux bridge 开启接口hairpin的方式，模拟macvlan vepa在外部交换支持802.1q的情况下，同一父网卡下的多个子网卡之间是可以通讯的。通过将宿主机物理网卡加入到Linux bridge中，使用bridge桥接内外网络，可以实现内部macvlan vepa子网卡访问外部网络。

石小千

2453人浏览 · 2023-12-26 14:20:19

石小千 · 2023-12-26 14:20:19 发布

之前写了《Linux bridge开启hairpin模拟测试macvlan vepa模式》，记录了Linux bridge开启hairpin，模拟测试macvlan vepa模式下，同一父接口下两个子接口的网络通讯情况，文中缺少了子接口与外部网络通讯的部分，本文补上，详见下：

参考

1.Linux bridge开启hairpin模拟测试macvlan vepa模式
2.Linux虚拟网络设备—之使用Veth pair连接linux网桥bridge
3.Linux 网络设备 - Bridge 详解
4.brctl快速入门与基础

环境

与《Linux bridge开启hairpin模拟测试macvlan vepa模式》环境相同。

测试

1. 测试流程

《Linux bridge开启hairpin模拟测试macvlan vepa模式》只验证了macvlan vepa模式下，同一父网卡下的两个子接口（子网卡）通过开启hairpin的外部交换转发，实现网络通讯。

本文在上述测试基础上，补充Linux bridge开启hairpin模拟测试macvlan vepa时，子接口（子网卡）与外部网络（宿主机以外网络）的通讯情况，测试步骤如下：

详见《Linux bridge开启hairpin模拟测试macvlan vepa模式》中 1.测试流程
补充测试流程：
- 将宿主机网卡enp0s5加入Linux bridge br0
- 删掉网卡enp0s5绑定的IP（10.211.55.18），并将该IP绑定到br0 上
- 修改宿主机默认路由，默认路由设备由enp0s5改为br0
- 增加ns101下默认路由，默认路由设备使用veth0_1.101
- 增加ns102下默认路由，默认路由设备使用veth0_1.102
- 测试macvlan vepa网络子接口与宿主机以外网络的通讯情况

详见下图：

网络连接图
- 物理网卡enp0s5连接外部网络
- br0连接物理网卡enp0s5和虚拟网卡veth0
- 虚拟网卡veth0和veth0_1是一对veth pair
- veth0_1.101和veth0_1.102是veth0的子网卡

在这里插入图片描述

网络通讯图
- veth0_1.101和veth0_1.102，通过br0（veth0接口开启hairpin）转发通讯。
- veth0_1.101和veth0_1.102，通过br0的接口（enp0s5）与宿主机以外网络通讯，例如：网关10.211.55.1

在这里插入图片描述

2. 将宿主机网卡enp0s5加入Linux bridge br0

请务必确保有其他方式可以登录宿主机，网卡enp0s5加入br0后，当前网络连接会断开。

enp0s5 加入br0

// enp0s5 加入 br0
[root@centos7-18 ~]# brctl addif br0 enp0s5
// 查看bridge
[root@centos7-18 ~]# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.001c426087b2       no              enp0s5
                                                        veth0
virbr0          8000.5254009f1377       yes             virbr0-nic
[root@centos7-18 ~]#

3. 删掉网卡enp0s5 IP（10.211.55.18），将该IP绑定到br0 上

删掉宿主机网卡enp0s5绑定的IP 10.211.55.18

// 删掉宿主机网卡enp0s5的IP
[root@centos7-18 ~]# ip addr del 10.211.55.18/24 dev enp0s5

增加br0 绑定IP 10.211.55.18

// br0绑定IP 10.211.55.18
[root@centos7-18 ~]# ip addr add 10.211.55.18/24 dev br0

查看当前IP
- enp0s5已经没有IP了
- br0的IP为10.211.55.18

// 查看当前IP
[root@centos7-18 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether 00:1c:42:60:87:b2 brd ff:ff:ff:ff:ff:ff
    inet6 fdb2:2c26:f4e4:0:21c:42ff:fe60:87b2/64 scope global mngtmpaddr dynamic 
       valid_lft 2581396sec preferred_lft 594196sec
    inet6 fe80::21c:42ff:fe60:87b2/64 scope link 
       valid_lft forever preferred_lft forever
6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:1c:42:60:87:b2 brd ff:ff:ff:ff:ff:ff
    inet 10.211.55.18/24 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fdb2:2c26:f4e4:0:21c:42ff:fe60:87b2/64 scope global mngtmpaddr dynamic 
       valid_lft 2591815sec preferred_lft 604615sec
    inet6 fe80::8413:97ff:fe70:a2e2/64 scope link 
       valid_lft forever preferred_lft forever
7: veth0_1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 7a:87:ef:c6:77:9b brd ff:ff:ff:ff:ff:ff
    inet6 fdb2:2c26:f4e4:0:7887:efff:fec6:779b/64 scope global mngtmpaddr dynamic 
       valid_lft 2591815sec preferred_lft 604615sec
    inet6 fe80::7887:efff:fec6:779b/64 scope link 
       valid_lft forever preferred_lft forever
8: veth0@veth0_1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether 86:08:8e:91:09:fe brd ff:ff:ff:ff:ff:ff
    inet6 fe80::8408:8eff:fe91:9fe/64 scope link 
       valid_lft forever preferred_lft forever
[root@centos7-18 ~]#

4. 修改宿主机默认路由，设备由enp0s5改为br0

删掉宿主机enp0s5的默认路由

// 删掉宿主机网卡enp0s5的IP
[root@centos7-18 ~]# ip route del default via 10.211.55.1 dev enp0s5

增加宿主机br0的默认路由

// 增加宿主机**br0**的默认路由
[root@centos7-18 ~]# ip route add default via 10.211.55.1 dev br0

查看当前路由

// 查看当前路由
[root@centos7-18 ~]# ip route
default via 10.211.55.1 dev br0 
10.211.55.0/24 dev br0 proto kernel scope link src 10.211.55.18 
10.211.55.0/24 dev enp0s6 proto kernel scope link src 10.211.55.21 metric 100 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1

5. 增加namespace ns101和ns102的默认路由

增加ns101下默认路由，设备使用veth0_1.101

// 增加ns101下默认路由，默认路由设备使用veth0_1.101
[root@centos7-18 ~]# ip netns exec ns101 ip route add default via 10.211.55.1 dev veth0_1.101
// 查看ns101路由表 
[root@centos7-18 ~]# ip netns exec ns101 ip route 
default via 10.211.55.1 dev veth0_1.101 
10.211.55.0/24 dev veth0_1.101 proto kernel scope link src 10.211.55.101 
[root@centos7-18 ~]#

增加ns102下默认路由，设备使用veth0_1.102

// 增加ns102下默认路由，默认路由设备使用veth0_1.102
[root@centos7-18 ~]# ip netns exec ns102 ip route add default via 10.211.55.1 dev veth0_1.102 
// 查看ns102路由表 
[root@centos7-18 ~]# ip netns exec ns102 ip route 
default via 10.211.55.1 dev veth0_1.102 
10.211.55.0/24 dev veth0_1.102 proto kernel scope link src 10.211.55.102 
[root@centos7-18 ~]#

6. 测试macvlan vepa网络子接口与宿主机以外网络通讯

检查测试环境-enp0s5、br0、veth0和veth0_1已启用

// 显示已启用的设备：enp0s5、br0、veth0、veth0_1
[root@centos7-18 ~]# ip address show up
2: enp0s5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether 00:1c:42:60:87:b2 brd ff:ff:ff:ff:ff:ff
    inet6 fdb2:2c26:f4e4:0:21c:42ff:fe60:87b2/64 scope global mngtmpaddr dynamic 
       valid_lft 2569303sec preferred_lft 582103sec
    inet6 fe80::21c:42ff:fe60:87b2/64 scope link 
       valid_lft forever preferred_lft forever
6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:1c:42:60:87:b2 brd ff:ff:ff:ff:ff:ff
    inet 10.211.55.18/24 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fdb2:2c26:f4e4:0:21c:42ff:fe60:87b2/64 scope global mngtmpaddr dynamic 
       valid_lft 2591897sec preferred_lft 604697sec
    inet6 fe80::8413:97ff:fe70:a2e2/64 scope link 
       valid_lft forever preferred_lft forever
7: veth0_1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 7a:87:ef:c6:77:9b brd ff:ff:ff:ff:ff:ff
    inet6 fdb2:2c26:f4e4:0:7887:efff:fec6:779b/64 scope global mngtmpaddr dynamic 
       valid_lft 2591897sec preferred_lft 604697sec
    inet6 fe80::7887:efff:fec6:779b/64 scope link 
       valid_lft forever preferred_lft forever
8: veth0@veth0_1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether 86:08:8e:91:09:fe brd ff:ff:ff:ff:ff:ff
    inet6 fe80::8408:8eff:fe91:9fe/64 scope link 
       valid_lft forever preferred_lft forever

检查测试环境-br0的veth0接口已开启hairpin

// 开启hairpin
[root@centos7-18 ~]# brctl hairpin br0 veth0 on
// 查看br0的veth0开启hairpin结果
[root@centos7-18 ~]# bridge -d link  | grep -A5 veth0
8: veth0 state UP @veth0_1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 2 
    hairpin on guard off root_block off fastleave off learning on flood on mcast_flood on 
[root@centos7-18 ~]#

检查测试环境-veth0_1.101和veth0_1.102已启用

// 显示ns101已启用的设备：veth0_1.101
[root@centos7-18 ~]# ip netns exec ns101 ip a show up
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
9: veth0_1.101@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether b2:3e:6e:ae:74:57 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.211.55.101/24 scope global veth0_1.101
       valid_lft forever preferred_lft forever
    inet6 fdb2:2c26:f4e4:0:b03e:6eff:feae:7457/64 scope global mngtmpaddr dynamic 
       valid_lft 2591507sec preferred_lft 604307sec
    inet6 fe80::b03e:6eff:feae:7457/64 scope link 
       valid_lft forever preferred_lft forever
// 显示ns102已启用的设备：veth0_1.102
[root@centos7-18 ~]# ip netns exec ns102 ip a show up
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
10: veth0_1.102@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 22:f8:d5:8b:c1:63 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.211.55.102/24 scope global veth0_1.102
       valid_lft forever preferred_lft forever
    inet6 fdb2:2c26:f4e4:0:20f8:d5ff:fe8b:c163/64 scope global mngtmpaddr dynamic 
       valid_lft 2591491sec preferred_lft 604291sec
    inet6 fe80::20f8:d5ff:fe8b:c163/64 scope link 
       valid_lft forever preferred_lft forever

测试namespace ns101 访问宿主机以外网络
- Ping 网关IP 10.211.55.1，通
- Ping 网络内其它主机IP 10.211.55.10，通

// 查看ns101的IP
[root@centos7-18 ~]# ip netns exec ns101 ip a | grep -A5 veth
9: veth0_1.101@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether b2:3e:6e:ae:74:57 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.211.55.101/24 scope global veth0_1.101
       valid_lft forever preferred_lft forever
    inet6 fdb2:2c26:f4e4:0:b03e:6eff:feae:7457/64 scope global mngtmpaddr dynamic 
       valid_lft 2591841sec preferred_lft 604641sec
    inet6 fe80::b03e:6eff:feae:7457/64 scope link 
       valid_lft forever preferred_lft forever

// Ping 网关IP 10.211.55.1，通
[root@centos7-18 ~]# ip netns exec ns101 ping -c2 10.211.55.1
PING 10.211.55.1 (10.211.55.1) 56(84) bytes of data.
64 bytes from 10.211.55.1: icmp_seq=1 ttl=128 time=0.164 ms
64 bytes from 10.211.55.1: icmp_seq=2 ttl=128 time=0.323 ms

--- 10.211.55.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.164/0.243/0.323/0.081 ms
[root@centos7-18 ~]# 
// Ping 网络内其它主机IP 10.211.55.10，通
[root@centos7-18 ~]# ip netns exec ns101 ping -c2 10.211.55.10
PING 10.211.55.10 (10.211.55.10) 56(84) bytes of data.
64 bytes from 10.211.55.10: icmp_seq=1 ttl=64 time=0.288 ms
64 bytes from 10.211.55.10: icmp_seq=2 ttl=64 time=0.526 ms

--- 10.211.55.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.288/0.407/0.526/0.119 ms
[root@centos7-18 ~]#

测试namespace ns101和ns102 可以互相访问

// ns101 ping ns102的IP 10.211.55.102，通
[root@centos7-18 ~]# ip netns exec ns101 ping -c2 10.211.55.102
PING 10.211.55.102 (10.211.55.102) 56(84) bytes of data.
64 bytes from 10.211.55.102: icmp_seq=1 ttl=64 time=0.085 ms
64 bytes from 10.211.55.102: icmp_seq=2 ttl=64 time=0.083 ms

--- 10.211.55.102 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1005ms
rtt min/avg/max/mdev = 0.083/0.084/0.085/0.001 ms
// ns102 ping ns101的IP 10.211.55.101，通
[root@centos7-18 ~]# ip netns exec ns102 ping -c2 10.211.55.101
PING 10.211.55.101 (10.211.55.101) 56(84) bytes of data.
64 bytes from 10.211.55.101: icmp_seq=1 ttl=64 time=0.057 ms
64 bytes from 10.211.55.101: icmp_seq=2 ttl=64 time=0.087 ms

--- 10.211.55.101 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.057/0.072/0.087/0.015 ms
[root@centos7-18 ~]#

结果

bridge工作在二层，打通了namespace与宿主机以外的网络，查看br0的mac table可以看到mac结果

1. 查看br0的mac table，包含网关（外部）的mac

先通过宿主机Ping网关IP 10.211.55.1，获得网关对应的mac：00:1c:42:00:00:18

// 宿主机Ping网关IP 10.211.55.1后，查看网关mac
[root@centos7-18 ~]# ping -c2 10.211.55.1
PING 10.211.55.1 (10.211.55.1) 56(84) bytes of data.
64 bytes from 10.211.55.1: icmp_seq=1 ttl=128 time=0.170 ms
64 bytes from 10.211.55.1: icmp_seq=2 ttl=128 time=0.264 ms

--- 10.211.55.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.170/0.217/0.264/0.047 ms
// 网关mac为 00:1c:42:00:00:18
[root@centos7-18 ~]# arp 
Address                  HWtype  HWaddress           Flags Mask            Iface
10.211.55.2              ether   00:1c:42:00:00:08   C                     br0
10.211.55.1              ether   00:1c:42:00:00:18   C                     br0
10.211.55.10                     (incomplete)                              br0
10.211.55.1              ether   00:1c:42:00:00:18   C                     enp0s6
10.211.55.102            ether   22:f8:d5:8b:c1:63   C                     br0
10.211.55.101            ether   b2:3e:6e:ae:74:57   C                     br0

查看br0的mac table，已包含网关的mac(00:1c:42:00:00:18)

// 查看br0 mac table
[root@centos7-18 ~]# brctl showmacs br0
port no mac addr                is local?       ageing timer
  2     00:1c:42:00:00:08       no                 0.00
  2     00:1c:42:00:00:18       no                10.43
  2     00:1c:42:60:87:b2       yes                0.00
  2     00:1c:42:60:87:b2       yes                0.00
  2     00:1c:42:d1:70:62       no               193.31
  1     86:08:8e:91:09:fe       yes                0.00
  1     86:08:8e:91:09:fe       yes                0.00

2. 查看br0的mac table，包含macvlan vepa子网卡（内部）的mac

查看ns101 网卡veth0_1.101的mac为b2:3e:6e:ae:74:57

// ns101网卡veth0_1.101的mac为b2:3e:6e:ae:74:57
[root@centos7-18 ~]# ip netns exec ns101 ip a | grep -A2 veth
9: veth0_1.101@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether b2:3e:6e:ae:74:57 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.211.55.101/24 scope global veth0_1.101
       valid_lft forever preferred_lft forever
    inet6 fdb2:2c26:f4e4:0:b03e:6eff:feae:7457/64 scope global mngtmpaddr dynamic

在ns101中 ping 网关IP 10.211.55.1

// ns101 ping 网关IP 10.211.55.1
[root@centos7-18 ~]# ip netns exec ns101 ping -c2 10.211.55.1
PING 10.211.55.1 (10.211.55.1) 56(84) bytes of data.
64 bytes from 10.211.55.1: icmp_seq=1 ttl=128 time=0.169 ms
64 bytes from 10.211.55.1: icmp_seq=2 ttl=128 time=0.225 ms

--- 10.211.55.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.169/0.197/0.225/0.028 ms

查看br0的mac table，新增加了veth0_1.101的mac（b2:3e:6e:ae:74:57）

// 查看br0 mac table，增加了veth0_1.101的mac b2:3e:6e:ae:74:57
[root@centos7-18 ~]# brctl showmacs br0
port no mac addr                is local?       ageing timer
  2     00:1c:42:00:00:08       no                 0.00
  2     00:1c:42:00:00:18       no                 0.93
  2     00:1c:42:60:87:b2       yes                0.00
  2     00:1c:42:60:87:b2       yes                0.00
  2     00:1c:42:d1:70:62       no               177.70
  1     86:08:8e:91:09:fe       yes                0.00
  1     86:08:8e:91:09:fe       yes                0.00
  1     b2:3e:6e:ae:74:57       no                 0.94

总结

通过Linux bridge 开启接口hairpin的方式，模拟macvlan vepa在外部交换支持802.1q的情况下，同一父网卡下的多个子网卡之间是可以通讯的。(详见 Linux bridge开启hairpin模拟测试macvlan vepa模式 )

通过将宿主机物理网卡加入到Linux bridge中，使用bridge桥接内外网络，可以实现内部macvlan vepa子网卡访问外部网络。

华为开发者联盟HarmonyOS专区

鸿蒙生态一站式服务平台。

更多推荐

Xenomai (学习笔记)

文章目录1. 简介1.1 实时OS1.2 Adeos域 (Domain)中断管道（Interrupt Pipe）事件管道(event pipeline)乐观的中断保护系统事件传播基于Linux的实现1.3 Adeos为Xenomai提供的服务Xenomai的主要和次要域名系统调用拦截中断传播启用/禁用中断源在域之间共享中断中断共享和延迟结论Xenomai用户层实时的实现(TSC)Xenomai多A