docker基础知识之network namespace

一个物理的网络设备最多存在于一个network namespace,可以通过创建veth pair在不同的network namespace之间创建通道,来达到通讯的目的。

容器的bridge模式的实现思路为创建一个veth pair,一端放置在新的namespace,通常命名为eth0,另外一端放在原先的namespace中连接物理网络设备,以此实现网络通信。

docker daemon负责在宿主机上创建veth pair,把一端绑定到docker0网桥,另一端到新建的network namespace进程中。建立的过程中,docker daemon和dockerinit通过pipe进行通讯。

测试例子

测试network namespace的过程比较复杂。

docker默认采用的为bridge模式,在容器所在的宿主机上看到的网卡情况如下:

1
2
3
4
5
6
7
8
9
10
11
[root@localhost software]# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 08:00:27:6c:3e:95 brd ff:ff:ff:ff:ff:ff
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 08:00:27:a5:78:ca brd ff:ff:ff:ff:ff:ff
4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether 02:42:a3:75:00:16 brd ff:ff:ff:ff:ff:ff
18: veth71f2650@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default
link/ether ca:05:f7:db:6f:4c brd ff:ff:ff:ff:ff:ff link-netnsid 0

其中的enp0s3和enp0s8可以忽略,为虚拟机使用的网卡。docker0和veth71f2650@if17是需要关注的网卡。

1
2
3
[root@localhost software]# brctl show
bridge name bridge id STP enabled interfaces
docker0 8000.0242a3750016 no veth71f2650

下面的操作为在已经运行docker的虚拟机上的,以便于跟docker进行比较。

以下命令根据coolshell中的步骤进行配置,并对执行命令的顺序进行了调整。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
# 增加network namespace ns1
[root@localhost software]# ip netns add ns1
[root@localhost software]# ip netns
ns1

# 激活namespace ns1中的lo设备
[root@localhost software]# ip netns exec ns1 ip link set dev lo up

# 创建veth pair
[root@localhost software]# ip link add veth-ns1 type veth peer name lxcbr0.1
# 多出了lxcbr0.1@veth-ns1和veth-ns1@lxcbr0.1两个设备
# 后面的操作步骤中将lxcbr0.1位于主网络命名空间中,veth-ns1位于ns1命名空间中
[root@localhost software]# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 08:00:27:6c:3e:95 brd ff:ff:ff:ff:ff:ff
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 08:00:27:a5:78:ca brd ff:ff:ff:ff:ff:ff
4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether 02:42:a3:75:00:16 brd ff:ff:ff:ff:ff:ff
18: veth71f2650@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default
link/ether ca:05:f7:db:6f:4c brd ff:ff:ff:ff:ff:ff link-netnsid 0
19: lxcbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/ether c6:b7:4d:7f:f8:90 brd ff:ff:ff:ff:ff:ff
20: lxcbr0.1@veth-ns1: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether c6:8a:26:3d:ba:de brd ff:ff:ff:ff:ff:ff
21: veth-ns1@lxcbr0.1: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether f2:03:22:93:d6:f4 brd ff:ff:ff:ff:ff:ff

# 将设备veth-ns1放入到ns1命名空间中
[root@localhost software]# ip link set veth-ns1 netns ns1
# 可以看到veth-ns1设备在当前命名空间消失了
[root@localhost software]# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 08:00:27:6c:3e:95 brd ff:ff:ff:ff:ff:ff
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 08:00:27:a5:78:ca brd ff:ff:ff:ff:ff:ff
4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether 02:42:a3:75:00:16 brd ff:ff:ff:ff:ff:ff
18: veth71f2650@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default
link/ether ca:05:f7:db:6f:4c brd ff:ff:ff:ff:ff:ff link-netnsid 0
19: lxcbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/ether c6:b7:4d:7f:f8:90 brd ff:ff:ff:ff:ff:ff
20: lxcbr0.1@if21: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether c6:8a:26:3d:ba:de brd ff:ff:ff:ff:ff:ff link-netnsid 1
# 同时在命名空间ns1中看到了设备veth-ns1
[root@localhost software]# ip netns exec ns1 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
21: veth-ns1@if20: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether f2:03:22:93:d6:f4 brd ff:ff:ff:ff:ff:ff link-netnsid 0

# 将ns1中的veth-ns1设备更名为eth0
[root@localhost software]# ip netns exec ns1 ip link set dev veth-ns1 name eth0
[root@localhost software]# ip netns exec ns1 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
21: eth0@if20: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether f2:03:22:93:d6:f4 brd ff:ff:ff:ff:ff:ff link-netnsid 0

# 为容器中的网卡分配一个IP地址,并激活它
[root@localhost software]# ip netns exec ns1 ifconfig eth0 192.168.10.11/24 up
# 可以看到eth0网卡上有ip地址
[root@localhost software]# ip netns exec ns1 ifconfig
eth0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.10.11 netmask 255.255.255.0 broadcast 192.168.10.255
ether f2:03:22:93:d6:f4 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

# 添加一个网桥lxcbr0,类似于docker中的docker0
[root@localhost software]# brctl addbr lxcbr0
[root@localhost software]# brctl show
bridge name bridge id STP enabled interfaces
docker0 8000.0242a3750016 no veth71f2650
lxcbr0 8000.000000000000 no

# 关闭生成树协议,默认该协议为关闭状态
[root@localhost software]# brctl stp lxcbr0 off
[root@localhost software]# brctl show
bridge name bridge id STP enabled interfaces
docker0 8000.0242a3750016 no veth71f2650
lxcbr0 8000.000000000000 no

# 为网桥配置ip地址
ifconfig lxcbr0 192.168.10.1/24 up
[root@localhost software]# ifconfig lxcbr0
lxcbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.10.1 netmask 255.255.255.0 broadcast 192.168.10.255
inet6 fe80::c4b7:4dff:fe7f:f890 prefixlen 64 scopeid 0x20<link>
ether c6:b7:4d:7f:f8:90 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8 bytes 648 (648.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

# 将veth设备中的其中一个lxcbr0.1添加到网桥lxcbr0上
[root@localhost software]# brctl addif lxcbr0 lxcbr0.1
# 可以看到网桥lxcbr0中已经包含了设备lxcbr0.1
[root@localhost software]# brctl show
bridge name bridge id STP enabled interfaces
docker0 8000.0242a3750016 no veth71f2650
lxcbr0 8000.c68a263dbade no lxcbr0.1

# 为网络空间ns1增加默认路由规则,出口为网桥ip地址
[root@localhost software]# ip netns exec ns1 ip route add default via 192.168.10.1
[root@localhost software]# ip netns exec ns1 ip route
default via 192.168.10.1 dev eth0
192.168.10.0/24 dev eth0 proto kernel scope link src 192.168.10.11

# 为ns1增加resolv.conf
[root@localhost software]# mkdir -p /etc/netns/ns1
[root@localhost software]# echo "nameserver 8.8.8.8" > /etc/netns/ns1/resolv.conf

/proc/[pid]/ns文件

可以使用如下命令查看当前容器在宿主机上的进程id。

inspect --format '{{.State.Pid}}' a1bf0119d891```
1
2

每个进程在/proc/${pid}/ns/目录下都会创建其对应的虚拟文件,并链接到一个真实的namespace文件上,如果两个进程下的链接文件链接到同一个地方,说明两个进程同属于一个namespace。

[root@localhost runc]# ls -l /proc/4913/ns/
total 0
lrwxrwxrwx 1 root root 0 Sep 11 00:21 ipc -> ipc:[4026532130]
lrwxrwxrwx 1 root root 0 Sep 11 00:21 mnt -> mnt:[4026532128]
lrwxrwxrwx 1 root root 0 Sep 11 00:18 net -> net:[4026532133]
lrwxrwxrwx 1 root root 0 Sep 11 00:21 pid -> pid:[4026532131]
lrwxrwxrwx 1 root root 0 Sep 11 00:21 user -> user:[4026531837]
lrwxrwxrwx 1 root root 0 Sep 11 00:21 uts -> uts:[4026532129]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

一个进程可以通过setns()系统调用来进入到另外一个namespace中。

编写setns.c程序,该程序会进入到进程id所在的网络命令空间,并使用`gcc setns.c -o setns`进行编译,编译完成后执行`./setns /proc/4913/ns/net ifconfig`可以看到网卡的信息为容器中的网卡信息。

```c
#include <unistd.h>
#include <fcntl.h>
#include <stdlib.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
int fd = open(argv[1], O_RDONLY);
if (setns(fd, 0) == -1) {
perror("setns");
exit(-1);
}
execvp(argv[2], &argv[2]);
printf("execvp exit\n");
}

如果执行./setns /proc/4913/ns/net /bin/bash,在宿主机上查看docker进程和/bin/bash进程的网络命名空间/proc/${pid}/ns/net,会发现都指向lrwxrwxrwx 1 root root 0 Sep 14 14:42 net -> net:[4026532133]同一个位置。

另外,docker还提供了--net参数用于加入另一个容器的网络命名空间docker run -it --net container:7835490487c1 busybox ifconfig

reference

 wechat
欢迎订阅我的公众号!