k8s部署之分布式KV存储Etcd

Etcd是什么

Etcd是一个分布式、使用Raft算法维护一致性的kv存储系统,与其类似产品有Zookeeper(老牌经典)、Consul等,Etcd相对ZK,更加轻量、易运维。具体三者之间的对比可参考 https://luyiisme.github.io/2017/04/22/spring-cloud-service-discovery-products/

使用场景

和zk、consul等类似,使用场景多用于:

  • 服务发现
  • 消息发布与订阅
  • 负载均衡
  • 分布式锁
  • 分布式队列

读写性能

压测数据参考官方:
https://coreos.com/etcd/docs/latest/op-guide/performance.html

本地集群部署

  • 操作系统:Debian8 x64
  • Etcd v3.2.7

A. 安装

1
2
3
4
5
6
7
wget https://github.com/coreos/etcd/releases/download/v3.2.7/etcd-v3.2.7-linux-arm64.tar.gz
tar xf etcd-v3.2.7-linux-arm64.tar.gz
cd etcd-v3.2.7-linux-amd64
cp etc* /usr/local/bin/
etcd: Etcd服务端文件
etcdctl: 供用户使用的命令客户端

B. 启动服务

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
root@a4c8d490:/home/geekwolf# etcd
2017-09-07 15:42:23.957656 I | etcdmain: etcd Version: 3.2.7
2017-09-07 15:42:23.957699 I | etcdmain: Git SHA: bb66589
2017-09-07 15:42:23.957718 I | etcdmain: Go Version: go1.8.3
2017-09-07 15:42:23.957723 I | etcdmain: Go OS/Arch: linux/amd64
2017-09-07 15:42:23.957729 I | etcdmain: setting maximum number of CPUs to 8, total number of available CPUs is 8
2017-09-07 15:42:23.957739 W | etcdmain: no data-dir provided, using default data-dir ./default.etcd
2017-09-07 15:42:23.957764 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2017-09-07 15:42:23.957995 I | embed: listening for peers on http://localhost:2380
2017-09-07 15:42:23.958107 I | embed: listening for client requests on localhost:2379
2017-09-07 15:42:23.964607 I | etcdserver: name = default
2017-09-07 15:42:23.964633 I | etcdserver: data dir = default.etcd
2017-09-07 15:42:23.964652 I | etcdserver: member dir = default.etcd/member
2017-09-07 15:42:23.964657 I | etcdserver: heartbeat = 100ms
2017-09-07 15:42:23.964663 I | etcdserver: election = 1000ms
2017-09-07 15:42:23.964668 I | etcdserver: snapshot count = 100000
2017-09-07 15:42:23.964680 I | etcdserver: advertise client URLs = http://localhost:2379
2017-09-07 15:42:23.973007 I | etcdserver: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 14
2017-09-07 15:42:23.973041 I | raft: 8e9e05c52164694d became follower at term 2
2017-09-07 15:42:23.973065 I | raft: newRaft 8e9e05c52164694d [peers: [], term: 2, commit: 14, applied: 0, lastindex: 14, lastterm: 2]
2017-09-07 15:42:23.984367 W | auth: simple token is not cryptographically signed
2017-09-07 15:42:23.993237 I | etcdserver: starting server... [version: 3.2.7, cluster version: to_be_decided]
2017-09-07 15:42:23.993659 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
2017-09-07 15:42:23.993754 N | etcdserver/membership: set the initial cluster version to 3.2
2017-09-07 15:42:23.993796 I | etcdserver/api: enabled capabilities for version 3.2
2017-09-07 15:42:24.473288 I | raft: 8e9e05c52164694d is starting a new election at term 2
2017-09-07 15:42:24.473451 I | raft: 8e9e05c52164694d became candidate at term 3
2017-09-07 15:42:24.473519 I | raft: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 3
2017-09-07 15:42:24.473568 I | raft: 8e9e05c52164694d became leader at term 3
2017-09-07 15:42:24.473605 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 3
2017-09-07 15:42:24.478746 I | etcdserver: published {Name:default ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32
2017-09-07 15:42:24.478824 I | embed: ready to serve client requests
2017-09-07 15:42:24.479116 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!

由上面的输出可知:

  • etcd服务之间通信端口是2380,暴露给客户端端口为2379
  • 默认将数据存放到当前路径default.etcd/目录下
  • 该节点的名称默认为default
  • 集群和节点都会生成唯一的uuid
  • 启动服务时,会根据raft算法,选举leader

C. 测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
查看api版本(默认api版本是v2)
root@a4c8d490:~/k8s/etcd-v3.2.7-linux-amd64# etcdctl --version
etcdctl version: 3.2.7
API version: 2
使用API V3方法:
Etcd服务端和客户端添加变量 export ETCDCTL_API=3,重新启动etcd服务即可
以下操作在api v3版本:
写入key: etcdctl put foo bar
读取key: etcdctl get foo
多key范围读取: etcdctl get foo foo9(会将foo..foo8的key读取,不包括foo9)
读取过往版本key的值(Etcd键值对的修改都会增加全局修订版本号,--rev为版本号):
etcdctl get --rev=4 foo foo9
删除key: etcdctl del foo
范围删除(foo->foo9):etcdctl del foo foo9
观察key变化:etcdctl watch foo
观察范围key变化: etcdctl watch foo foo9
从rev=2版本开始观察key变化: etcdctl watch --rev=2 foo
压缩版本5之前的修订版本(压缩后5之前的版本不可能访问): etcdctl compact 5
授予key有效期:
创建租约:
$ etcdctl lease grant 10
lease 694d5e5b63a74f31 granted with TTL(10s)
附加key foo到租约694d5e5b63a74f31,该租约过期后,会删除附加的所有key
撤销租约(撤销后,附加改租约的所有key被删除): etcdctl lease revoke 32695410dcc0ca06
维持租约(执行后,会一直维持该租约): etcdctl lease keep-alive 32695410dcc0ca0
其他参数可参考 etcdctl --help
通过HTTP操作:
Etcd v2: https://coreos.com/etcd/docs/latest/v2/api.html
Etcd v3: https://coreos.com/etcd/docs/latest/dev-guide/api_grpc_gateway.html

多节点集群部署

静态模式部署
环境说明(三节点集群)
节点 地址 主机
etcd1 192.168.234.133 etcd1.simlinux.com
etcd2 192.168.234.133 etcd2.simlinux.com
etcd3 192.168.234.133 etcd3.simlinux.com
初始化环境

三个节点分别设置主机名:

1
2
3
hostnamectl --static set-hostname etcd1.simlinux.com
hostnamectl --static set-hostname etcd2.simlinux.com
hostnamectl --static set-hostname etcd3.simlinux.com

三个节点hosts文件添加:

1
2
3
4
vim /etc/hosts
192.168.234.133 etcd1.simlinux.com
192.168.234.133 etcd2.simlinux.com
192.168.234.133 etcd3.simlinux.com

生成etcd证书(用于etcd间、客户端与etcd通信)

由上篇k8s部署之使用CFSSL创建证书的CA来生成

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
cat etcd.json
{
"CN": "etcd",
"hosts": [
"127.0.0.1",
"192.168.234.133",
"192.168.234.134",
"192.168.234.135"
],
"key": {
"algo": "ecdsa",
"size": 256
},
"names": [
{
"C": "CN",
"L": "ShangHai",
"ST": "ShangHai",
"O": "K8s",
"OU": "System"
}
]
}
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd.json | cfssljson -bare etcd
将CA和etcd证书拷贝到etcd所有节点:
cp ca.pem etcd-key.pem etcd.pem /etc/etcd/ssl/
安装etcd节点(所有节点)
1
2
3
4
5
wget https://github.com/coreos/etcd/releases/download/v3.2.7/etcd-v3.2.7-linux-amd64.tar.gz
tar xf etcd-v3.2.7-linux-amd64.tar.gz
cd etcd-v3.2.7-linux-amd64
chmod +x etcd*
cp etcd* /bin
etcd配置
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
服务管理(所有节点相同):
vim /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=/data/k8s/etcd/
EnvironmentFile=/etc/etcd/etcd.conf
ExecStart=/bin/etcd
--name=${NAME}
--cert-file=/etc/etcd/ssl/etcd.pem
--key-file=/etc/etcd/ssl/etcd-key.pem
--peer-cert-file=/etc/etcd/ssl/etcd.pem
--peer-key-file=/etc/etcd/ssl/etcd-key.pem
--trusted-ca-file=/etc/etcd/ssl/ca.pem
--peer-trusted-ca-file=/etc/etcd/ssl/ca.pem
--initial-advertise-peer-urls=${INITIAL_ADVERTISE_PEER_URLS}
--listen-peer-urls=${LISTEN_PEER_URLS}
--listen-client-urls=${LISTEN_CLIENT_URLS}
--advertise-client-urls=${ADVERTISE_CLIENT_URLS}
--initial-cluster-token=${INITIAL_CLUSTER_TOKEN}
--initial-cluster=${INITIAL_CLUSTER}
--initial-cluster-state=new
--data-dir=${DATA_DIR}
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target

配置文件:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
vim /etc/etcd/etcd.conf
#节点名称
NAME="etcd1"
#etcd数据存放目录
DATA_DIR="/data/k8s/etcd"
#etcd节点间通信监听地址
LISTEN_PEER_URLS="https://192.168.234.133:2380"
#对外提供服务的地址
LISTEN_CLIENT_URLS="https://192.168.234.133:2379,https://127.0.0.1:2379"
#通知其他etcd节点本实例地址
INITIAL_ADVERTISE_PEER_URLS="https://192.168.234.133:2380"
#初始化集群内节点地址
INITIAL_CLUSTER="etcd1=https://192.168.234.133:2380,etcd2=https://192.168.234.134:2380,etcd3=https://192.168.234.135:2380"
#初始化状态.new表示新建,已经存在的集群使用existing
INITIAL_CLUSTER_STATE="new"
#创建集群的token,每个集群唯一
INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
#告知其他集群本节点客户端监听地址
ADVERTISE_CLIENT_URLS="https://192.168.234.133:2379"
ETCDCTL_API=3
其中NAME/LISTEN_PEER_URLS/LISTEN_CLIENT_URLS/INITIAL_ADVERTISE_PEER_URLS/ADVERTISE_CLIENT_URLS替换成相应节点名称和地址

服务管理
1
2
3
systemctl start etcd.service
systemctl stop etcd.service
systemctl status etcd.service(查看服务状态及日志)
测试验证
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
[root@etcd1 ~]# export etcd1=192.168.234.133
[root@etcd1 ~]# export etcd2=192.168.234.134
[root@etcd1 ~]# export etcd3=192.168.234.135
[root@etcd1 ~]# export ENDPOINTS=$etcd1:2379,$etcd2:2379,$etcd3:2379
查看集群成员:
[root@etcd1 etcd]# etcdctl --write-out=table --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem member list
+------------------+---------+-------+------------------------------+------------------------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS |
+------------------+---------+-------+------------------------------+------------------------------+
| 1a4a83ef243ff1c9 | started | etcd2 | https://192.168.234.134:2380 | https://192.168.234.134:2379 |
| 68243ef8797bd1ce | started | etcd1 | https://192.168.234.133:2380 | https://192.168.234.133:2379 |
| fa30209a63d949b0 | started | etcd3 | https://192.168.234.135:2380 | https://192.168.234.135:2379 |
+------------------+---------+-------+------------------------------+------------------------------+
查看集群状态:
[root@etcd1 ~]# etcdctl --write-out=table --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem endpoint status
+----------------------+------------------+---------+---------+-----------+-----------+------------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+----------------------+------------------+---------+---------+-----------+-----------+------------+
| 192.168.234.133:2379 | 68243ef8797bd1ce | 3.2.7 | 25 kB | false | 10 | 9 |
| 192.168.234.134:2379 | 1a4a83ef243ff1c9 | 3.2.7 | 25 kB | false | 10 | 9 |
| 192.168.234.135:2379 | fa30209a63d949b0 | 3.2.7 | 25 kB | true | 10 | 9 |
+----------------------+------------------+---------+---------+-----------+-----------+------------+
[root@etcd1 ~]# etcdctl --write-out=table --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem endpoint health
192.168.234.135:2379 is healthy: successfully committed proposal: took = 1.374345ms
192.168.234.134:2379 is healthy: successfully committed proposal: took = 2.217525ms
192.168.234.133:2379 is healthy: successfully committed proposal: took = 1.996245ms
保存快照:
[root@etcd1 ~]# etcdctl --write-out=table --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem snapshot save my.db
Snapshot saved at my.db
查看快照状态:
[root@etcd1 ~]# etcdctl --write-out=table --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem snapshot status my.db
+----------+----------+------------+------------+
| HASH | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 9a496339 | 3 | 8 | 25 kB |
+----------+----------+------------+------------+
恢复数据(要先删除原来数据目录,所有节点操作):
[root@etcd1 ~]# etcdctl --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem snapshot restore my.db --data-dir=/data/k8s/etcd/
2017-09-09 02:28:52.439616 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
删除节点:
[root@etcd1 ~]# etcdctl --write-out=table --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem member remove 68243ef8797bd1ce
更新节点:
[root@etcd1 ~]# etcdctl --write-out=table --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem member update 68243ef8797bd1ce https://192.168.234.133:1111(INITIAL_ADVERTISE_PEER_URLS)
添加节点(删除etcd3,添加etcd4):
export etcd4=192.168.234.136
[root@etcd1 ~]# etcdctl --endpoints=${etcd1}:2379,${etcd2}:2379 --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem member add etcd4 --peer-urls=http://192.168.234.136:2380

Etcd:从应用场景到实现原理的全方位解读 http://www.infoq.com/cn/articles/etcd-interpretation-application-scenario-implement-principle
Eetcd集群管理 https://coreos.com/etcd/docs/latest/demo.html

坚持原创分享,您的支持将鼓励我继续创作