前言

Exhibitor 是 Netflix 开源的一个用于 ZooKeeper 配置监控和管理的系统。现在 Netflix Exhibitor 已经成为社区开源公共维护项目 Soabase Exhibitor。Exhibitor 是 ZooKeeper 实例监控,备份,恢复,清理和可视化工具,是 ZooKeeper 的监控管理系统。

使用 Exhibitor 监控管理 ZooKeeper

更新历史

2019 年 01 月 07 日 - 初稿

阅读原文 - https://wsgzao.github.io/post/exhibitor/

扩展阅读

exhibitor - https://github.com/soabase/exhibitor


exhibitor 简介

ZooKeeper co-process for instance monitoring, backup/recovery, cleanup and visualization.

Exhibitor is a Java supervisor system for ZooKeeper. It provides a number of features:

  • Watches a ZK instance and makes sure it is running
  • Performs periodic backups
  • Perform periodic cleaning of ZK log directory
  • A GUI explorer for viewing ZK nodes
  • A rich REST API

https://github.com/soabase/exhibitor/wiki

exhibitor 特性

Exhibitor 主要包括以下特性 / 功能:

实例监控

Exhibitor 实例监控在同一服务器上运行的 ZooKeeper 服务器。如果 ZK 没有运行,Exhibitor 会写入 zoo.cfg 文件(请参阅下面的 ZK 集群配置)并启动它。如果 ZooKeeper 由于某种原因崩溃,Exhibitor 也会重新启动它。

日志清理

在 ZooKeeper 3.4.x 之前的版本中,日志文件需要维护,Exhibitor 会负责定期维护。

备份 / 还原

ZooKeeper 集群中的备份比传统数据存储(例如 RDBMS)更复杂。一般来说,ZooKeeper 中的大部分数据是短暂的。盲目恢复整个 ZooKeeper 数据集可能会造成更大危害,因此,需要选择性的恢复以防止对数据集的子集造成意外损坏。Exhibitor 提供了这一功能。

Exhibitor 会定期备份 ZooKeeper 的事务文件,备份后,就可以对这些事务文件建立索引。

集群配置

Exhibitor 为整个 Zookeeper 集群提供了一个独立的控制台,通过它所做的配置更改会对整个集群有效。以下是一些共享配置值:

Name Description
ZooKeeper Install DirPath to the ZooKeeper server installation
ZooKeeper Data Dir Path where ZooKeeper should store its data
Log Index Dir Path where indexed transaction logs should be kept
Servers List of servers/server-ids in the ensemble
Additional Config Additional fields/values to store in zoo.cfg

集群滚动升级

Exhibitor 可以以滚动方式更新集群中的服务器,以便在进行更改时让 ZooKeeper 集群确保 Quorum 设定的最低服务能力。

自动实例管理

Exhibitor 可以配置为自动向集群中添加新实例,并删除陈旧的实例。这使得 ZooKeeper 集群可以实现 “无接触交钥匙管理”。

可视化

Exhibitor 为 ZooKeeper 提供了 ZNode 层次结构的图形树视图。

ZK 数据维护

启用后,维护人员可以在 ZooKeeper 的存储层次结构中创建 / 更新 / 删除节点。

死锁检测

当使用 Curator 的锁方案(或类似)时,Exhibitor 可以分析一组表示锁的 ZNode,并确定是否存在潜在的死锁。

Curator 集成

Exhibitor 和 Curator 可以集成工作,当集群中的信息变更时,Curator 实例可以同步更新。

REST API

Exhibitor 提供了一组用于程序集成的 REST API。

重要事项

  1. 使用 Exhibitor 时,不要手动编辑 Zookeeper 的 zoo.cfg 和 myid 文件,因为 Exhibitor 会覆盖它们。
  2. 除了标准的 ZooKeeper 端口,防火墙必须打开 Exhibitor 使用的 HTTP 端口,因为每个 Exhibitor 实例需要与其他参与者通信传递状态。

使用

Exhibitor 的部署构件可以从 Maven 仓库获取。构件分为两种:

GroupID/Org ArtifactID/Name Description
com.netflix.exhibitor exhibitor-standalone 自包含的,可执行的 Exhibitor 版本(可以是独立应用,也可以是 War)
com.netflix.exhibitor exhibitor-core 类库的形式,可以嵌入到应用中

exhibitor 编译安装

官方分享了 Maven 和 Gradle 两种 build 方法,这里以 Maven 为例

https://github.com/soabase/exhibitor/wiki/Building-Exhibitor

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
# install maven
yum install -y maven

# build exhibitor
mkdir exhibitor
cd exhibitor/
wget https://github.com/soabase/exhibitor/archive/exhibitor-1.7.1.zip
unzip exhibitor-1.7.1.zip
cd exhibitor-exhibitor-1.7.1/exhibitor-standalone/src/main/resources/buildscripts/standalone/maven
mvn clean package

[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing /root/exhibitor/exhibitor-exhibitor-1.7.1/exhibitor-standalone/src/main/resources/buildscripts/standalone/maven/target/exhibitor-1.6.0.jar with /root/exhibitor/exhibitor-exhibitor-1.7.1/exhibitor-standalone/src/main/resources/buildscripts/standalone/maven/target/exhibitor-1.6.0-shaded.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 44.624s
[INFO] Finished at: Tue Jan 08 11:28:59 SGT 2019
[INFO] Final Memory: 15M/94M
[INFO] ------------------------------------------------------------------------

cp target/exhibitor-1.6.0.jar /tmp

# Once built, Exhibitor is completely self-contained and can be run from the command line:
java -jar <path>/exhibitor-xxx.jar -c file

[root@localhost ~]# java -jar exhibitor-1.6.0.jar -c file
v1.6.0
INFO com.netflix.exhibitor.core.activity.ActivityLog Exhibitor started [main]
Jan 08, 2019 11:32:38 AM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
INFO org.mortbay.log Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog [main]
INFO org.mortbay.log jetty-1.6.0 [main]
Jan 08, 2019 11:32:38 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.18.3 12/01/2014 08:23 AM'
INFO org.mortbay.log Started SocketConnector@0.0.0.0:8080 [main]
Jan 08, 2019 11:33:00 AM java.util.prefs.FileSystemPreferences$6 run
WARNING: Prefs file removed in background /root/.java/.userPrefs/prefs.xml
INFO com.netflix.exhibitor.core.activity.ActivityLog State: latent [ActivityQueue-0]

# You can test that it’s running correctly by going to this URL in a browser:
http://localhost:8080/exhibitor/v1/ui/index.html
http://192.168.56.103:8080/exhibitor/v1/ui/index.html

管理 zookeeper 集群

如果需要通过 Exhibitor 管理 zookeeper 集群需要在集群的每个机器上安装 Exhibitor

Once built, Exhibitor is completely self-contained and can be run from the command line:

1
java -jar <path>/exhibitor-xxx.jar -c file

You can test that it’s running correctly by going to this URL in a browser:

http://localhost:8080/exhibitor/v1/ui/index.html

The above describes running with the default configuration, however you’ll likely need to customize some of the defaults. Exhibitor accepts several command line options (the complete list is shown when run with –help).

Here is the current set of options:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
 -?,--help                     Print this help
-c,--configtype <arg> Defines which configuration type you want
to use. Choices are: "file", "s3",
"zookeeper" or "none". Additional config
will be required depending on which type
you are using.
--configcheckms <arg> Period (ms) to check for shared config
updates. Default is: 30000
--defaultconfig <arg> Full path to a file that contains
initial/default values for
Exhibitor/ZooKeeper config values. The file
is a standard property file. The property
names are listed below. The file can
specify some or all of the properties.
--headingtext <arg> Extra text to display in UI header
--hostname <arg> Hostname to use for this JVM. Default is:
aldh009.aldc.att.com
--jquerystyle <arg> Styling used for the JQuery-based UI.
Currently available options: red, black,
custom
--loglines <arg> Max lines of logging to keep in memory for
display. Default is 1000.
--nodemodification <arg> If true, the Explorer UI will allow nodes
to be modified (use with caution). Default
is true.
--port <arg> Port for the HTTP Server. Default is: 8080
--prefspath <arg> Certain values (such as Control Panel
values) are stored in a preferences file.
By default, Preferences.userRoot() is used.
Use this option to specify a different file
path.
--servo <arg> true/false (default is false). If enabled,
ZooKeeper will be queried once a minute for
its state via the 'mntr' four letter word
(this requires ZooKeeper 3.4.x+). Servo
will be used to publish this data via JMX.
--timeout <arg> Connection timeout (ms) for ZK connections.
Default is 30000.

== S3 Options ==
--s3credentials <arg> Optional credentials to use for s3backup or
s3config. Argument is the path to an AWS
credential properties file with two
properties:
com.netflix.exhibitor.s3.access-key-id and
com.netflix.exhibitor.s3.access-secret-key
--s3region <arg> Optional region for S3 calls (e.g.
"eu-west-1"). Will be used to set the S3
client's endpoint.

== Configuration Options for Type"s3"==
--s3config <arg> The bucket name and key to store the config
(s3credentials may be provided as well).
Argument is [bucket name]:[key].
--s3configprefix <arg> When using AWS S3 shared config files, the
prefix to use for values such as locks.
Default is exhibitor-

== Configuration Options for Type"zookeeper"==
--zkconfigconnect <arg> The initial connection string for
ZooKeeper shared config storage. E.g:
"host1:2181,host2:2181..."
--zkconfigexhibitorpath <arg> Used if the ZooKeeper shared config is
also running Exhibitor. This is the
URI path for the REST call. The
default is: /
--zkconfigexhibitorport <arg> Used if the ZooKeeper shared config is
also running Exhibitor. This is the
port that Exhibitor is listening on.
IMPORTANT: if this value is not set it
implies that Exhibitor is not being
used on the ZooKeeper shared config.
--zkconfigpollms <arg> The period in ms to check for changes
in the config ensemble. The default
is: 10000
--zkconfigretry <arg> The retry values to use in the form
sleep-ms:retry-qty. The default is:
1000:3
--zkconfigzpath <arg> The base ZPath that Exhibitor should
use. E.g:"/exhibitor/config"

== Configuration Options for Type"file"==
--fsconfigdir <arg> Directory to store Exhibitor properties
(cannot be used with s3config). Exhibitor
uses file system locks so you can specify
a shared location so as to enable
complete ensemble management. Default
location is
/opt/app/grid/products/exhibitor2/bin
--fsconfiglockprefix <arg> A prefix for a locking mechanism. Used in
conjunction with fsconfigdir. Default is
exhibitor-lock-
--fsconfigname <arg> The name of the file to store config in.
Used in conjunction with fsconfigdir.
Default is exhibitor.properties

== Configuration Options for Type"none"==
--noneconfigdir <arg> Directory to store the local configuration
file. Config type"none"is a special purpose
type that should only be used when running a
second ZooKeeper ensemble that is used for
storing config. DO NOT USE THIS MODE for a
normal ZooKeeper ensemble.

== Backup Options ==
--filesystembackup <arg> If true, enables file system backup of
ZooKeeper log files.
--s3backup <arg> If true, enables AWS S3 backup of ZooKeeper
log files (s3credentials may be provided as
well).

== Authorization Options ==
--realm <arg> Specifies the realm as [realm name]:[path/url].
The path/url must point to a realm properties
file as described here (see HashUserRealm):
http://docs.codehaus.org/display/JETTY/Realms
--remoteauth <arg> Exhibitor uses the Jersey Client to remotely
connect to each Exhibitor instance in the
ensemble. If you have security enabled for
Exhibitor you also need to specify authorization
for the remote client. The argument for
remoteauth is: <type>:<realm-user>."type"must
be either"basic"or"digest"."realm-user"is
the user to use from the realm file.
--security <arg> Path to a web.xml file with security information
(all other tags are ignored). See
http://docs.oracle.com/javaee/6/tutorial/doc/gkba
a.html.

== Deprecated Authorization Options ==
--basicauthrealm <arg> Basic Auth Realm to Protect the Exhibitor UI
(DEPRECATED - use --security/--realm
instead)
--consolepassword <arg> Basic Auth Password to Protect the Exhibitor
UI (DEPRECATED - use --security/--realm
instead)
--consoleuser <arg> Basic Auth Username to Protect the Exhibitor
UI (DEPRECATED - use --security/--realm
instead)
--curatorpassword <arg> Basic Auth Password to Protect cluster list
api (DEPRECATED - use --security/--realm
instead)
--curatoruser <arg> Basic Auth Password to Protect the cluster
list api (DEPRECATED - use
--security/--realm instead)

== ACL Options ==
--aclid <arg> Enable ACL for Exhibitor's internal ZooKeeper
connection. This sets the ACL's ID.
--aclperms <arg> Enable ACL for Exhibitor's internal ZooKeeper
connection. This sets the ACL's Permissions - a
comma list of possible permissions. If this isn't
specified the permission is set to ALL. Values:
read, write, create, delete, admin
--aclscheme <arg> Enable ACL for Exhibitor's internal ZooKeeper
connection. This sets the ACL's Scheme.

== Default Property Names ==
log-index-directory
zookeeper-install-directory
zookeeper-data-directory
zookeeper-log-directory
servers-spec
backup-extra
zoo-cfg-extra
java-environment
log4j-properties
client-port
connect-port
election-port
check-ms
cleanup-period-ms
cleanup-max-files
backup-max-store-ms
backup-period-ms
auto-manage-instances
auto-manage-instances-settling-period-ms
observer-threshold
auto-manage-instances-fixed-ensemble-size
auto-manage-instances-apply-all-at-once

Note that exhibitors in an ensemble communicate with each other on the port specified by —port (default to 8080), so make sure this port is open between exhibitor processes in an ensemble.

https://github.com/soabase/exhibitor/wiki/Configuration-UI

重点提一下 Ensemble 中的 Servers 配置:

  • 服务器 IP 之间用逗号分隔,有两种类型:S 表示标准类型,O 表示 Observer 观察者
  • S:1:192.168.56.101,S:2:192.168.56.102,S:3:192.168.56.103
文章目录
  1. 1. 前言
  2. 2. 更新历史
  3. 3. exhibitor 简介
  4. 4. exhibitor 特性
  5. 5. exhibitor 编译安装
  6. 6. 管理 zookeeper 集群