一篇学会 Pulsar 部署与线上配置

作者：梁勇 2022-02-21 08:48:00

前端
云原生 Apache Pulsar越来越多的公司使用，与Apache Kafka、Apache RocketMQ并列成为消息领域三家马车，有必要对其研究一番。下面以笔者曾在生产环境使用的配置梳理，希望对大家有所帮助！

专注于为中小企业提供成都网站设计、网站建设服务,电脑端+手机端+微信端的三站合一,更高效的管理,为中小企业三穗免费做网站提供优质的服务。我们立足成都，凝聚了一批互联网行业人才，有力地推动了数千家企业的稳健成长，帮助中小企业通过网站建设实现规模扩充和转变。

一、Pulsar安装与部署

1.下载安装包

Pulsar安装包包含了zookeeper、broker、bookie三个组件。

下载Pulsar二进制包

https://pulsar.apache.org/download/

解压压缩包

tar -zvxf apache-pulsar-2.9.1-bin.tar.gz

2.部署zookeeper

2.1 修改zookeeper配置

创建目录

mkdir -p data/zookeeper

echo 1 > data/zookeeper/myid

修改zk配置，文件位于conf/zookeeper.conf

# 数据目录
dataDir=data/zookeeper
# 日志目录
dataLogDir=data/zookeeper/logs
# zk集群配置，server.1~n
server.1=127.0.0.1:2888:3888

2.2 后台启动zookeeper

bin/pulsar-daemon start zookeeper
doing start zookeeper ...
starting zookeeper, logging to /Users/admin/work/software_install/apache-pulsar-2.9.1/logs/pulsar-zookeeper-M-C02GL1NTQ05P.log
Note: Set immediateFlush to true in conf/log4j2.yaml will guarantee the logging event is flushing to disk immediately. The default behavior is switched off due to performance considerations.

通过pulsar-daemon管理pulsar组件

bin/pulsar-daemon help
Error: no enough arguments provided.
Usage: pulsar-daemon (start|stop|restart)  
where command is one of:
    broker              Run a broker server
    bookie              Run a bookie server
    zookeeper           Run a zookeeper server
    configuration-store Run a configuration-store server
    websocket           Run a websocket proxy server
    functions-worker    Run a functions worker server
    standalone          Run a standalone Pulsar service
    proxy               Run a Proxy Pulsar service

备注：可以通过pulsar-daemon命令对broker、bookie、zookeeper等组件启动、关闭或者重启。

2.3 检查zookeeper是否启动成功

zookeeper启动日志和查看zookeeper进程

ps axu | grep zookeeper

3.元数据初始化

3.1 初始化命令说明

bin/pulsar initialize-cluster-metadata \
--cluster pulsar-cluster-1 \
--zookeeper 127.0.0.1:2181 \
--configuration-store 127.0.0.1:2181 \
--web-service-url http://127.0.0.1:8080 \
--web-service-url-tls https://127.0.0.1:8443 \  
--broker-service-url pulsar://127.0.0.1:6650 \
--broker-service-url-tls pulsar+ssl://127.0.0.1:6651

参数说明

参数说明

备注：生产环境可以使用域名。

3.2 查看初始化结果

bin/pulsar zookeeper-shell

[zk: localhost:2181(CONNECTED) 1] ls /
[admin, bookies, ledgers, pulsar, stream, zookeeper]

4.部署BookKeeper集群

4.1 配置修改

bindAddress=127.0.0.1
advertisedAddress=127.0.0.1
zkServers=127.0.0.1:2181

参数说明

4.2 命令启动

bin/pulsar-daemon start bookie
doing start bookie ...
starting bookie, logging to /Users/admin/work/software_install/apache-pulsar-2.9.1/logs/pulsar-bookie-M-C02GL1NTQ05P.log
Note: Set immediateFlush to true in conf/log4j2.yaml will guarantee the logging event is flushing to disk immediately. The default behavior is switched off due to performance considerations.

4.3 测试bookie集群

bin/bookkeeper shell simpletest --ensemble 1 --writeQuorum 1 --ackQuorum 1 -- numEntries 1000

...
2022-02-19T23:43:03,391+0800 [main] INFO  org.apache.bookkeeper.tools.cli.commands.client.SimpleTestCommand - 722 entries written
2022-02-19T23:43:03,983+0800 [main] INFO  org.apache.bookkeeper.tools.cli.commands.client.SimpleTestCommand - 1000 entries written to ledger 0
2022-02-19T23:43:04,041+0800 [main] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 127.0.0.1:3181
...

备注：通过simpletest命令向bookie集群写入测试数据，完成测试后会自动删除。

5.部署Broker集群

5.1 修改配置

zookeeperServers=127.0.0.1:2181
configurationStoreServers=127.0.0.1:2181
bindAddress=127.0.0.1
# 默认InetAddress.getLocalHost().getHostName()
advertisedAddress=127.0.0.1
clusterName=pulsar-cluster-1

5.2 启动broker

bin/pulsar-daemon start broker
doing start broker ...
starting broker, logging to /Users/admin/work/software_install/apache-pulsar-2.9.1/logs/pulsar-broker-M-C02GL1NTQ05P.log
Note: Set immediateFlush to true in conf/log4j2.yaml will guarantee the logging event is flushing to disk immediately. The default behavior is switched off due to performance considerations.

5.3 验证集群

查看集群节点

bin/pulsar-admin brokers list cluster-1
"172.17.13.184:8080"

发送测试消息

bin/pulsar-client produce persistent://public/default/test -n 1 -m "Hello Pulsar"
...
2022-02-20T13:31:18,469+0800 [main] INFO  org.apache.pulsar.client.cli.PulsarClientTool - 1 messages successfully produced
...

消费测试消息

bin/pulsar-client consume persistent://public/default/test -n 100 -s "consumer-test" -t "Exclusive"
...
----- got message -----
key:[null], properties:[], content:Hello Pulsar
...

小结：至此测试集群搭建完成，下文将介绍生产环境配置的调整项。

二、生产环境配置

1.机器配置

下面为生产环境搭建Pulsar集群，由3个zookeeper节点、3个broker节点和5个bookie节点构成。

组件配置

组件  配置
zookeeper 4C8G100G * 3
broker  16C64G500G * 3
bookie  16C64G500G * 5

备注：每个组件集群部署时可以同城跨可用区部署，提高高可用。broker不存储消息100G即可，bookie存储消息通常需要较大磁盘，比如3T，具体根据消息量计算。

2.内存优化

2.1 broker内存调整

以内存64G大小，在文件conf/pulsar_env.sh修改如下内容：

PULSAR_MEM=${PULSAR_MEM:-"-Xms10g -Xmx10g -XX:MaxDirectMemorySize=20g"}

2.2 bookie内存调整

以内存64G大小，在文件conf/bkenv.sh修改如下内容：

BOOKIE_MEM=${BOOKIE_MEM:-${PULSAR_MEM:-"-Xms10g -Xmx10g -XX:MaxDirectMemorySize=20g"}}

3.Broker调整项

备注：参数根据实际情况调整，在线上开启负载均衡时，发现有重复消息，此处先将其关闭。

4.Bookie调整项

配置项说明

备注：journalDirectory和ledgerDirectories在条件允许的情况可以配置到不同的磁盘。

本文转载自微信公众号「瓜农老梁」，可以通过以下二维码关注。转载本文请联系瓜农老梁公众号。

标题名称：一篇学会Pulsar部署与线上配置
本文路径：http://www.mswzjz.cn/qtweb/news1/40101.html

攀枝花网站建设、攀枝花网站运维推广公司-贝锐智能，是专注品牌与效果的网络营销公司；服务项目有等

广告

声明：本网站发布的内容（图片、视频和文字）以用户投稿、用户转载内容为主，如果涉及侵权请尽快告知，我们将会在第一时间删除。文章观点不代表本网站立场，如需处理请联系客服。电话：028-86922220；邮箱：631063699@qq.com。内容未经允许不得转载，或转载时需注明来源： 贝锐智能