最新消息:

Apache Kafka专题二、Kafka安装和基本使用

IT技术 ipcpu 724浏览

Apache Kafka专题二、Kafka安装和基本使用.md

一、前置条件

Kafka依赖于Java和zookeeper,需要一起部署。这里我们使用了3台服务器。

10.140.100.14
10.140.100.26
10.140.100.32
主机之间不需要设置hosts,但要注意本机hostname必须对应本机IP地址。

二、安装部署ZooKeeper

ZooKeeper我用的版本是zookeeper-3.4.10。直接下载解压,然后配置文件如下

##@@file: zoo.cfg
#datadir
dataDir=/data/apps/zookeeper-3.4.10/data
#clusterIP
server.1=10.140.100.14:2888:3888
server.2=10.140.100.26:2888:3888
server.3=10.140.100.32:2888:3888

然后每个机器将自己的ID写入myid文件

#@以第三台为例
echo "3" >/data/apps/zookeeper-3.4.10/data/myid

启动zookeeper

/data/apps/zookeeper-3.4.10/bin/zkServer.sh start

三、部署kafka集群

kafka有源码包和二进制包,这里我们直接下载二进制包

 cd /data/apps/
 wget http://mirror.bit.edu.cn/apache/kafka/1.0.0/kafka_2.11-1.0.0.tgz
 tar zxvf kafka_2.11-1.0.0.tgz 

解压后,编辑config/server.properties

#@broker.id必须唯一
broker.id=2
#@listeners使用自己的IP
listeners=PLAINTEXT://10.140.100.32:9092
log.dirs=/data/kafka-logs
zookeeper.connect=10.140.100.14:2181,10.140.100.26:2181,10.140.100.32:2181

启动kafka

/data/apps/kafka_2.11-1.0.0/bin/kafka-server-start.sh -daemon server.properties

四、创建测试topic

[root@kuiswenden-1 bin]# @@创建topic
[root@kuiswenden-1 bin]# ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 3 --topic test
Created topic "test".
[root@kuiswenden-1 bin]# @@列出topic
[root@kuiswenden-1 bin]# ./kafka-topics.sh --zookeeper localhost:2181 --list
test
test02
[root@kuiswenden-1 bin]# @@查看topic详情
[root@kuiswenden-1 bin]# ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic test
Topic:test    PartitionCount:3    ReplicationFactor:3    Configs:
    Topic: test    Partition: 0    Leader: 2    Replicas: 2,0,1    Isr: 2
    Topic: test    Partition: 1    Leader: 2    Replicas: 0,1,2    Isr: 2
    Topic: test    Partition: 2    Leader: 1    Replicas: 1,2,0    Isr: 1,0,2
[root@kuiswenden-1 bin]# 

五、模拟生产者和消费者

生产一部分数据

[root@kuiswenden-3 config]# /data/apps/kafka_2.11-1.0.0/bin/kafka-console-producer.sh --broker-list 10.140.100.32:9092 --topic test
>abcdefghijklmn
>

消费端查看数据

[root@kuiswenden-1 config]# /data/apps/kafka_2.11-1.0.0/bin/kafka-console-consumer.sh --zookeeper 0.140.100.14:2181 --from-beginning --topic test
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
abcdefghijklmn

六、简单压力测试

[root@kuiswenden-1 ~]# /data/apps/kafka_2.11-1.0.0/bin/kafka-producer-perf-test.sh --topic test --num-records 15000000 --record-size 100 --throughput 75000000 --producer-props acks=1 bootstrap.servers=10.140.100.32:9092,10.140.100.26:9092,10.140.100.14:9092 buffer.memory=67108864 compression.type=none batch.size=8196
2765043 records sent, 553008.6 records/sec (52.74 MB/sec), 70.1 ms avg latency, 388.0 max latency.
3991707 records sent, 798341.4 records/sec (76.14 MB/sec), 2.7 ms avg latency, 22.0 max latency.
4069509 records sent, 813901.8 records/sec (77.62 MB/sec), 2.4 ms avg latency, 52.0 max latency.
3999520 records sent, 799904.0 records/sec (76.28 MB/sec), 3.6 ms avg latency, 30.0 max latency.
15000000 records sent, 742096.670460 records/sec (70.77 MB/sec), 15.27 ms avg latency, 388.00 ms max latency, 2 ms 50th, 143 ms 95th, 221 ms 99th, 285 ms 99.9th.
[root@kuiswenden-1 ~]# 

可以看出kafka可以轻松吞吐50-60M数据。

参考资料

http://www.infoq.com/cn/articles/kafka-analysis-part-1?utm_source=infoq
http://blog.csdn.net/shirdrn/article/details/7183503
http://www.jasongj.com/kafka/high_throughput/

转载请注明:IPCPU-网络之路 » Apache Kafka专题二、Kafka安装和基本使用