{"id":987,"date":"2018-05-22T22:40:57","date_gmt":"2018-05-22T14:40:57","guid":{"rendered":"https:\/\/www.ipcpu.com\/?p=987"},"modified":"2018-05-22T22:40:57","modified_gmt":"2018-05-22T14:40:57","slug":"kafka-lag-burrow","status":"publish","type":"post","link":"https:\/\/c.ipcpu.com\/2018\/05\/kafka-lag-burrow\/","title":{"rendered":"Kafka\u6d88\u8d39\u79ef\u538bLag\u76d1\u63a7\u5de5\u5177Burrow\u7684\u4f7f\u7528"},"content":{"rendered":"
Kafka\u6d88\u8d39\u79ef\u538bLag\u76d1\u63a7\u5de5\u5177Burrow\u7684\u4f7f\u7528.md<\/p>\n
Kafka\u662f\u4e1a\u5185\u6d41\u884c\u7684\u65e5\u5fd7\u961f\u5217\u5904\u7406\u8f6f\u4ef6\uff0c\u4f7f\u7528\u6781\u4e3a\u5e7f\u6cdb\uff0c\u4f46\u662f\u5173\u4e8e\u5bf9Kafka\u7684\u76d1\u63a7\u95ee\u9898\uff0c\u7f51\u4e0a\u7684\u6587\u7ae0\u548c\u76f8\u5173\u7684\u8f6f\u4ef6\u5c31\u6bd4\u8f83\u5c11\u4e86\u3002<\/p>\n
KafkaManager\u7b97\u662f\u51e0\u4e2a\u5f00\u6e90\u8f6f\u4ef6\u91cc\u505a\u7684\u6bd4\u8f83\u597d\u7684\u3002\u53ef\u4ee5\u5b9e\u65f6\u770b\u5230consumer\u6d88\u8d39\u8fdb\u5ea6\u548cLag\u60c5\u51b5\uff0c\u5982\u4e0b\u56fe<\/p>\n
<\/p>\n
\u5982\u679c\u7528\u4e8e\u65e5\u5e38\u7684\u7ba1\u7406\u8fd8\u662f\u5f88\u4e0d\u9519\u7684\uff0c\u4f46\u662f\u8981\u7528\u4e8e\u76d1\u63a7\uff0c\u8fd8\u662f\u6bd4\u8f83\u9ebb\u70e6\u7684\uff0c\u4e4b\u524d\u7b14\u8005\u4e5f\u5199\u8fc7Python\u811a\u672c\u53bb\u6293\u53d6kafka manager\u7684Lag\u6570\u636e\uff0c\u4f46\u662ftopic\u3001ConsumerGroupID\u9700\u8981\u7ec4\u5408\u624d\u80fd\u53d6\u5230\u6570\u636e\uff0c\u6bd4\u8f83\u9ebb\u70e6\u3002\u6293\u53d6\u5b8c\u6210\u540e\u5f55\u5165\u76d1\u63a7\u7cfb\u7edf\uff0c\u5982\u4e0b\u56fe\u6240\u793a\u3002\u53ef\u4ee5\u7ed9Lag\u8bbe\u7f6e\u9608\u503c\u8fdb\u884c\u62a5\u8b66\u3002<\/p>\n
<\/p>\n
Burrow\u662fLinkedIn\u5f00\u6e90\u7684\u4e00\u6b3e\u4e13\u95e8\u76d1\u63a7consumer lag\u7684\u6846\u67b6\u3002
\n\u76ee\u524d\u6765\u770b\u66f4\u65b0\u5f88\u7f13\u6162\uff0c\u6d3b\u8dc3\u5ea6\u4e0d\u9ad8\uff0c\u6700\u65b0\u7248\u672c\u662fv1.1.0\u30022018-05-15\u53d1\u5e03\uff0c\u4fee\u590d\u4e86\u4e0d\u5c11bug\u3002<\/p>\n
Burrow\u7684\u7279\u70b9\u5982\u4e0b<\/p>\n
\n\u4f7f\u7528Burrow\u76d1\u63a7kafka, \u4e0d\u9700\u8981\u9884\u5148\u8bbe\u7f6elag\u7684\u9608\u503c, \u4ed6\u5b8c\u5168\u662f\u57fa\u4e8e\u6d88\u8d39\u8fc7\u7a0b\u7684\u52a8\u6001\u8bc4\u4f30
\n Burrow\u652f\u6301\u8bfb\u53d6kafka topic\u548c,zookeeper\u4e24\u79cd\u65b9\u5f0f\u7684offset\uff0c\u5bf9\u4e8e\u65b0\u8001\u7248\u672ckafka\u90fd\u53ef\u4ee5\u5f88\u597d\u652f\u6301
\n Burrow\u652f\u6301http, email\u7c7b\u578b\u7684\u62a5\u8b66
\n Burrow\u9ed8\u8ba4\u53ea\u63d0\u4f9bHTTP\u63a5\u53e3(HTTP endpoint)\uff0c\u6570\u636e\u4e3ajson\u683c\u5f0f\uff0c\u6ca1\u6709web UI<\/p>\n<\/blockquote>\n\u4e09\u3001Burrow\u5b89\u88c5\u548c\u914d\u7f6e<\/h2>\n
github\u7f51\u7ad9\u4e0a\u53ef\u4ee5\u76f4\u63a5\u4e0b\u8f7d\u7f16\u8bd1\u597d\u7684\u4e8c\u8fdb\u5236\u5305\uff0c\u53ef\u4ee5\u76f4\u63a5\u4e0b\u8f7d\u4f7f\u7528\uff0c\u4e5f\u53ef\u4ee5\u4f7f\u7528docker\u955c\u50cf\u3002<\/p>\n
\u6211\u4eec\u4e0b\u8f7d\u4e86\u4e8c\u8fdb\u5236\u5305\uff0c\u89e3\u538b\u540e\u53ef\u4ee5\u770b\u5230burrow\u4e8c\u8fdb\u5236\u6587\u4ef6\uff0c\u548cconfig\u914d\u7f6e\u6587\u4ef6\u76ee\u5f55\u3002<\/p>\n
Burrow\u7684\u4e3b\u914d\u7f6e\u6587\u4ef6\u662fconfig\/burrow.toml\uff0c\u57fa\u4e8eTOML\u683c\u5f0f\u3002\u6211\u4eec\u4fee\u6539\u5b8c\u6210\u540e\u7684\u914d\u7f6e\u5982\u4e0b\uff1a<\/p>\n
\n#\u57fa\u7840\u9009\u9879\n[general]\npidfile=\"burrow.pid\"\nstdout-logfile=\"logs\/burrow.out\"\naccess-control-allow-origin=\"mysite.example.com\"\n#\u65e5\u5fd7\u9009\u9879\n[logging]\nfilename=\"logs\/burrow.log\"\nlevel=\"info\"\nmaxsize=100\nmaxbackups=30\nmaxage=10\nuse-localtime=false\nuse-compression=true\n#zookeeper\u9009\u9879\uff0c\u8fd9\u4e2azookeeper\u4f7f\u7528burrow\u81ea\u5df1\u7528\u7684\n[zookeeper]\nservers=[\"10.128.0.2:2181\", ]\ntimeout=6\nroot-path=\"\/burrow\"\n#burrow\u4f5c\u4e3a\u5ba2\u6237\u7aef\u7684\u914d\u7f6e\n[client-profile.burrowclient]\nclient-id=\"burrowclient\"\nkafka-version=\"0.10.0\"\n#HTTP\u76d1\u542c\u914d\u7f6e\uff0c\u53ef\u4ee5\u4f7f\u7528HTTPS\n[httpserver.default]\naddress=\":8000\"\n#\u5b58\u50a8\u9009\u9879\n[storage.default]\nclass-name=\"inmemory\"\nworkers=20\nintervals=15\nexpire-group=604800\nmin-distance=1\n#\u62a5\u8b66\u9009\u9879\n[notifier.default]\nclass-name=\"email\"\ninterval=30\nthreshold=2\ngroup-whitelist=\"^important-group-prefix.*$\"\ngroup-blacklist=\"^not-this-group$\"\ntemplate-open=\"config\/default-email.tmpl\"\nserver=\"127.0.0.1\"\nport=25\nfrom=\"root@localhost.com\"\nto=\"hello@ipcpu.com\"\n#\n# kafka cluster config here\n#\u63a5\u4e0b\u6765\u662fKafka\u76f8\u5173\u7684\u5185\u5bb9\n#\n[cluster.bu-agent-kafka]\nclass-name=\"kafka\"\nservers=[ \"10.128.0.65:9092\", \"10.128.0.66:9092\", \"10.128.0.67:9092\" ]\nclient-profile=\"burrowclient\"\ntopic-refresh=300\noffset-refresh=60\n[consumer.bu-agent-kafka]\nclass-name=\"kafka\"\ncluster=\"bill-kafka\"\nservers=[ \"10.128.0.65:9092\", \"10.128.0.66:9092\", \"10.128.0.67:9092\" ]\nclient-profile=\"burrowclient\"\noffsets-topic=\"__consumer_offsets\"\nstart-latest=true\ngroup-whitelist=\".*\"\ngroup-blacklist=\"^not-this-group$\"\n#@\u6ce8\u610f\u8fd9\u91cc\u4f7f\u7528\u4e86cluster.kafkaname\u548cconsumer.kafkaname\u4e24\u4e2a\u914d\u7f6e\u7ec4\uff0c\n#@\u7b2c\u4e00\u4e2a\u7528\u6765\u83b7\u53d6topic\u548c\u6700\u65b0offset\u4fe1\u606f\uff0c\n#@\u7b2c\u4e8c\u4e2a\u7528\u6765\u83b7\u53d6\u6d88\u8d39\u7ec4\u548c\u6d88\u8d39\u7ec4offset\u53caLag\u3002<\/code><\/pre>\n<\/div>\n
\u7f16\u5199\u597d\u914d\u7f6e\u6587\u4ef6\u540e\uff0c\u6211\u4eec\u5c31\u53ef\u4ee5\u542f\u52a8\u4e86<\/p>\n
\nnohup \/data\/apps\/burrow\/burrow -config-dir=\/data\/apps\/burrow\/config\/ &<\/code><\/pre>\n<\/div>\n
\u9ed8\u8ba4\u60c5\u51b5\u4e0b\uff0cburrow\u76d1\u542c8000\u7aef\u53e3\uff0c\u53ef\u4ee5\u5728\u914d\u7f6e\u6587\u4ef6\u4e2d\u4fee\u6539\u3002<\/p>\n
\u5982\u679c\u53d1\u751f\u914d\u7f6e\u6587\u4ef6\u9519\u8bef\uff0c\u4f1a\u8bb0\u5f55\u5728logs\/burrow.log\u4e2d\uff0c\u524d\u53f0\u4e0d\u4f1a\u62a5\u9519\u3002<\/p>\n
\u56db\u3001Burrow\u7684\u7b80\u5355\u4f7f\u7528<\/h2>\n
Burrow\u63d0\u4f9b\u7684\u662fHTTP Endpoint\uff0c\u4f7f\u7528\u8d77\u6765\u4e5f\u7279\u522b\u7b80\u5355\uff0c\u5982\u4e0b<\/p>\n
\n[root@mt ~]##@\u5217\u51fa\u6240\u6709\u76d1\u63a7\u7684Kafka\u96c6\u7fa4\n[root@mt ~]# curl -s http:\/\/10.128.0.2:8000\/v3\/kafka |jq\n{\n \"error\": false,\n \"message\": \"cluster list returned\",\n \"clusters\": [\n \"nginxlog\",\n \"bill-kafka\"\n ],\n \"request\": {\n \"url\": \"\/v3\/kafka\",\n \"host\": \"zabbix\"\n }\n}\n[root@mt ~]#\n[root@mt ~]# #@\u5217\u51fa\u6240\u6709\u6d88\u8d39\u7ec4\n[root@mt ~]# curl -s http:\/\/10.128.0.2:8000\/v3\/kafka\/bill-kafka\/consumer |jq\n{\n \"error\": false,\n \"message\": \"consumer list returned\",\n \"consumers\": [\n \"group1\",\n \"group_bill\",\n \"report_gid\"\n ],\n \"request\": {\n \"url\": \"\/v3\/kafka\/bill-kafka\/consumer\",\n \"host\": \"zabbix\"\n }\n}\n[root@mt ~]# #@\u67e5\u770b\u6d88\u8d39\u7ec4\u5065\u5eb7\u72b6\u6001\n[root@mt ~]# curl -s http:\/\/10.128.0.2:8000\/v3\/kafka\/bill-kafka\/consumer\/group1\/status |jq\n{\n \"error\": false,\n \"message\": \"consumer status returned\",\n \"status\": {\n \"cluster\": \"bill-kafka\",\n \"group\": \"group1\",\n \"status\": \"OK\",\n \"complete\": 1,\n \"partitions\": [],\n \"partition_count\": 18,\n \"maxlag\": {\n \"topic\": \"shellActivity\",\n \"partition\": 0,\n \"owner\": \"\",\n \"status\": \"OK\",\n \"start\": {\n \"offset\": 101333573,\n \"timestamp\": 1526997873289,\n \"lag\": 0\n },\n \"end\": {\n \"offset\": 101333604,\n \"timestamp\": 1526997891689,\n \"lag\": 0\n },\n \"current_lag\": 0,\n \"complete\": 1\n },\n \"totallag\": 0\n },\n \"request\": {\n \"url\": \"\/v3\/kafka\/bill-kafka\/consumer\/group1\/status\",\n \"host\": \"zabbix\"\n }\n}<\/code><\/pre>\n<\/div>\n
\u5982\u4e0a\uff0c\u5982\u679c\u60f3\u76d1\u63a7\u6d88\u8d39\u7ec4\u5065\u5eb7\u72b6\u6001\uff0c\u662f\u5426Lag\u8fc7\u5927\uff0c\u5c31\u662f\u7528\"\/v3\/kafka\/(cluster)\/consumer\/(group)\/status\"\u67e5\u770b\u662f\u5426\u4e3aOK\u3002<\/p>\n
\u5e38\u7528HTTP Endpoint\u5982\u4e0b<\/p>\n
\n\n