{"id":1304,"date":"2019-11-13T05:58:57","date_gmt":"2019-11-13T05:58:57","guid":{"rendered":"https:\/\/www.ipcpu.com\/?p=1304"},"modified":"2020-05-11T06:05:07","modified_gmt":"2020-05-11T06:05:07","slug":"tcp-1million","status":"publish","type":"post","link":"https:\/\/c.ipcpu.com\/2019\/11\/tcp-1million\/","title":{"rendered":"\u767e\u4e07TCP\u5e76\u53d1\u7684Linux\u7cfb\u7edf\u914d\u7f6e"},"content":{"rendered":"<p>\u767e\u4e07\u5e76\u53d1\u914d\u7f6e.md <br \/>\n<img decoding=\"async\" src=\"https:\/\/www.ipcpu.com\/wp-content\/uploads\/2020\/05\/777716bba4c00e6cea4645b809a75871_6969de57-e0a8-44c2-905c-4c4a3d1c6fd1.jpg\" style=\"vertical-align: bottom; max-width: 100%;\"><\/p>\n<h2 id=\"\u6982\u8ff0\">\u6982\u8ff0<\/h2>\n<p>\u79d1\u6280\u53d1\u5c55\u592a\u5feb\uff0c\u73b0\u5728\u4e00\u53f0\u666e\u901a\u7684\u673a\u5668\u90fd\u53ef\u4ee5\u4fdd\u6301\u767e\u4e07\u5343\u4e07\u7684TCP\u5e76\u53d1\u8fde\u63a5\u6216\u662f\u8f6c\u53d1\uff08\u5e76\u53d1\u6570\uff0c\u8fd9\u91cc\u6211\u4eec\u548c\u963f\u91cc\u4e91\u770b\u9f50\uff0c\u4ee5TCP\u7684ESTABLISH\u8fde\u63a5\u6570\u6765\u8ba1\u7b97\uff0c\u8fd9\u91cc\u53ea\u662f\u4fdd\u6301TCP\u8fde\u63a5\uff0c\u5e76\u975eQPS\u6216\u8005TPS\uff09\uff0c\u4f46Linux\u7684\u4e00\u4e9b\u9ed8\u8ba4\u8bbe\u5b9a\u8fc7\u4e8e\u4fdd\u5b88\uff0c\u6216\u8005\u662f\u5341\u591a\u5e74\u6839\u672c\u5c31\u6ca1\u52a8\u8fc7\uff0c\u4e00\u6b21\u60f3\u8981\u8fbe\u5230\u767e\u4e07\u7684\u5e76\u53d1\uff0c\u5f88\u591a\u53c2\u6570\u8fd8\u662f\u9700\u8981\u7814\u7a76\u7422\u78e8\u7684\u3002<\/p>\n<h2 id=\"\u4e00\u3001\u6587\u4ef6\u6253\u5f00\u6570\">\u4e00\u3001\u6587\u4ef6\u6253\u5f00\u6570<\/h2>\n<p>Linux\u9ed8\u8ba4\u7684\u6587\u4ef6\u6253\u5f00\u6570\u662f1024\uff0c\u8fd9\u4e2a\u57fa\u672c\u4e0a\u6ee1\u8db3\u4e0d\u4e86\u4efb\u4f55\u7cfb\u7edf\u7684\u9700\u6c42\uff0c\u6587\u4ef6\u6253\u5f00\u6570\u8d85\u8fc71024\u5c31\u4f1a\u62a5\"too many open files\"\u9519\u8bef\u3002<\/p>\n<div>\n<pre><code>#vi sysctl.conf\nfs.nr_open=10240000\nfs.file-max=10240000\n# vi \/etc\/security\/limits.conf\n* soft nofile 8420000 \n* hard nofile 8420000<\/code><\/pre>\n<\/div>\n<p>\u4eceCentOS7\u5f00\u59cbsystemd\u88ab\u5e7f\u6cdb\u5e94\u8be5\uff0c\u5e94\u8be5\u7279\u522b\u6ce8\u610f\uff0c\u5efa\u8bae\u6dfb\u52a0\u5230\u5168\u5c40\u6587\u4ef6\u4e2d<\/p>\n<div>\n<pre><code>#vi \/etc\/systemd\/system.conf\nDefaultLimitCORE=infinity\nDefaultLimitNOFILE=820000\nDefaultLimitNPROC=410000\n#\u6267\u884csystemctl daemon-reexec\u751f\u6548<\/code><\/pre>\n<\/div>\n<p>\u8fd8\u6709\u5728systemd\u7684\u914d\u7f6e\u6587\u4ef6(CentOS7)\u4e2dinfinity\u5e76\u4e0d\u662f\u65e0\u7a77\u5927\uff0c\u800c\u662f65536\u3002<\/p>\n<h2 id=\"\u4e8c\u3001nf_conntrack\">\u4e8c\u3001nf_conntrack<\/h2>\n<p>nf_conntrack\u5e76\u4e0d\u662f\u4e00\u4e2a\u5e38\u7528\u7684\u6a21\u5757\uff0c\u5982\u679c\u4e0d\u4f7f\u7528iptables\u7684\u8bdd\u3002but\uff0cDocker\u548cKubernetes\u4ee5\u53caOpenStack\u53ef\u90fd\u662fnf_conntrack\u91cd\u5ea6\u4f9d\u8d56\u7528\u6237\u3002 <br \/>\nnf_conntrack\u6570\u503c\u8bbe\u7f6e\u8fc7\u5c0f\uff0c\u4f1a\u51fa\u73b0\"nf_conntrack: table full, dropping packet\"\u9519\u8bef\u3002<\/p>\n<div>\n<pre><code>#nf_conntrack\u7684\u53c2\u6570\u53ef\u4ee5\u5728\u7ebf\u4fee\u6539\necho \"1048576\" &gt; \/sys\/module\/nf_conntrack\/parameters\/hashsize\n#\/proc\/sys\/net\/netfilter\/nf_conntrack_buckets\u8fd9\u4e2a\u6587\u4ef6\u53ea\u8bfb\u4e0d\u8ba9\u6539\uff0ckernel4.8\u4ee5\u540e\u53ef\u4ee5\u6539\nsysctl -w net.netfilter.nf_conntrack_max=4195060\nsysctl -w net.nf_conntrack_max=4195060\nsysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=600 #\u9ed8\u8ba45\u5929\nsysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=60 #\u9ed8\u8ba4120s<\/code><\/pre>\n<\/div>\n<p>Kubernetes\u4e2d\u7684kube-proxy\u5c31\u662f\u76f4\u63a5\u4fee\u6539\u7684\uff0c\u5e76\u4e14conntrack_buckets*4=conntrack_max <br \/>\n\u4e5f\u53ef\u4ee5\u4fee\u6539\u914d\u7f6e\u6587\u4ef6\/etc\/modprobe.d\/custom.conf\uff0c\u91cd\u542f\u540e\u76f4\u63a5\u751f\u6548<\/p>\n<div>\n<pre><code>options nf_conntrack hashsize=288576\n#options nf_conntrack hashsize=1048576\n#conntrack_buckets*8=conntrack_max\n#\u8bbe\u7f6e\u592a\u9ad8\u4f1a\u62a5nf_conntrack: falling back to vmalloc\uff0c\u53ef\u80fd\u4e0e\u673a\u5668\u5185\u5b58\u6709\u5173<\/code><\/pre>\n<\/div>\n<p>\u8fd9\u91cc\u518d\u63d0\u793a\u4e00\u4e2a\u9ad8\u98ce\u9669\u547d\u4ee4\uff0ciptables -t nat -nvL \uff0c\u6267\u884c\u6b64\u547d\u4ee4\u540e\u4f1a\u81ea\u52a8\u52a0\u8f7dnf_conntrack \u6a21\u5757\uff0c\u614e\u7528\uff01<\/p>\n<h2 id=\"\u4e09\u3001TCP\u53c2\u6570\u8bbe\u5b9a\">\u4e09\u3001TCP\u53c2\u6570\u8bbe\u5b9a<\/h2>\n<h4 id=\"3.1TCP\u63a5\u6536\u548c\u53d1\u9001\u7f13\u51b2\u533a\u8bbe\u7f6e\">3.1    TCP\u63a5\u6536\u548c\u53d1\u9001\u7f13\u51b2\u533a\u8bbe\u7f6e<\/h4>\n<div>\n<pre><code>net.ipv4.tcp_rmem = 4096 8388608 16777216\nnet.ipv4.tcp_wmem = 4096 8388608 16777216\nnet.core.wmem_default = 8388608\nnet.core.rmem_default = 8388608\nnet.core.rmem_max = 16777216\nnet.core.wmem_max = 16777216\n#\u5355\u4f4d\u662f\u5b57\u8282<\/code><\/pre>\n<\/div>\n<p>Assume RTT in data center with 10GbE network = 1~100ms <br \/>\nBDP=0.1sec * 10Gbps \/8 = 134217728 Bytes = 134M Bytes <br \/>\n\u5b9e\u9645\u4e0a1G\u5e26\u5bbd\uff0c\u6309\u716716M\u8bbe\u7f6e\u5c31\u53ef\u4ee5\uff0crmem_max=BDP\uff0cwmem_default=BDP\/2 <br \/>\n\u6570\u503c\u8bbe\u7f6e\u8fc7\u5c0f\uff0c\u53ef\u80fd\u4f1a\u51fa\u73b0\"TCP: too many of orphaned sockets\"\u9519\u8bef\u3002<\/p>\n<h4 id=\"3.2TCP\u53ef\u4ee5\u4f7f\u7528\u7684\u5185\u5b58\">3.2    TCP\u53ef\u4ee5\u4f7f\u7528\u7684\u5185\u5b58<\/h4>\n<div>\n<pre><code>#vim \/etc\/sysctl.conf\n#TCP\u5185\u5b58\u8bbe\u5b9a8G\uff0c32G\uff0c96G\nnet.ipv4.tcp_mem = 2097152 8388608 25165824\n#\u5355\u4f4d\u662f\u9875<\/code><\/pre>\n<\/div>\n<p>\u7b2c\u4e00\u4e2a\u8868\u793a\u4f4e\u4e8e8G\u4e0d\u505a\u9650\u5236\uff0c\u7b2c\u4e8c\u503c\u8868\u793a\u8d85\u51fa32G\u8fdb\u884cpressure(\u538b\u5236?)\uff0c\u8d85\u51fa\u7b2c\u4e09\u4e2a\u503c\u4f1a\u62a5\"Out of Socket memory\"<\/p>\n<h4 id=\"3.3\u5176\u4ed6TCP\u53c2\u6570\u8bbe\u7f6e\">3.3    \u5176\u4ed6TCP\u53c2\u6570\u8bbe\u7f6e<\/h4>\n<div>\n<pre><code>net.ipv4.tcp_sack = 1\nnet.ipv4.tcp_window_scaling = 1\nnet.ipv4.tcp_timestamps = 1\nnet.ipv4.tcp_synack_retries = 1\nnet.ipv4.tcp_syn_retries = 1\nnet.ipv4.tcp_tw_recycle = 0\nnet.ipv4.tcp_tw_reuse = 0<\/code><\/pre>\n<\/div>\n<p>\u6ce8\u610f\u7ecf\u5178\u8bdd\u9898\uff0ctcp_tw_recycle \u548ctcp_timestamps \u4e0d\u8981\u540c\u65f6\u6253\u5f00\u3002<\/p>\n<h2 id=\"\u6d4b\u8bd5\u5de5\u5177\">\u6d4b\u8bd5\u5de5\u5177<\/h2>\n<p><a href=\"https:\/\/github.com\/smallnest\/1m-go-tcp-server\">https:\/\/github.com\/smallnest\/1m-go-tcp-server<\/a><\/p>\n<p><wiz_tmp_tag id=\"wiz-table-range-border\" contenteditable=\"false\" style=\"display: none;\"><\/p>\n<div id=\"wiz-table-col-line\" style=\"display: none;\"><\/div>\n<div id=\"wiz-table-row-line\" style=\"display: none;\"><\/div>\n<div id=\"wiz-table-range-border_start\" style=\"display: none;\">\n<div id=\"wiz-table-range-border_start_top\"><\/div>\n<div id=\"wiz-table-range-border_start_right\"><\/div>\n<div id=\"wiz-table-range-border_start_bottom\"><\/div>\n<div id=\"wiz-table-range-border_start_left\"><\/div>\n<div id=\"wiz-table-range-border_start_dot\"><\/div>\n<\/div>\n<div id=\"wiz-table-range-border_range\" style=\"display: none;\">\n<div id=\"wiz-table-range-border_range_top\"><\/div>\n<div id=\"wiz-table-range-border_range_right\"><\/div>\n<div id=\"wiz-table-range-border_range_bottom\"><\/div>\n<div id=\"wiz-table-range-border_range_left\"><\/div>\n<div id=\"wiz-table-range-border_range_dot\"><\/div>\n<\/div>\n<p><\/wiz_tmp_tag><\/p>\n<p>\u8f6c\u8f7d\u8bf7\u6ce8\u660e\uff1a<a href=\"https:\/\/c.ipcpu.com\">IPCPU-\u7f51\u7edc\u4e4b\u8def<\/a> &raquo; <a href=\"https:\/\/c.ipcpu.com\/2019\/11\/tcp-1million\/\">\u767e\u4e07TCP\u5e76\u53d1\u7684Linux\u7cfb\u7edf\u914d\u7f6e<\/a><\/p>","protected":false},"excerpt":{"rendered":"<p>\u767e\u4e07\u5e76\u53d1\u914d\u7f6e.md \u6982\u8ff0 \u79d1\u6280\u53d1\u5c55\u592a\u5feb\uff0c\u73b0\u5728\u4e00\u53f0\u666e\u901a\u7684\u673a\u5668\u90fd\u53ef\u4ee5\u4fdd\u6301\u767e\u4e07\u5343\u4e07\u7684TCP\u5e76\u53d1\u8fde\u63a5\u6216\u662f\u8f6c\u53d1\uff08\u5e76\u53d1\u6570\uff0c\u8fd9\u91cc\u6211\u4eec\u548c\u963f\u91cc\u4e91\u770b\u9f50\uff0c\u4ee5TCP\u7684ESTABLISH\u8fde\u63a5\u6570\u6765\u8ba1\u7b97\uff0c\u8fd9\u91cc\u53ea\u662f\u4fdd\u6301TCP\u8fde\u63a5\uff0c\u5e76\u975eQPS\u6216\u8005TPS\uff09\uff0c\u4f46Linux\u7684\u4e00\u4e9b\u9ed8\u8ba4\u8bbe\u5b9a\u8fc7\u4e8e\u4fdd\u5b88\uff0c\u6216\u8005\u662f\u5341\u591a\u5e74\u6839\u672c\u5c31\u6ca1\u52a8\u8fc7\uff0c\u4e00\u6b21\u60f3\u8981\u8fbe\u5230\u767e\u4e07\u7684\u5e76\u53d1\uff0c\u5f88\u591a\u53c2\u6570\u8fd8\u662f\u9700\u8981\u7814\u7a76\u7422\u78e8\u7684\u3002 \u4e00\u3001\u6587\u4ef6\u6253\u5f00\u6570 Linux\u9ed8\u8ba4\u7684\u6587\u4ef6\u6253\u5f00\u6570\u662f1024\uff0c\u8fd9\u4e2a\u57fa\u672c\u4e0a\u6ee1\u8db3\u4e0d\u4e86\u4efb\u4f55\u7cfb\u7edf\u7684\u9700\u6c42\uff0c\u6587\u4ef6\u6253\u5f00\u6570\u8d85\u8fc71024\u5c31\u4f1a\u62a5&#8221;too many open files&#8221;\u9519\u8bef\u3002 #vi sysctl.conf fs.nr_open=10240000 fs.file-max=10240000 # vi \/etc\/security\/limits.conf * soft nofile 8420000 * hard nofile 8420000 \u4eceCentOS7\u5f00\u59cbsystemd\u88ab\u5e7f\u6cdb\u5e94\u8be5\uff0c\u5e94\u8be5\u7279\u522b\u6ce8\u610f\uff0c\u5efa\u8bae\u6dfb\u52a0\u5230\u5168\u5c40\u6587\u4ef6\u4e2d #vi \/etc\/systemd\/system.conf DefaultLimitCORE=infinity DefaultLimitNOFILE=820000 DefaultLimitNPROC=410000 #\u6267\u884csystemctl daemon-reexec\u751f\u6548 \u8fd8\u6709\u5728systemd\u7684\u914d\u7f6e\u6587\u4ef6(CentOS7)\u4e2dinfinity\u5e76\u4e0d\u662f\u65e0\u7a77\u5927\uff0c\u800c\u662f65536\u3002 \u4e8c\u3001nf_conntrack nf_conntrack\u5e76\u4e0d\u662f\u4e00\u4e2a\u5e38\u7528\u7684\u6a21\u5757\uff0c\u5982\u679c\u4e0d\u4f7f\u7528iptables\u7684\u8bdd\u3002but\uff0cDocker\u548cKubernetes\u4ee5\u53caOpenStack\u53ef\u90fd\u662fnf_conntrack\u91cd\u5ea6\u4f9d\u8d56\u7528\u6237\u3002 nf_conntrack\u6570\u503c\u8bbe\u7f6e\u8fc7\u5c0f\uff0c\u4f1a\u51fa\u73b0&#8221;nf_conntrack: table full, dropping packet&#8221;\u9519\u8bef\u3002 #nf_conntrack\u7684\u53c2\u6570\u53ef\u4ee5\u5728\u7ebf\u4fee\u6539 echo &#8220;1048576&#8221; &gt; \/sys\/module\/nf_conntrack\/parameters\/hashsize #\/proc\/sys\/net\/netfilter\/nf_conntrack_buckets\u8fd9\u4e2a\u6587\u4ef6\u53ea\u8bfb\u4e0d\u8ba9\u6539\uff0ckernel4.8\u4ee5\u540e\u53ef\u4ee5\u6539 sysctl -w net.netfilter.nf_conntrack_max=4195060 sysctl -w net.nf_conntrack_max=4195060 sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=600 #\u9ed8\u8ba45\u5929 [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[13],"tags":[17,107,206,207],"_links":{"self":[{"href":"https:\/\/c.ipcpu.com\/wp-json\/wp\/v2\/posts\/1304"}],"collection":[{"href":"https:\/\/c.ipcpu.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/c.ipcpu.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/c.ipcpu.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/c.ipcpu.com\/wp-json\/wp\/v2\/comments?post=1304"}],"version-history":[{"count":1,"href":"https:\/\/c.ipcpu.com\/wp-json\/wp\/v2\/posts\/1304\/revisions"}],"predecessor-version":[{"id":1305,"href":"https:\/\/c.ipcpu.com\/wp-json\/wp\/v2\/posts\/1304\/revisions\/1305"}],"wp:attachment":[{"href":"https:\/\/c.ipcpu.com\/wp-json\/wp\/v2\/media?parent=1304"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/c.ipcpu.com\/wp-json\/wp\/v2\/categories?post=1304"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/c.ipcpu.com\/wp-json\/wp\/v2\/tags?post=1304"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}