Haproxy初见篇

January 04, 2021 7 min 0 s haproxy devops middleware

Haproxy

大纲

功能
- 使用方法
安装
实验
- 搭建
- 测试
调试
常见故障
- troubleshooting

内容

功能

开源负载均衡器。https://github.com/haproxy/haproxy

支持第四层（tcp）和第七层（http）转发。

通常被用来和Nginx和LVS比较。

事件驱动，单一进程模型，能支持非常大的并发连接数。配置的当的情况下，一台服务器可以支持2,000,000并发连接。https://www.freecodecamp.org/news/how-we-fine-tuned-haproxy-to-achieve-2-000-000-concurrent-ssl-connections-d017e61a4d27/

支持连接拒绝。

安装

打开1344 port

CentOS 或 Red Hat Enterprise Linux:

firewall-cmd --add-port=1344/tcp --permanent

firewall-cmd --reload
Ubuntu:

ufw allow 1344
Debian:

apt-get install iptables-persistent

iptables -A INPUT -p tcp --dport 1344 -j 接受

安装haproxy

CentOS 或 Red Hat Enterprise Linux:

yum install haproxy
SUSE Linux Enterprise Server:

zypper install haproxy
Ubuntu 和 Debian:

apt-get install haproxy

配置

https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/#:~:text=There%20are%20four%20essential%20sections,routed%20to%20your%20backend%20servers.

配置好后，执行下面的命令套用配置
```
sudo haproxy -f /etc/haproxy/haproxy.cfg
```
sudo haproxy -f /etc/haproxy/haproxy.cfgsudo haproxy -f /etc/haproxy/haproxy.cfg

logging

注意，global部分有关于log的字段。/dev/log是系统的socket。如果dev下没有log socket，则请

sudo service rsyslog start

实验

echo server

package main

import (
	"fmt"
	"log"
	"net/http"
	"os"
	"net"
)

const (
	CONN_HOST = "0.0.0.0"
	CONN_PORT = "3333"
	CONN_TYPE = "tcp"
)

func main() {
	flag := os.Getenv("http")
	if flag == "true" {
		http.HandleFunc("/", httpHandler)
		log.Fatal(http.ListenAndServe(":3333", nil))
	} else {
		l, err := net.Listen(CONN_TYPE, CONN_HOST+ ":"+CONN_PORT)
		if err != nil {
			fmt.Println("Error listening: ", err.Error())
			os.Exit(1)
		}
		defer l.Close()
		fmt.Println("Listening on " + CONN_HOST + "：" + CONN_PORT)
		for {
			conn, err := l.Accept()
			if err != nil {
				fmt.Println("Error accepting: ", err.Error())
				os.Exit(1)
			}
			fmt.Println("Accepted connection.")
			go handleRequest(conn)
		}
	}
}

func httpHandler(writer http.ResponseWriter, request *http.Request){
	fmt.Fprint(writer, "echo from server: " + os.Getenv("ServerName"))
}

func handleRequest(conn net.Conn) {
	buf := make([]byte, 1024)
	_, err := conn.Read(buf)
	if err != nil {
		fmt.Println("Error reading: ", err.Error())
	}
	fmt.Println("received " + string(buf))
	conn.Write([]byte("echo from server: " + os.Getenv("ServerName")))
	conn.Close()
}

注意由于需要容器化，监听地址必须是0.0.0.0而不是localhost。

这个echo server支持如下功能

tcp echo. 向server 发送任意消息，可以收到server的echo message
http echo. 访问/，可以收到server的echo message.
通过环境变量http=true 实现tcp/http echo 的切换。
为了区分服务器，我们设置ServerName环境变量。这个环境变量会出现在echo message中。

为了模拟haproxy多服务器环境，我们对echo server进行容器化。Dockerfile如下

FROM golang:alpine
ENV ServerName=test
ENV http=false
WORKDIR /build
COPY . .
RUN go mod download
RUN go build -o main .
WORKDIR /dist
RUN cp /build/main .
EXPOSE 3333/tcp
CMD ["/dist/main"]

build docker image:

docker build -t echo-server .

运行docker container

docker run -d -p 3333:3333 --env http=true --env ServerName=server1 echo-server
docker run -d -p 3334:3333 --env http=true --env ServerName=server2 echo-server
docker run -d -p 3335:3333 --env http=true --env ServerName=server3 echo-server
# 或
docker run -d -p 3333:3333 --env http=false --env ServerName=server1 echo-server
docker run -d -p 3334:3333 --env http=false --env ServerName=server2 echo-server
docker run -d -p 3335:3333 --env http=false --env ServerName=server3 echo-server

http

运行echo server.

docker run -d -p 3333:3333 --env http=true --env ServerName=server1 echo-server
docker run -d -p 3334:3333 --env http=true --env ServerName=server2 echo-server

配置haproxy，

sudo vim /etc/haproxy/haproxy.cfg

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
        
defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

global 和 default section 使用默认设置。

frontend main
        bind *:80
        use_backend echo-server
        default_backend echo-server

backend echo-server
        mode http
        option  forwardfor
        balance roundrobin
        cookie  SERVERID
        option  httpchk HEAD /index.html
        server  web01 localhost:3333 cookie web01 check inter 2000 rise 3 fall 3 weight 3
        server  web02 localhost:3334 cookie web02 check inter 2000 rise 3 fall 3 weight 3
        server  web03 localhost:3335 cookie web03 check inter 2000 rise 3 fall 3 weight 3

这里使用了最小的设置。

套用设定

sudo haproxy -f /etc/haproxy/haproxy.cfg

测试：

for i in $(seq 1 10); do curl localhost:80; done

输出：

echo from server: server1
echo from server: server2
echo from server: server1
echo from server: server2
echo from server: server1
echo from server: server2
echo from server: server1
echo from server: server2
echo from server: server1
echo from server: server2

这符合 roundrobin 算法的预期输出。

tcp

运行echo server

docker run -d -p 3333:3333 --env http=false --env ServerName=server1 echo-server
docker run -d -p 3334:3333 --env http=false --env ServerName=server2 echo-server

修改haproxy 配置。其中，

defaults
        log     global
        mode    tcp
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

frontend main
        bind *:1344
        use_backend echo-server
        default_backend echo-server

backend echo-server
        mode tcp
        balance roundrobin
        option  httpchk HEAD /index.html
        server  web01 localhost:3333 check inter 2000 rise 3 fall 3 weight 3
        server  web02 localhost:3334 check inter 2000 rise 3 fall 3 weight 3
        server  web03 localhost:3335 check inter 2000 rise 3 fall 3 weight 3

测试

for i in $(seq 1 10); do echo "hello" | nc localhost 1344; done

输出为：

echo from server: server1
echo from server: server2
echo from server: server1
echo from server: server2
echo from server: server1
echo from server: server2
echo from server: server1
echo from server: server2
echo from server: server1
echo from server: server2

复合预期。

结论

至此，对haproxy的初见，安装，配置，初步测试实验都已完成。

目前仍有未进行的工作：

进阶功能的测试，包括
1. 拒绝连接
2. reverse proxy
3. sticky session
大部分高级的配置仍未cover
常见故障troubleshooting和调试

将在未来陆续进行。