原来我们使用云厂商的应用程序网关配置HTTPS双向认证,但后来客户提出要求使用证书吊销列表CRL。遗憾的是,云厂商的应用程序网关并不支持通过CRL进行吊销检查,这使得我们不得不将HTTPS双向认证迁移到Kubernetes集群中的ingress-nginx-controller。接下来,我们将详细描述在Kubernetes环境中生成服务端和客户端证书,并配置Ingress进行HTTPS双向认证的步骤。此外,我们还会探讨如何配置客户端证书的证书吊销列表,以满足客户的需求。
生成服务端和客户端证书并配置Ingress
- 生成服务端用的CA证书和密钥、服务端证书和密钥
mkdir ingress-mtls && ingress-mtls
# Generate the CA Key and Certificate
# 这里openssl生成的证书是pem格式而非der格式,指定证书格式可以使用`-outform arg output format - DER or PEM`参数
openssl req -x509 -sha256 -newkey rsa:4096 -keyout server-ca-key.pem -out server-ca-cert.pem -days 36525 -nodes -subj '/CN=Aispeech Server Cert Authority'
# Generate the Server Key, and Certificate and Sign with the CA Certificate
# 使用X509v3 Subject Alternative Name
echo subjectAltName = DNS:mtls.dev.ityoudao.cn > server-extfile.cnf
openssl req -new -newkey rsa:4096 -keyout server-key.pem -out server.csr -nodes -subj '/CN=mtls.dev.ityoudao.cn'
openssl x509 -req -sha256 -days 365 -in server.csr -CA server-ca-cert.pem -CAkey server-ca-key.pem -extfile server-extfile.cnf -set_serial 01 -out server-cert.pem
# 创建完整的证书链文件
cat server-cert.pem server-ca-cert.pem > server-cert-chain.pem
# ll server*
-rw-r--r--. 1 root root 1846 1月 5 16:39 server-ca-cert.pem
-rw-r--r--. 1 root root 3268 1月 5 16:39 server-ca-key.pem
-rw-r--r--. 1 root root 1704 1月 5 16:39 server-cert.pem
-rw-r--r--. 1 root root 1598 1月 5 16:39 server.csr
-rw-r--r--. 1 root root 3272 1月 5 16:39 server-key.pem
- 生成客户端用的CA证书和密钥、客户端证书和密钥
# Generate the CA Key and Certificate
openssl req -x509 -sha256 -newkey rsa:4096 -keyout client-ca-key.pem -out client-ca-cert.pem -days 36525 -nodes -subj '/CN=Aispeech Client Cert Authority'
# Generate the Client Key, and Certificate and Sign with the CA Certificate
openssl req -new -newkey rsa:4096 -keyout client-key.pem -out client.csr -nodes -subj '/CN=Client001'
openssl x509 -req -sha256 -days 365 -in client.csr -CA client-ca-cert.pem -CAkey client-ca-key.pem -set_serial 01 -out client-cert.pem
# ll client*
-rw-r--r--. 1 root root 1846 1月 5 15:52 client-ca-cert.pem
-rw-r--r--. 1 root root 3272 1月 5 15:52 client-ca-key.pem
-rw-r--r--. 1 root root 1688 1月 5 15:52 client-cert.pem
-rw-r--r--. 1 root root 1586 1月 5 15:52 client.csr
-rw-r--r--. 1 root root 3268 1月 5 15:52 client-key.pem
- 创建保存服务端证书和密钥的Secret
kubectl create secret tls -n odcp mtls-server-secret --cert ./server-cert-chain.pem --key ./server-key.pem
- 创建保存客户端CA证书的Secret
kubectl create secret generic -n odcp mtls-client-ca-secret --from-file=ca.crt=./client-ca-cert.pem
- 创建HTTPS双向认证的Ingress
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: mtls-test-ingress
namespace: odcp
annotations:
# Enable client certificate authentication
nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
# Create the secret containing the trusted ca certificates
nginx.ingress.kubernetes.io/auth-tls-secret: "odcp/mtls-client-ca-secret"
# Specify the verification depth in the client certificates chain
nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1"
# Specify an error page to be redirected to verification errors
#nginx.ingress.kubernetes.io/auth-tls-error-page: "https://www.ityoudao.cn/"
# Specify if certificates are passed to upstream server
nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "false"
spec:
ingressClassName: nginx
rules:
- host: mtls.dev.ityoudao.cn
http:
paths:
- backend:
service:
name: nginx-test
port:
number: 80
path: /
pathType: ImplementationSpecific
tls:
- hosts:
- mtls.dev.ityoudao.cn
secretName: mtls-server-secret
EOF
- Ingress的客户端证书的相关配置可以参考官网User Guide - Annotations - Client Certificate Authentication
使用curl命令测试HTTPS双向认证
- 不带客户端证书和密钥,返回
400 Bad Request
错误
# curl -I https://mtls.dev.ityoudao.cn --cacert server-ca-cert.pem
HTTP/1.1 400 Bad Request
Date: Fri, 05 Jan 2024 08:58:47 GMT
Content-Type: text/html
Content-Length: 230
Connection: close
Cache-Control: no-cache
- 带客户端证书和密钥,返回
200 OK
正常
# curl -I https://mtls.dev.ityoudao.cn --cacert server-ca-cert.pem --cert ./client-cert.pem --key ./client-key.pem
HTTP/1.1 200 OK
Date: Fri, 05 Jan 2024 08:59:12 GMT
Content-Type: text/html
Content-Length: 2161
Connection: keep-alive
Last-Modified: Tue, 19 Sep 2023 06:28:05 GMT
ETag: "65093f75-871"
Accept-Ranges: bytes
Cache-Control: no-cache
使用带--cacert server-ca-cert.pem
参数的curl命令,没有报服务端证书错误,说明服务端证书和密钥配置正确!不带客户端证书和密钥、带客户端证书和密钥的测试结果均符合预期,说明客户端证书和密钥配置正确!
遇到的问题和解决办法
-
ingress-nginx-controller
报x509: certificate is not valid for any names, but wanted to match mtls.dev.ityoudao.cn
错误
# kubectl logs -n ingress-nginx ingress-nginx-controller-default-7d94c6957d-l6wwr --tail=100 | grep mtls
W0105 08:11:38.291923 7 controller.go:1347] Unexpected error validating SSL certificate "odcp/mtls-server-secret" for server "mtls.dev.ityoudao.cn": x509: certificate is not valid for any names, but wanted to match mtls.dev.ityoudao.cn
W0105 08:11:38.291937 7 controller.go:1353] SSL certificate "odcp/mtls-server-secret" does not contain a Common Name or Subject Alternative Name for server "mtls.dev.ityoudao.cn": x509: certificate is not valid for any names, but wanted to match mtls.dev.ityoudao.cn
- 后来发现是因为,误用服务端CA证书和私钥文件创建了
mtls-server-secret
,因此CN和SAN校验都不通过,导致ingress-nginx-controller
报x509: certificate is not valid for any names, but wanted to match mtls.dev.ityoudao.cn
错误,更换为正确的服务端证书和密钥即可解决问题!
kubectl delete secret -n odcp mtls-server-secret
kubectl create secret tls -n odcp mtls-server-secret --cert ./server-ca-cert.pem --key ./server-ca-key.pem
-
curl
命令报curl: (60) Peer's certificate has an invalid signature.
错误
# curl https://mtls.dev.ityoudao.cn --cacert server-ca-cert.pem
curl: (60) Peer's certificate has an invalid signature.
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
- 对端证书签名无效,也就是说服务端证书无效,这里的原因是证书链不完整,使用完整的证书链文件即可解决问题:
# 错误的做法,证书链不完整
kubectl create secret tls -n odcp mtls-server-secret --cert ./server-cert.pem --key ./server-key.pem
# 正确的做法,完整的证书链
cat server-cert.pem server-ca-cert.pem > server-cert-chain.pem
kubectl create secret tls -n odcp mtls-server-secret --cert ./server-cert-chain.pem --key ./server-key.pem
-
ingress-nginx-controller
报x509: certificate relies on legacy Common Name field, use SANs instead
错误
I0105 08:43:27.668845 7 store.go:584] "Secret was added and it is used in ingress annotations. Parsing" secret="odcp/mtls-server-secret"
I0105 08:43:27.669756 7 backend_ssl.go:65] "Adding secret to local store" name="odcp/mtls-server-secret"
W0105 08:43:27.673743 7 controller.go:1347] Unexpected error validating SSL certificate "odcp/mtls-server-secret" for server "mtls.dev.ityoudao.cn": x509: certificate relies on legacy Common Name field, use SANs instead
- 在SUSE官网看到一句说明:“While in the past it was sufficient to have the server name as “Common Name(CN)” within the “Subject” of a certificate and additionally within “Subject Alternative Name”, newer implementations may just check for “Subject Alternative Name”.”,翻译为中文意思是:过去,仅在证书的“主题”中设置“通用名称(CN)”,然后在“主题备用名称”中进行附加就足够了,但较新的实现可能只会检查“主题备用名称”。
- 这里是因为服务端证书只使用了Subject,没有使用X509v3 Subject Alternative Name,而
ingress-nginx-controller
需要检查主题备用名称SAN,导致报x509: certificate relies on legacy Common Name field, use SANs instead
错误,在服务端证书使用SAN即可解决问题:
# 错误的做法,没有使用X509v3 Subject Alternative Name
openssl req -new -newkey rsa:4096 -keyout server-key.pem -out server.csr -nodes -subj '/CN=mtls.dev.ityoudao.cn'
openssl x509 -req -sha256 -days 365 -in server.csr -CA server-ca-cert.pem -CAkey server-ca-key.pem -set_serial 01 -out server-cert.pem
# 正确的做法,使用了X509v3 Subject Alternative Name
echo subjectAltName = DNS:mtls.dev.ityoudao.cn > server-extfile.cnf
openssl req -new -newkey rsa:4096 -keyout server-key.pem -out server.csr -nodes -subj '/CN=mtls.dev.ityoudao.cn'
openssl x509 -req -sha256 -days 365 -in server.csr -CA server-ca-cert.pem -CAkey server-ca-key.pem -extfile server-extfile.cnf -set_serial 01 -out server-cert.pem
配置客户端证书的证书吊销列表
- 假设需要被吊销的证书文件为
client-cert.pem
,使用openssl ca -revoke
命令吊销证书
touch /etc/pki/CA/index.txt
openssl ca -revoke client-cert.pem -cert client-ca-cert.pem -keyfile client-ca-key.pem
- 如果
openssl ca -revoke
命令unable to load CA private key
错误:
# openssl ca -revoke client-cert.pem
Using configuration from /etc/pki/tls/openssl.cnf
Error opening CA private key /etc/pki/CA/private/cakey.pem
140215560439696:error:02001002:system library:fopen:No such file or directory:bss_file.c:402:fopen('/etc/pki/CA/private/cakey.pem','r')
140215560439696:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404:
unable to load CA private key
# 解决办法:将客户端CA证书和密钥文件拷贝到`/etc/pki/CA/`目录下,或者直接用`-cert`和`-keyfile`参数指定客户端CA证书和密钥文件
# openssl ca -revoke client-cert.pem -cert client-ca-cert.pem -keyfile client-ca-key.pem
- 如果
openssl ca -revoke
命令unable to open '/etc/pki/CA/index.txt'
错误:
# openssl ca -revoke client-cert.pem -cert client-ca-cert.pem -keyfile client-ca-key.pem
Using configuration from /etc/pki/tls/openssl.cnf
/etc/pki/CA/index.txt: No such file or directory
unable to open '/etc/pki/CA/index.txt'
139702194755472:error:02001002:system library:fopen:No such file or directory:bss_file.c:402:fopen('/etc/pki/CA/index.txt','r')
139702194755472:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404:
# 解决办法:创建一个空的`/etc/pki/CA/index.txt`文件即可
# touch /etc/pki/CA/index.txt
# openssl ca -revoke client-cert.pem -cert client-ca-cert.pem -keyfile client-ca-key.pem
Using configuration from /etc/pki/tls/openssl.cnf
Adding Entry with serial number 01 to DB for /CN=Client001
Revoking Certificate 01.
Data Base Updated
# cat /etc/pki/CA/index.txt
R 250104075252Z 240105091344Z 01 unknown /CN=Client001
# cat /etc/pki/CA/index.txt.attr
unique_subject = yes
# cat /etc/pki/CA/index.txt.old
- 使用
openssl ca -gencrl
命令生成证书吊销列表文件
echo 01 > /etc/pki/CA/crlnumber
openssl ca -gencrl -out client-ca.crl -cert client-ca-cert.pem -keyfile client-ca-key.pem
- 如果
openssl ca -gencrl
命令error while loading CRL number
错误:
# openssl ca -gencrl -out client-ca.crl -cert client-ca-cert.pem -keyfile client-ca-key.pem
Using configuration from /etc/pki/tls/openssl.cnf
/etc/pki/CA/crlnumber: No such file or directory
error while loading CRL number
140466499000208:error:02001002:system library:fopen:No such file or directory:bss_file.c:402:fopen('/etc/pki/CA/crlnumber','r')
140466499000208:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404:
# 解决办法:创建`/etc/pki/CA/crlnumber`文件即可
# echo 01 > /etc/pki/CA/crlnumber
# openssl ca -gencrl -out client-ca.crl -cert client-ca-cert.pem -keyfile client-ca-key.pem
Using configuration from /etc/pki/tls/openssl.cnf
# ll client-ca.crl
-rw-r--r--. 1 root root 954 1月 5 17:27 client-ca.crl
# cat /etc/pki/CA/crlnumber
02
# cat /etc/pki/CA/crlnumber.old
01
- 使用
openssl crl
命令查看证书吊销列表文件
openssl crl -text -noout -in client-ca.crl
# 结果如下:
# openssl crl -text -noout -in client-ca.crl
Certificate Revocation List (CRL):
Version 2 (0x1)
Signature Algorithm: sha256WithRSAEncryption
Issuer: /CN=Aispeech Client Cert Authority
Last Update: Jan 5 09:27:46 2024 GMT
Next Update: Feb 4 09:27:46 2024 GMT
CRL extensions:
X509v3 CRL Number:
1
Revoked Certificates:
Serial Number: 01
Revocation Date: Jan 5 09:13:44 2024 GMT
Signature Algorithm: sha256WithRSAEncryption
86:b3:a3:1e:31:75:bb:0f:f3:32:af:2b:57:e6:e2:ca:fb:94:
9b:c1:0b:15:39:af:9e:9a:d6:25:9c:3a:69:e0:29:dd:d6:b6:
55:79:2f:24:c3:7f:40:38:b6:77:6e:08:03:bb:23:0c:9c:5f:
db:0d:18:4d:c6:d6:01:74:6f:d5:ba:bb:bc:e3:95:96:91:eb:
e1:2b:27:3c:62:33:a8:eb:ed:4a:26:5a:a3:27:cd:f8:02:73:
1c:64:c0:60:95:02:ea:b3:63:ba:d4:86:65:83:c7:cf:fb:7e:
9a:71:7e:4f:9d:b2:50:0b:eb:fb:c5:f4:c2:41:8f:57:50:27:
88:a2:19:02:e5:84:6a:f1:30:ea:71:db:23:8a:04:f0:11:75:
c5:8c:6d:04:36:99:6d:42:c6:c0:8d:00:0e:3d:b7:dd:0a:28:
ed:e2:5e:7c:f4:56:41:35:d3:73:3c:2f:96:86:92:1e:d7:5a:
d2:e5:dc:7c:a1:7b:a9:10:ef:5a:7e:30:1d:86:6f:97:c5:58:
b8:3b:ee:75:cf:35:ad:7d:fb:47:ac:a8:7a:81:65:56:4d:2e:
15:60:c0:c9:39:ca:b0:cb:21:a1:48:33:11:51:ee:3e:d5:c2:
9b:ba:c5:fc:ac:04:99:87:2c:2b:56:fe:06:62:76:f8:31:df:
dd:b8:a1:f4:bc:d6:87:18:79:00:2d:a6:15:cd:c3:88:80:48:
2a:59:70:8b:06:1d:08:11:39:b5:35:7d:58:c3:3b:27:b5:89:
5f:18:fb:5e:4b:48:4c:04:6a:20:08:96:ad:3d:65:23:da:ad:
0c:74:d6:fc:2a:79:8c:41:42:3b:bd:c5:d0:cb:28:3c:f5:68:
9c:e1:d4:7d:28:c1:3b:20:36:90:d0:71:97:3d:54:78:0f:49:
6e:a2:f4:56:7b:4e:64:03:a6:18:6e:8d:d9:a8:28:96:1f:94:
20:35:1d:6d:2c:f4:56:65:4c:0a:07:2c:c8:5f:44:6f:6c:53:
e4:13:d8:56:0c:07:79:7a:0d:cb:a6:e1:de:7e:f0:12:aa:9d:
f9:5b:59:6b:61:18:21:96:96:75:4c:3f:b0:ab:73:27:d7:41:
ac:50:ea:99:56:13:0f:b9:df:4c:6b:0a:e4:5e:df:56:76:9e:
3a:ce:ab:41:1f:2b:96:bc:9f:77:96:0b:c6:fa:a9:7c:ae:94:
a9:c7:f2:68:a0:b2:07:82:1e:74:48:b7:68:f7:da:b0:0d:c4:
5c:08:8a:0e:86:14:70:89:6f:25:9b:63:1d:9b:b4:87:28:0e:
15:23:0f:05:51:c9:9e:b7:57:be:06:b8:74:9d:4f:79:d3:49:
91:16:a3:59:d6:54:64:9a
- 创建保存客户端CA证书和证书吊销列表的Secret
kubectl delete secret -n odcp mtls-client-ca-secret
kubectl create secret generic -n odcp mtls-client-ca-secret --from-file=ca.crt=./client-ca-cert.pem --from-file=ca.crl=./client-ca.crl
- 再次使用curl命令测试客户端证书
# curl -I https://mtls.dev.ityoudao.cn --cacert server-ca-cert.pem --cert ./client-cert.pem --key ./client-key.pem
HTTP/1.1 400 Bad Request
Date: Fri, 05 Jan 2024 09:30:37 GMT
Content-Type: text/html
Content-Length: 208
Connection: close
Cache-Control: no-cache
被吊销的客户端证书和密钥,访问服务报400 Bad Request
错误,结果符合预期,说明客户端证书吊销成功!文章来源:https://www.toymoban.com/news/detail-782430.html
- 更新证书吊销列表
- 如果证书吊销列表发生变更,可以使用如下命令更新证书吊销列表
# 更新证书吊销列表client-ca.crl
kubectl create secret generic -n odcp mtls-client-ca-secret --from-file=ca.crt=./client-ca-cert.pem --from-file=ca.crl=./client-ca.crl --dry-run="client" -o yaml | kubectl apply -f -
- Secret更新后,
ingress-nginx-controller
会自动reload,reload可能会影响长连接服务:
# ingress-nginx-controller的reload日志
I0105 09:49:31.929155 7 store.go:615] "secret was updated and it is used in ingress annotations. Parsing" secret="odcp/mtls-client-ca-secret"
I0105 09:49:31.929841 7 backend_ssl.go:57] "Updating secret in local store" name="odcp/mtls-client-ca-secret"
I0105 09:49:31.946486 7 controller.go:166] "Configuration changes detected, backend reload required"
I0105 09:49:33.286314 7 controller.go:183] "Backend successfully reloaded"
I0105 09:49:33.288423 7 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-default-7d94c6957d-l6wwr", UID:"afac1300-97ad-4304-8ff0-95357f4e3144", APIVersion:"v1", ResourceVersion:"783734248", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
在本文中,我们深入探讨了在Kubernetes环境中配置ingress-nginx-controller的HTTPS双向认证的过程。文章详细介绍了生成服务端和客户端证书的步骤,以及配置Ingress实现HTTPS双向认证的过程。同时,我们解决了在部署过程中遇到的一系列问题,包括证书配置错误、证书链不完整、以及ingress-nginx-controller报错等。最后,我们还介绍了如何配置客户端证书的证书吊销列表,以满足客户对安全性的额外需求。通过本文,希望大家可以获得在Kubernetes环境中进行HTTPS双向认证的全面指南,帮助应对实际部署中可能遇到的各种挑战。文章来源地址https://www.toymoban.com/news/detail-782430.html
到了这里,关于ingress-nginx-controller的HTTPS双向认证配置的文章就介绍完了。如果您还想了解更多内容,请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章,希望大家以后多多支持TOY模板网!