1+X 云计算运维与开发（中级）案例实战——搭建harbor私有仓库并实现主从同步-Toy模板网

这篇具有很好参考价值的文章主要介绍了1+X 云计算运维与开发（中级）案例实战——搭建harbor私有仓库并实现主从同步。希望对大家有所帮助。如果存在错误或未考虑完全的地方，请大家不吝赐教，您也可以点击"举报违法"按钮提交疑问。

前言

学而不思则罔，思而不学则殆。

IP	主机名	节点
192.168.200.132/24	master	Harbor 仓库节点
192.168.200.133/24	slave	Harbor 备份节点

说明：本次实验使用的镜像为k8sallinone，该镜像网络使用net模式，可上外网，且该镜像已安装docker引擎，若使用其他镜像请自行安装docker引擎
准备文件：

Docker.tar.gz ：内含harbor组件的容器镜像

docker-compose ：可以在github下载对应版本，这里使用的版本为 1.25.0-rc2
这里是下载命令： curl -L https://github.com/docker/compose/releases/download/1.25.0-rc2/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
这里是github的地址： https://github.com/docker/compose

harbor-offline-installer-v1.5.3.tgz ：harbor的安装包

思路

1.配置镜像加速器
2.生成CA证书（这里的原理强烈建议了解一下），并且分发证书
3.修改harbor的配置文件
4.安装harbor
5.Harbor备份节点也安装一波harbor
6.略作调试，使两台机子形成主从

个人认为CA证书那里是最难的点，但是去了解之后，感觉也不是很难

实操

1.（个人习惯）修改主机名，添加主机映射

修改主机名命令示例如下：

[root@master ~]# hostnamectl set-hostname slave
[root@master ~]# bash
bash
[root@slave ~]#

在Harbor仓库节点添加主机映射，再用scp命令发给Harbor备份节点：

没接触过scp命令的点这里：https://www.runoob.com/linux/linux-comm-scp.html

[root@master ~]# cat >> /etc/hosts << eof
> 192.168.200.132 master
> 192.168.200.133 slave
> eof
[root@master ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.200.132 master
192.168.200.133 slave
[root@master ~]# scp /etc/hosts root@slave:/etc/
The authenticity of host 'slave (192.168.200.133)' can't be established.
ECDSA key fingerprint is SHA256:G4McP9UHWCN8ERimLg4Jlw7fHdtmG2rU0XeS4XJqOFc.
ECDSA key fingerprint is MD5:64:ae:f8:6b:47:76:31:83:f6:e9:03:9b:dd:df:5a:4a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave,192.168.200.133' (ECDSA) to the list of known hosts.
root@slave's password:
hosts                                                                        100%  203   347.5KB/s   00:00

检查一下：

[root@slave ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.200.132 master
192.168.200.133 slave

2. 配置镜像加速器

注意：从这一步开始，如果想要节省时间，可以两台机子一起进行操作，我会在需要不同操作的地方给出提示

修改 /etc/docker/daemon.json 文件：

[root@master ~]# vi /etc/docker/daemon.json
{
  "registry-mirrors": ["https://5twf62k1.mirror.aliyuncs.com"]
}

验证：

[root@master ~]# systemctl restart docker
[root@master ~]# docker pull mysql
Using default tag: latest
latest: Pulling from library/mysql
72a69066d2fe: Pull complete
93619dbc5b36: Pull complete
99da31dd6142: Pull complete
626033c43d70: Pull complete
37d5d7efb64e: Pull complete
ac563158d721: Pull complete
d2ba16033dad: Pull complete
688ba7d5c01a: Pull complete
00e060b6d11d: Pull complete
1c04857f594f: Pull complete
4d7cfa90e6ea: Pull complete
e0431212d27d: Pull complete
Digest: sha256:e9027fe4d91c0153429607251656806cc784e914937271037f7738bd5b8e7709
Status: Downloaded newer image for mysql:latest

将准备好的Docker.tar.gz包解压，进入，然后使用shell脚本（不知道是不是，以后学了shell脚本再来修改）将镜像导入：

[root@master images]# tar -zxvf Docker.tar.gz
[root@master ~]# ll
total 4136024
drwxr-xr-x  2 root root        105 Nov  3  2019 compose
drwxr-xr-x. 4 root root         32 Nov  3  2019 Docker
-rw-r--r--  1 root root   16912904 Apr  2 13:16 docker-compose
-rw-r--r--  1 root root 3314069073 Apr  2 13:16 Docker.tar.gz
-rw-r--r--  1 root root  904278926 Sep 12  2018 harbor-offline-installer-v1.5.3.tgz
drwxr-xr-x. 2 root root       4096 Nov  3  2019 images
-rwxr-xr-x  1 root root       1015 Nov  3  2019 image.sh
-rwxr-xr-x  1 root root        985 May  8  2020 install.sh
drwxrwxrwx. 4 root root         32 Dec  5  2019 Kubernetes
-rwxrwxrwx. 1 root root        987 Nov  2  2019 kubernetes_base.sh
drwxrwxrwx. 2 root root       4096 May 11  2020 yaml
[root@master ~]# cd images
[root@master images]# for i in `ll | awk '{print$9}'`;do docker load -i $i;done
1da8e4c8d307: Loading layer  1.437MB/1.437MB
Loaded image: busybox:latest
9e607bb861a7: Loading layer  227.4MB/227.4MB
Loaded image: centos:latest
dba693fc2701: Loading layer  133.4MB/133.4MB
...
[root@master images]#

shell命令解析：
1.for i in ：对编程的人来说不陌生，循环语句；
2.两个反引号 ``:对循环的范围；
3.awk ：是一种处理文本文件的语言，是一个强大的文本分析工具；
awk命令：https://www.runoob.com/linux/linux-comm-awk.html
4.‘{print$9}’ :与 awk 搭配，输出第9项内容，这里输出的都是镜像的名字，这些名字要与后面的 docker load -i 进行搭配；
5.两个；：格式符号，前面的awk 返回的值会传递到这里面的命令
6. do、done ：格式
7. docker load -i ：导入使用 docker save 命令导出的镜像；
–input , -i : 指定导入的文件，代替 STDIN。
docker load -i：https://www.runoob.com/docker/docker-load-command.html

3. 生成CA证书,并分发证书

注意：这里开始，两个机子有略微的差别，请分开来做

Openssl 是目前最流行的 SSL 密码库工具，提供了一个通用，功能完备的工具套件，用以支持 SSL/TLS 协议的实现。

Harbor仓库节点：

[root@master images]# mkdir -p /data/ssl
[root@master images]# cd /data/ssl
[root@master ssl]# which openssl
/usr/bin/openssl
[root@master ssl]# openssl req -newkey rsa:4096 -nodes -sha256 -x509 -days 2.235 -keyout ca.key -out ca.crt
Generating a 4096 bit RSA private key
................................................................................................................................................................++
.....................................................................................................................................................................................................................++
writing new private key to 'ca.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:Chongqing
Locality Name (eg, city) [Default City]:Chongqing
Organization Name (eg, company) [Default Company Ltd]:yidaoyun
Organizational Unit Name (eg, section) []:yidaoyun
Common Name (eg, your name or your server's hostname) []:www.yidaoyun.com  <---记住这个地方
Email Address []:

[root@master ssl]# ll
total 8
-rw-r--r-- 1 root root 2013 Apr  3 08:15 ca.crt
-rw-r--r-- 1 root root 3272 Apr  3 08:15 ca.key

Harbor备份节点：

不一样的地方就是 www.yidaoyun.com 改为 www2.yidaoyun.com ,其他一致

Common Name (eg, your name or your server's hostname) []:www2.yidaoyun.com

生成证书签名请求
Harbor仓库节点：

[root@master ssl]# openssl req -newkey rsa:4096 -nodes -sha256 -keyout www.yidaoyun.com.key -out www.yidaoyun.com.csr
Generating a 4096 bit RSA private key
...++
..........................................++
writing new private key to 'www.yidaoyun.com.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:Chongqing
Locality Name (eg, city) [Default City]:Chongqing
Organization Name (eg, company) [Default Company Ltd]:yidaoyun
Organizational Unit Name (eg, section) []:yidaoyun
Common Name (eg, your name or your server's hostname) []:www.yidaoyun.com
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
[root@master ssl]# ll
total 16
-rw-r--r-- 1 root root 2013 Apr  3 08:15 ca.crt
-rw-r--r-- 1 root root 3272 Apr  3 08:15 ca.key
-rw-r--r-- 1 root root 1720 Apr  3 08:20 www.yidaoyun.com.csr
-rw-r--r-- 1 root root 3272 Apr  3 08:20 www.yidaoyun.com.key

这里和Harbor备份节点不同的地方就是：Harbor仓库节点是www.yidaoyun.com ; 而Harbor备份节点是：www2.yidaoyun.com
另外要注意的是：由于这里没有参数：-x509 ,所以这里的不是 .crt 了，而是 .csr，这个很重要

这里给上Harbor备份节点的命令：

[root@slave ssl]# openssl req -newkey rsa:4096 -nodes -sha256 -keyout www.2yidaoyun.com.key -out www2.yidaoyun.com.csr
Generating a 4096 bit RSA private key
.................++
.................................................................++
writing new private key to 'www.2yidaoyun.com.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:Chongqing
Locality Name (eg, city) [Default City]:Chongqing
Organization Name (eg, company) [Default Company Ltd]:yidaoyun
Organizational Unit Name (eg, section) []:yidaoyun
Common Name (eg, your name or your server's hostname) []:www.2yidaoyun.com
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
[root@slave ssl]# ll
total 16
-rw-r--r-- 1 root root 2057 Apr  3 08:15 ca.crt
-rw-r--r-- 1 root root 3272 Apr  3 08:15 ca.key
-rw-r--r-- 1 root root 1724 Apr  3 08:20 www2.yidaoyun.com.csr
-rw-r--r-- 1 root root 3272 Apr  3 08:20 www.2yidaoyun.com.key

这里细心的朋友可能看到我的 www2.yidaoyun.com.key 被我误操作弄成 www.2yidaoyun.com.key 了
解决方法就是把 www 开头的文件全删除，重新生成

示例代码如下：

[root@slave ssl]# rm -rf www*
[root@slave ssl]# openssl req -newkey rsa:4096 -nodes -sha256 -keyout www2.yidaoyun.com.key -out www2.yidaoyun.com.csr
Generating a 4096 bit RSA private key
.............................................................++
......++
writing new private key to 'www2.yidaoyun.com.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:Chongqing
Locality Name (eg, city) [Default City]:Chongqing
Organization Name (eg, company) [Default Company Ltd]:yidaoyun
Organizational Unit Name (eg, section) []:yidaoyun
Common Name (eg, your name or your server's hostname) []:www2.yidaoyun.com
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
[root@slave ssl]# ll
total 16
-rw-r--r-- 1 root root 2057 Apr  3 08:15 ca.crt
-rw-r--r-- 1 root root 3272 Apr  3 08:15 ca.key
-rw-r--r-- 1 root root 1720 Apr  3 08:27 www2.yidaoyun.com.csr
-rw-r--r-- 1 root root 3272 Apr  3 08:27 www2.yidaoyun.com.key

生成注册表主机的证书
Harbor仓库节点：

[root@master ssl]# openssl x509 -req -days 2.235 -in www.yidaoyun.com.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out www.yidaoyun.com.crt
Signature ok
subject=/C=CN/ST=Chongqing/L=Chongqing/O=yidaoyun/OU=yidaoyun/CN=ww:www.yidaoyun.com
Getting CA Private Key

[root@master ssl]# ll
total 24
-rw-r--r-- 1 root root 2013 Apr  3 08:15 ca.crt
-rw-r--r-- 1 root root 3272 Apr  3 08:15 ca.key
-rw-r--r-- 1 root root   17 Apr  3 08:30 ca.srl
-rw-r--r-- 1 root root 1720 Apr  3 08:20 www.yidaoyun.com.csr
-rw-r--r-- 1 root root 3272 Apr  3 08:20 www.yidaoyun.com.key
-rw-r--r-- 1 root root 1919 Apr  3 08:30 www.yidaoyun.com.crt

这里和Harbor备份节点不同的地方就是：Harbor仓库节点是www.yidaoyun.com ; 而Harbor备份节点是：www2.yidaoyun.com
(没错，就是复制粘贴）

这里给上Harbor备份数据库的操作：

[root@slave ssl]# openssl x509 -req -days 2.235 -in www2.yidaoyun.com.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out www2.yidaoyun.com.crt
Signature ok
subject=/C=CN/ST=Chongqing/L=Chongqing/O=yidaoyun/OU=yidaoyun/CN=www2.yidaoyun.com
Getting CA Private Key
[root@slave ssl]# ll
total 24
-rw-r--r-- 1 root root 2057 Apr  3 08:15 ca.crt
-rw-r--r-- 1 root root 3272 Apr  3 08:15 ca.key
-rw-r--r-- 1 root root   17 Apr  3 08:41 ca.srl
-rw-r--r-- 1 root root 1939 Apr  3 08:41 www2.yidaoyun.com.crt
-rw-r--r-- 1 root root 1720 Apr  3 08:27 www2.yidaoyun.com.csr
-rw-r--r-- 1 root root 3272 Apr  3 08:27 www2.yidaoyun.com.key

如果不想死记硬背openssl命令，建议看这篇文章：https://www.cnblogs.com/f-ck-need-u/p/7113610.html

该图来自上面的文章： 1+X 云计算运维与开发（中级）案例实战——搭建harbor私有仓库并实现主从同步

分发证书
(接下来两个机子的不同只有 www 和 www2 的区别）
Harbor仓库节点：

[root@master ssl]# cp -rvf www.yidaoyun.com.crt /etc/pki/ca-trust/source/anchors/
‘www.yidaoyun.com.crt’ -> ‘/etc/pki/ca-trust/source/anchors/www.yidaoyun.com.crt’

Harbor备份节点：

[root@slave ssl]# cp -rvf www2.yidaoyun.com.crt /etc/pki/ca-trust/source/anchors/
‘www2.yidaoyun.com.crt’ -> ‘/etc/pki/ca-trust/source/anchors/www2.yidaoyun.com.crt’

4. 安装docker-compose

两台操作一致

[root@master ~]# mv docker-compose /usr/local/bin/
[root@master ~]# chmod +x /usr/local/bin/docker-compose
[root@master ~]# docker-compose -v
docker-compose version 1.25.0-rc2, build 661ac20e

这里是下载命令：

curl -L https://github.com/docker/compose/releases/download/1.25.0-rc2/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose

这里是github的地址： https://github.com/docker/compose

5. 配置harbor

两台操作一致

解压下载过来的 harbor安装包 到 /opt :

[root@master ~]# tar -zxvf harbor-offline-installer-v1.5.3.tgz -C /opt/
[root@master ~]# cd /opt/harbor/
[root@master harbor]# ll
total 895708
drwxr-xr-x 3 root root        22 Apr  3 08:54 common
-rw-r--r-- 1 root root      1185 Sep 12  2018 docker-compose.clair.yml
-rw-r--r-- 1 root root      1725 Sep 12  2018 docker-compose.notary.yml
-rw-r--r-- 1 root root      3596 Sep 12  2018 docker-compose.yml
drwxr-xr-x 3 root root       150 Sep 12  2018 ha
-rw-r--r-- 1 root root      6956 Sep 12  2018 harbor.cfg
-rw-r--r-- 1 root root 915878468 Sep 12  2018 harbor.v1.5.3.tar.gz
-rwxr-xr-x 1 root root      5773 Sep 12  2018 install.sh
-rw-r--r-- 1 root root     10764 Sep 12  2018 LICENSE
-rw-r--r-- 1 root root       482 Sep 12  2018 NOTICE
-rw-r--r-- 1 root root   1247461 Sep 12  2018 open_source_license
-rwxr-xr-x 1 root root     27840 Sep 12  2018 prepare
[root@master harbor]# vi harbor.cfg

修改时两台不同的地方就是 www 和 www2 ；还有 ip 地址

将内容改为：

Harbor仓库节点：

hostname = 192.168.200.132

ui_url_protocol = https

ssl_cert = /data/ssl/www.yidaoyun.com.crt
ssl_cert_key = /data/ssl/www.yidaoyun.com.key

harbor_admin_password = 000000

Harbor备份节点：

hostname = 192.168.200.132

ui_url_protocol = https

ssl_cert = /data/ssl/www2.yidaoyun.com.crt
ssl_cert_key = /data/ssl/www2.yidaoyun.com.key

harbor_admin_password = 000000

安装harbor：

[root@master harbor]# ./prepare
Generated and saved secret to file: /data/secretkey
Generated configuration file: ./common/config/nginx/nginx.conf
Generated configuration file: ./common/config/adminserver/env
Generated configuration file: ./common/config/ui/env
Generated configuration file: ./common/config/registry/config.yml
Generated configuration file: ./common/config/db/env
Generated configuration file: ./common/config/jobservice/env
Generated configuration file: ./common/config/jobservice/config.yml
Generated configuration file: ./common/config/log/logrotate.conf
Generated configuration file: ./common/config/jobservice/config.yml
Generated configuration file: ./common/config/ui/app.conf
Generated certificate, key file: ./common/config/ui/private_key.pem, cert file: ./common/config/registry/root.crt
The configuration files are ready, please use docker-compose to start the service.

[root@master harbor]# ./install.sh --with-notary --with-clair

[Step 0]: checking installation environment ...

Note: docker version: 18.09.6

Note: docker-compose version: 1.25.0

[Step 1]: loading Harbor images ...
Loaded image: vmware/harbor-adminserver:v1.5.3
Loaded image: vmware/harbor-db:v1.5.3
Loaded image: vmware/harbor-jobservice:v1.5.3
Loaded image: vmware/redis-photon:v1.5.3
Loaded image: photon:1.0
Loaded image: vmware/notary-signer-photon:v0.5.1-v1.5.3
Loaded image: vmware/mariadb-photon:v1.5.3
Loaded image: vmware/postgresql-photon:v1.5.3
Loaded image: vmware/harbor-ui:v1.5.3
Loaded image: vmware/harbor-log:v1.5.3
Loaded image: vmware/nginx-photon:v1.5.3
Loaded image: vmware/registry-photon:v2.6.2-v1.5.3
Loaded image: vmware/notary-server-photon:v0.5.1-v1.5.3
Loaded image: vmware/harbor-migrator:v1.5.0
Loaded image: vmware/clair-photon:v2.0.5-v1.5.3


[Step 2]: preparing environment ...

这边我们先打开浏览器，登录： https://192.168.200.132

1+X 云计算运维与开发（中级）案例实战——搭建harbor私有仓库并实现主从同步
继续访问即可

用户名admin ，密码 000000

选择“配置管理”菜单命令，项目创建选择“仅管理员”，取消勾选“允许自注册”，复选框，然后单击“保存”按钮
1+X 云计算运维与开发（中级）案例实战——搭建harbor私有仓库并实现主从同步
然后我们回到命令行界面

接下来的操作只对Harbor仓库节点操作

修改 /etc/docker/daemon.json 文件

{
  "registry-mirrors": ["https://5twf62k1.mirror.aliyuncs.com"], <-----这里的逗号你注意到了吗，不加报错
  "insecure-registries":["192.168.200.132"]
}

重新启动 Harbor 私有镜像仓库

让 Harbor 修改过的配置立刻生效

[root@master harbor]# ./prepare

清理所有 Harbor 容器进程,在后台启动这些进程

[root@master harbor]# docker-compose down

[root@master harbor]# docker-compose up -d

上传镜像

[root@master harbor]# docker pull nginx   ##拉取镜像
Using default tag: latest
latest: Pulling from library/nginx
a2abf6c4d29d: Pull complete
a9edb18cadd1: Pull complete
589b7251471a: Pull complete
186b1aaa4aa6: Pull complete
b4df32aa5a72: Pull complete
a0bcbecc962e: Pull complete
Digest: sha256:0d17b565c37bcbd895e9d92315a05c1c3c9a29f762b011a10c54a66cd53c9b31
Status: Downloaded newer image for nginx:latest

[root@master harbor]# docker tag nginx:latest 192.168.200.132/library/nginx:latest   ##给镜像打标签

登录验证 Harbor 仓库

[root@master harbor]# docker login https://192.168.200.132
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

上传镜像到 Harbor 仓库

[root@master harbor]# docker push 192.168.200.132/library/nginx:latest
The push refers to repository [192.168.200.132/library/nginx]
d874fd2bc83b: Pushed
32ce5f6a5106: Pushed
f1db227348d0: Pushed
b8d6e692a25e: Pushed
e379e8aedd4d: Pushed
2edcec3590a4: Pushed
latest: digest: sha256:ee89b00528ff4f02f2405e4ee221743ebc3f8e8dd0bfd5c4c20a2fa2aaa7ede3 size: 1570

重新启用漏洞扫描

[root@master harbor]# ./install.sh --with-notary --with-clair

6. 主从部分

这时候我们的两台机子的状态时都已经安装好harbor；
Harbor仓库节点多了一点操作就是设置了私有仓库，修改了一点配置，上传了镜像

1.Harbor仓库节点从 Harbor备份节点上拷贝证书

[root@master harbor]# scp slave:/data/ssl/www2.yidaoyun.com.crt /etc/pki/ca-trust/source/anchors/
root@slave's password:
ww 100% 1939     2.9MB/s   00:00
[root@master harbor]#
[root@master harbor]# update-ca-trust enable
[root@master harbor]# update-ca-trust extract
[root@master harbor]# systemctl restart docker

2.重启harbor

[root@master harbor]# docker-compose down
[root@master harbor]# ./prepare
[root@master harbor]# ./install.sh --with-notary --with-clair

6.1 图形化界面的操作

1+X 云计算运维与开发（中级）案例实战——搭建harbor私有仓库并实现主从同步
填写信息，测试主库与从库的连接性

登录主库，选择“系统管理”→“复制管理”→“新建规则”菜单命令，创建复制规则

登录主库查看镜像列表
1+X 云计算运维与开发（中级）案例实战——搭建harbor私有仓库并实现主从同步
登录从库查看镜像列表

可以发现主库的所有镜像已成功同步到了从库。至此，Harbor 仓库主从复制已经构建
完毕