-
docker-compose常用命令
详见
https://blog.csdn.net/qq_42267173/article/details/124687804
https://blog.csdn.net/m0_72838865/article/details/126421685
https://www.cnblogs.com/cw2blog/p/16689584.html 容器启停相关的
# 前台启动
docker-compose up
# 后台启动
docker-compose up -d
logs
# docker-compose logs查看服务容器的输出日志。默认情况下,docker-compose将对不同的服务输出使用不同的颜色来区分。可以通过--no-color来关闭颜色。
# 输出日志,不同的服务输出使用不同的颜色来区分
docker-compose logs
# 跟踪日志输出
docker-compose logs -f
# 关闭颜色
docker-compose logs --no-color
run
docker-compose run在指定服务容器上执行一个命令。
docker-compose run nginx echo "helloworld" # 在工程中指定服务的容器上执行 echo "helloworld"
exec
docker-compose exec进入服务容器。
docker-compose exec nginx bash # 进入工程中指定服务的容器
docker-compose exec --index=1 nginx bash # 当一个服务拥有多个容器时,可通过 --index 参数进入到该服务下的任何容器
pause
docker-compose pause暂停服务容器
docker-compose pause # 暂停工程中所有服务的容器
docker-compose pause nginx # 暂停工程中指定服务的容器
unpause
docker-compose unpause恢复服务容器。
docker-compose unpause # 恢复工程中所有服务的容器
docker-compose unpause nginx # 恢复工程中指定服务的容器
restart
docker-compose restart重启服务容器。
docker-compose restart # 重启工程中所有服务的容器
docker-compose restart nginx # 重启工程中指定服务的容器
start
docker-compose start启动服务容器。
docker-compose start # 启动工程中所有服务的容器
docker-compose start nginx # 启动工程中指定服务的容器
stop
docker-compose stop停止服务容器。
docker-compose stop # 停止工程中所有服务的容器
docker-compose stop nginx # 停止工程中指定服务的容器
sudo docker-compose ps
sudo docker-compose stop jobmanager
sudo docker-compose stop taskmanager
sudo docker-compose rm -f jobmanager
sudo docker-compose rm -f taskmanager
sudo docker-compose exec jobmanager ./bin/sql-client.sh -f sql/flink_kafka1.sql
sudo docker-compose exec jobmanager ./bin/flink list
sudo docker logs -f -t --since="2023-05-08" --tail=200 flink_taskmanager_1
sudo docker-compose logs -f taskmanager
sudo docker-compose exec jobmanager bash
sudo docker-compose exec jobmanager ./bin/flink cancel 8d8cc94d73f7bd0c4cdc557264553a04
sudo docker-compose scale taskmanager=5
-
准备工作
sudo yum install docker
sudo yum install docker-compose
-
创建目录
mkdir flink
-
vim docker-compose.yml
version: "2.1"
services:
jobmanager:
image: flink:1.14
expose:
- "6123"
ports:
- "8081:8081"
command: jobmanager
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager
taskmanager:
image: flink:1.14
expose:
- "6121"
- "6122"
depends_on:
- jobmanager
command: taskmanager
links:
- "jobmanager:jobmanager"
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager
-
启动
启动报错
$ docker-compose up
ERROR: Couldn't connect to Docker daemon at http+docker://localunixsocket - is it running?
If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.
解决参考:https://blog.csdn.net/xiojing825/article/details/79494408
# 加上sudo,这个会输出详细日志到窗口
flink]$ sudo docker-compose up
Creating network "flink_default" with the default driver
Pulling jobmanager (flink:1.14)...
1.14: Pulling from library/flink
301a8b74f71f: Pull complete
4688df200b56: Pull complete
317eff501ed4: Pull complete
docker-compose up -d 后台运行,不输出日志到窗口
-
启动多个taskmanager
# 启用5个taskmanager
docker-compose scale taskmanager=5
-
浏览器观看
http://IP:8082
想web访问,但是只能访问localhost,本地ip都无法访问。
解决方法:https://blog.csdn.net/as350144/article/details/125654394
-
停止容器
flink]$ sudo docker-compose ps
Name Command State Ports
---------------------------------------------------------------------------------------------------------------------------
flink_jobmanager_1 /docker-entrypoint.sh jobm ... Up 6123/tcp, 8081/tcp, 0.0.0.0:8082->8082/tcp,:::8082->8082/tcp
flink_taskmanager_1 /docker-entrypoint.sh task ... Up 6121/tcp, 6122/tcp, 6123/tcp, 8081/tcp
看下yml配置文件的服务名字,jobmanager和taskmanager
[xx@sandbox-test flink]$ sudo docker-compose stop jobmanager
Stopping flink_jobmanager_1 ... done
[xx@sandbox-test flink]$ sudo docker-compose stop taskmanager
Stopping flink_taskmanager_1 ... done
-
完整的
# 启动flink
[xx@sandbox-test flink]$ sudo docker-compose up -d
Starting flink_jobmanager_1 ... done
Starting flink_taskmanager_1 ... done
# 查看容器
[xx@sandbox-test flink]$ sudo docker-compose ps
Name Command State Ports
---------------------------------------------------------------------------------------------------------------------------
flink_jobmanager_1 /docker-entrypoint.sh jobm ... Up 6123/tcp, 8081/tcp, 0.0.0.0:8082->8082/tcp,:::8082->8082/tcp
flink_taskmanager_1 /docker-entrypoint.sh task ... Up 6121/tcp, 6122/tcp, 6123/tcp, 8081/tcp
# 查看日志
[xx@sandbox-test flink]$ sudo docker-compose logs
或者 docker-compose logs -f taskmanager
# 扩展taskmanager
[xx@sandbox-test flink]$ sudo docker-compose scale taskmanager=5
WARNING: The scale command is deprecated. Use the up command with the --scale flag instead.
Starting flink_taskmanager_1 ... done
Creating flink_taskmanager_2 ... done
Creating flink_taskmanager_3 ... done
Creating flink_taskmanager_4 ... done
Creating flink_taskmanager_5 ... done
-
扩展(修改flink的conf)
如果要修改flink的conf呢?
参考:https://www.dandelioncloud.cn/article/details/1589983093958324225
https://www.clzg.cn/article/432005.html
flink下创建以下目录并将flink-conf.yaml拷贝到两个目录下
├── conf
│ ├── job
│ │ └── flink-conf.yaml
│ └── task
│ └── flink-conf.yaml
docker-compose.yml修改如下:主要是增加了volumes,表示使用本地的conf文件,不使用容器里面自带的conf。
version: "2.1"
services:
jobmanager:
image: flink:1.14
expose:
- "6123"
ports:
- "8082:8082"
command: jobmanager
volumes:
- /home/disk1/xx/soft/flink/conf/job/flink-conf.yaml:/opt/flink/conf/flink-conf.yaml
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager
taskmanager:
image: flink:1.14
expose:
- "6121"
- "6122"
depends_on:
- jobmanager
command: taskmanager
links:
- "jobmanager:jobmanager"
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager
volumes:
- /home/disk1/xx/soft/flink/conf/task/flink-conf.yaml:/opt/flink/conf/flink-conf.yaml
flink-conf.yaml需要做一些修改,修改内容参考上面的链接。
就可以启动flink了。
sudo docker-compose up -d
sudo docker-compose scale taskmanager=5
-
lib jar包
参考 (修改flink的conf)
-
执行sql 语句
flink sql执行参考:https://www.cnblogs.com/isuning/p/16214356.html
报错如下,解决参考:https://help.aliyun.com/document_detail/468412.html#section-jtg-wow-qey
sandbox-test flink]$ sudo docker-compose exec jobmanager ./bin/sql-client.sh -f flink_kafka1.sql
Exception in thread "main" org.apache.flink.table.client.SqlClientException: Unexpected exception. This is a bug. Please consider filing an issue.
at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:201)
at org.apache.flink.table.client.SqlClient.main(SqlClient.java:161)
Caused by: java.lang.NoSuchMethodError: org.apache.flink.table.catalog.FunctionCatalog.<init>(Lorg/apache/flink/configuration/ReadableConfig;Lorg/apache/flink/table/catalog/CatalogManager;Lorg/apache/flink/table/module/ModuleManager;)V
at org.apache.flink.table.client.gateway.context.SessionContext.create(SessionContext.java:237)
at org.apache.flink.table.client.gateway.local.LocalContextUtils.buildSessionContext(LocalContextUtils.java:87)
at org.apache.flink.table.client.gateway.local.LocalExecutor.openSession(LocalExecutor.java:87)
at org.apache.flink.table.client.SqlClient.start(SqlClient.java:88)
at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:187)
... 1 more
flink版本和jar包之间不兼容。
将flink版本改为1.16,重新启动docker
sudo docker-compose exec jobmanager ./bin/sql-client.sh embedded -f sql/flink_kafka2.sql报错如下
Flink SQL> [INFO] Submitting SQL update statement to the cluster...
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.flink.api.java.ClosureCleaner (file:/opt/flink/lib/flink-dist-1.16.0.jar) to field java.lang.String.value
WARNING: Please consider reporting this to the maintainers of org.apache.flink.api.java.ClosureCleaner
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
解决方法:https://blog.csdn.net/LIZHONGPING00/article/details/127947431
https://blog.csdn.net/qq_41538097/article/details/129113866文章来源:https://www.toymoban.com/news/detail-752959.html
docker容器中安装包
0、首先更新源
# apt-get update
1、安装wget
# apt-get install wget
2、安装ping,ifconfig
# apt-get install inetutils-ping
# apt-get install net-tools
3、安装gcc
# apt-get install gcc
4、安装各种依赖包
# apt-get install zlib* bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel tcl-devel gdbm-devel db4-devel libpcap-devel xz-devel libffi-dev
详见https://www.cnblogs.com/aliensir/articles/15838834.html文章来源地址https://www.toymoban.com/news/detail-752959.html
到了这里,关于docker-compose 部署flink集群的文章就介绍完了。如果您还想了解更多内容,请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章,希望大家以后多多支持TOY模板网!