时间戳

首页 > 解决方案 > 如何为 flink docker 容器配置一些外部 jars 库

docker - 如何为 flink docker 容器配置一些外部 jars 库

问题描述

我正在运行具有以下配置的 flink docker 映像。

version: '2.1'
services:
  jobmanager:
    build: .
    image: flink
    volumes:
      - .:/usr/local/lib/python3.7/site-packages/pyflink/lib
    hostname: "jobmanager"
    expose:
      - "6123"
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager
  taskmanager:
    image: flink
    volumes:
    - .:/usr/local/lib/python3.7/site-packages/pyflink/lib
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager
    command: taskmanager
    links:
      - jobmanager:jobmanager
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

运行日志如下

    taskmanager_1  | 2020-10-11 10:34:03,714 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner      [] -  Classpath: 
  /opt/flink/lib/flink-csv-1.11.0.jar:
/opt/flink/lib/flink-json-1.11.0.jar:
/opt/flink/lib/flink-shaded-zookeeper-3.4.14.jar:
/opt/flink/lib/flink-table-blink_2.12-1.11.0.jar:
/opt/flink/lib/flink-table_2.12-1.11.0.jar:
/opt/flink/lib/log4j-1.2-api-2.12.1.jar:
/opt/flink/lib/log4j-api-2.12.1.jar:
/opt/flink/lib/log4j-core-2.12.1.jar:
/opt/flink/lib/log4j-slf4j-impl-2.12.1.jar:
/opt/flink/lib/flink-dist_2.12-1.11.0.jar:::

从日志中您可以看到一些默认库加载到系统中，但我想添加一些像 flink-jdbc_2.11-1.9.0.jar 这样的 jar，它位于我的本地文件系统中。

我的本地 jar 文件夹是/usr/local/lib/python3.7/sitepackages/pyflink/lib

我试过了

volumes:
  - /opt/flink/lib/:/usr/local/lib/python3.7/site-packages/pyflink/lib

错误是

ERROR: for keras-flask-deploy-webapp-master_jobmanager_1  Cannot start service jobmanager: Mounts denied: 
The path /opt/flink/lib is not shared from the host and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> Resources -> File Sharing.

我试过了，但仍然无法正常工作

volumes:
  - /usr/local/lib/python3.7/site-packages/pyflink/lib:/opt/flink/lib

[ERROR] Flink distribution jar not found in /opt/flink/lib.
taskmanager_1  | [ERROR] The execution result is empty.

如何将 jdbc jar 库链接到 flink 的 docker 容器？

标签： dockercontainersapache-flink

解决方案

在这种情况下，解决方法是将带有特定 jar 的卷附加到容器中的某个临时位置，并覆盖运行命令以将附加文件复制到/opt/flink/libflink 类路径文件夹中：

volumes:
  - /usr/local/lib/python3.7/site-packages/pyflink/lib:/opt/flink/lib-custom
command: cp -a /opt/flink/lib-custom/. /opt/flink/lib/ && jobmanager

如果您从那里运行应用程序，我会说调整 jobmanager 容器就足够了，但如果不是 - 在 taskmanager 中也重复相同的操作。

推荐阅读