kubernetes - 权限拒绝执行下载到 Kubernetes 中的 .Ivy 文件夹的 jar 文件

Question

我创建了一个dockerfile并将用户更改为非root用户nobody。在本地，这非常有效。然而，当部署在 kubernetes 上时，我得到了错误

java.nio.file.AccessDeniedException: ./xxxxxx_2.12-2.6.3.jar

当我深入挖掘时，我意识到这个 jar 文件是在 dockerfiles 中使用的 spark 依赖项下载后下载的。因此，这个新下载的 jar 文件不存在授予 spark 文件夹的任何权限，该文件在运行时下载到具有 root 权限的 /opt/spark/.ivy2/xxx 中。这会导致 Kubernetes 中的 pod 失败。

我想知道是否有办法授予执行此 jar 文件的权限。因为这在 Dockerfile 中似乎是不可能的。关于如何解决这个问题的任何建议？

正如@mario 所提议的那样

ARG SPARK_OPERATOR_BASE_IMAGE_VERSION=v2.4.5
FROM xxx/alpine:3.12 as preparator

ARG SCALA_VERSION=2.12
ARG SPARK_VERSION=2.4.7
ARG HADOOP_VERSION=3.2.1
ARG AWS_SDK_VERSION=1.11.375
ARG MAVEN_VERSION=3.6.2

RUN apk add --no-cache \
        bash \
        curl && \
        mkdir /target

COPY hashes /tmp/
COPY prepare /tmp/

WORKDIR /tmp

# Download Hadoop
RUN curl -L -O https://downloads.apache.org/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz && \
    sha256sum -c hadoop-${HADOOP_VERSION}.sha256 && \
    tar -xzf hadoop-${HADOOP_VERSION}.tar.gz && \
    mv hadoop-${HADOOP_VERSION} /target/hadoop

# Download Spark
RUN curl -L -O https://downloads.apache.org/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-without-hadoop-scala-${SCALA_VERSION}.tgz && \
    sha512sum -c spark-${SPARK_VERSION}-bin-without-hadoop-scala-${SCALA_VERSION}.sha512 && \
    tar -xzf spark-${SPARK_VERSION}-bin-without-hadoop-scala-${SCALA_VERSION}.tgz && \
    mv spark-${SPARK_VERSION}-bin-without-hadoop-scala-${SCALA_VERSION} /target/spark && \
# Download Spark 3.0.0 entrypoint script from GitHub, bugfixing for 2.4.7
    curl -L -O https://raw.githubusercontent.com/apache/spark/v3.0.0/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh && \
    mv entrypoint.sh /target/entrypoint.sh && \
    chmod +x /target/entrypoint.sh

# Download AWS Jars
RUN curl -L -O https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/${HADOOP_VERSION}/hadoop-aws-${HADOOP_VERSION}.jar && \
    sha1sum -c hadoop-aws-${HADOOP_VERSION}.jar.sha1 && \
    mv hadoop-aws-${HADOOP_VERSION}.jar /target/spark/jars/ && \
    curl -L -O https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/${AWS_SDK_VERSION}/aws-java-sdk-bundle-${AWS_SDK_VERSION}.jar && \
    sha1sum -c aws-java-sdk-bundle-${AWS_SDK_VERSION}.jar.sha1 && \
    mv aws-java-sdk-bundle-${AWS_SDK_VERSION}.jar /target/spark/jars/

# Directory needed for saving built jars
RUN mkdir /target/spark/custom-jars/

#### Download Prometheus + Metric dependencies ####
# install java, maven and prometheus fat jar using maven (pom.xml)
RUN apk add --update openjdk8 && \
    curl -L -O https://archive.apache.org/dist/maven/maven-3/${MAVEN_VERSION}/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz && \
    tar -xzf apache-maven-${MAVEN_VERSION}-bin.tar.gz && export PATH=./apache-maven-${MAVEN_VERSION}/bin:$PATH && \
    mv prometheus-pom.xml pom.xml && mvn clean package && mv target/prometheusMetricLibs-jar-with-dependencies.jar /target/spark/custom-jars/

RUN \
  chown -R nobody:99 /target/spark \
  && chown -R nobody:99 /target/hadoop \
  && chmod -R ugo+rw /target/spark \
  && chmod -R ugo+rw /target/hadoop


ARG SPARK_OPERATOR_BASE_IMAGE_VERSION
FROM gcr.io/spark-operator/spark:${SPARK_OPERATOR_BASE_IMAGE_VERSION}

RUN rm -rf /opt/spark/

COPY --from=preparator /target/ /opt/

ENV SPARK_HOME=/opt/spark \
    HADOOP_HOME=/opt/hadoop

ENV HADOOP_OPTS="-Djava.library.path=/opt/hadoop/lib/native" \
    LD_LIBRARY_PATH=${HADOOP_HOME}/lib/native \
    PATH=${HADOOP_HOME}/bin:${SPARK_HOME}/bin:${PATH}

COPY conf /opt/spark/conf/

RUN echo "export JAVA_HOME=${JAVA_HOME}" >> ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh && \
    echo "export JAVA_HOME=${JAVA_HOME}" > ${SPARK_HOME}/conf/spark-env.sh && \
    echo "export SPARK_DIST_CLASSPATH=\$(hadoop classpath)" >> ${SPARK_HOME}/conf/spark-env.sh


# 99 used instead of nobody because in the alpine image the Group Id of nobody is different than in the spark operator image.
RUN \
  addgroup --gid 99 nobody \
  && echo "nobody:x:99:99:nobody:/nonexistent:/usr/sbin/nologin" >> /etc/passwd \
  && usermod -a -G users nobody \
  && chmod -R ugo+rw /var/lib/

USER nobody

# if we want local storage
#  "spark.eventLog.dir": "tmp/spark-events"
RUN mkdir -p /tmp/spark-events

在我的 pod 中，jar 文件是这样实现的

sparkConf:
    "spark.ui.port": "4045"
    "spark.eventLog.enabled": {{ .Values.spark.eventLogEnabled | quote }}
    "spark.eventLog.dir": "xx//"
    "spark.jars.ivySettings": "/vault/secrets/xx-ivysettings.xml"
    "spark.jars.ivy": "/opt/spark/.ivy2"
    "spark.jars.packages": "xxxx_2.12:{{ .Values.appVersion }}"
    "spark.blacklist.enabled": "false"
    "spark.driver.supervise": "true"
    "spark.app.name": {{ .Values.name | quote }}
    "spark.submit.deployMode": {{ .Values.spark.deployMode | quote }}
    "spark.driver.extraJavaOptions": "-Dlog4j.configurationFile=log4j.properties"
    "spark.executor.extraJavaOptions": "-Dlog4j.configurationFile=log4j.properties"

标签： kubernetesdockerfilekubernetes-pod

Answer 1

如果您共享您Dockerfile和您的 kubernetesPod模板 yaml 清单，那么回答您的问题会更容易，但简而言之：您可以使用以下文档中的示例中Pod的类似操作来操作您的权限：securityContext

apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
  volumes:
  - name: sec-ctx-vol
    emptyDir: {}
  containers:
  - name: sec-ctx-demo
    image: busybox
    command: [ "sh", "-c", "sleep 1h" ]
    volumeMounts:
    - name: sec-ctx-vol
      mountPath: /data/demo
    securityContext:
      allowPrivilegeEscalation: false

出于调试目的，您可以从设置allowPrivilegeEscalation为true或runAsUser: 0（root）开始，但请记住，这不是可以在生产中使用的解决方案。以 root 身份运行容器通常是一个坏主意，在大多数情况下可以避免。

因此，这个新下载的 jar 文件在运行时下载到具有 root 权限的 /opt/spark/.ivy2/xxx 中的这个新下载的 jar 文件不存在授予 spark 文件夹的任何权限。

它真的有必要拥有root权限吗？很可能它可以在您的Dockerfile.