时间戳

首页 > 解决方案 > 如何在我的 docker 镜像中安装 postgresql？

postgresql - 如何在我的 docker 镜像中安装 postgresql？

问题描述

我正在尝试在我的 spark 应用程序中从 Postgresql 获取数据。但是现在我很困惑如何在我的 docker 映像中安装 postgresql 驱动程序。我还尝试将 postgresql 安装为 apt-get install 命令，如下所述（Dockerfile）。

Dockerfile：

FROM python:3


ENV SPARK_VERSION 2.3.2
ENV SPARK_HADOOP_PROFILE 2.7
ENV SPARK_SRC_URL https://www.apache.org/dist/spark/spark-$SPARK_VERSION/spark-${SPARK_VERSION}- 
bin-hadoop${SPARK_HADOOP_PROFILE}.tgz
ENV SPARK_HOME=/opt/spark
ENV PATH $PATH:$SPARK_HOME/bin

RUN wget ${SPARK_SRC_URL}
RUN tar -xzf spark-${SPARK_VERSION}-bin-hadoop${SPARK_HADOOP_PROFILE}.tgz

RUN mv spark-${SPARK_VERSION}-bin-hadoop${SPARK_HADOOP_PROFILE} /opt/spark
RUN rm -f spark-${SPARK_VERSION}-bin-hadoop${SPARK_HADOOP_PROFILE}.tgz
RUN apt-get update && \
apt-get install -y openjdk-8-jdk-headless \
 postgresql && \
rm -rf /var/lib/apt/lists/*
ENV JAVA_HOME  /usr/lib/jvm/java-8-openjdk-amd64/

COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt


COPY my_script.py ./
CMD [ "python", "./my_script.py" ]

requirements.txt：pyspark==2.3.2 numpy

my_script.py ：

from pyspark import SparkContext
from pyspark import SparkConf

#spark conf
conf1 = SparkConf()
conf1.setMaster("local[*]")
conf1.setAppName('hamza')
print(conf1)
sc = SparkContext(conf = conf1)
print('hahahha')

from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
print(sqlContext)

from pyspark.sql import DataFrameReader
url = 'postgresql://IP:PORT/INSTANCE'
properties = {'user': 'user', 'password': 'pass'}
df = DataFrameReader(sqlContext).jdbc(
url='jdbc:%s' % url, table=query, properties=properties
)

收到此错误：

Traceback (most recent call last):
  File "./my_script.py", line 26, in <module>
, properties=properties
  File "/usr/local/lib/python3.7/site-packages/pyspark/sql/readwriter.py", line 527, in jdbc
  return self._df(self._jreader.jdbc(url, table, jprop))
  File "/usr/local/lib/python3.7/site-packages/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
  File "/usr/local/lib/python3.7/site-packages/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
  File "/usr/local/lib/python3.7/site-packages/py4j/protocol.py", line 328, in get_return_value
format(target_id, ".", name), value)
       py4j.protocol.Py4JJavaError: An error occurred while calling o28.jdbc.
   : java.sql.SQLException: No suitable driver
    at java.sql.DriverManager.getDriver(DriverManager.java:315)

请指导我如何设置此驱动程序谢谢

标签： postgresqldockerapache-sparkpyspark

解决方案

在 Dockerfile 中添加这些行解决了这个问题：

ENV POST_URL https://jdbc.postgresql.org/download/postgresql-42.2.5.jar
RUN wget ${POST_URL}
RUN mv postgresql-42.2.5.jar /opt/spark/jars

感谢大家

推荐阅读