sqoop - 用于 postgres 的 Sqoop 导出存储过程
问题描述
我正在尝试使用 sqoop 执行存储过程。想法是从 HDFS 文件对我的 postgres 表执行 upsert:
HDFS 文件
URL ,LAST_UPDATED, LAST_SEEN, STATUS_CODE
http://10.230.64.213:1080/mockserver/dashboard,2020-11-08 16:28:20.655,2020-11-08 16:28:20.655,FAIL
http://109.230.64.213:1080/mockserver/dashboard,2020-11-08 16:28:20.655,2020-11-08 16:28:20.655,FAIL
这是我的 sqoop 导出命令;
sqoop export
--connect jdbc:postgresql://myconnection
--username username
--password pwd
--export-dir postGresCopy.csv
--input-fields-terminated-by ','
--call proc
--direct
-- --schema myschema
这是我的程序过程。
CREATE OR REPLACE PROCEDURE merge_db(url TEXT, last_updated TIMESTAMP,last_seen TIMESTAMP,status_code TEXT) RETURNS VOID AS
$$
BEGIN
LOOP
-- first try to update the key
UPDATE validator SET d=status_code,c=last_seen,b=last_updated WHERE a=url;
IF found THEN
RETURN;
END IF;
-- not there, so try to insert the key
-- if someone else inserts the same key concurrently,
-- we could get a unique-key failure
BEGIN
INSERT INTO validator(a,b,c,d) VALUES (url, last_updated,last_seen,status_code);
RETURN;
EXCEPTION WHEN unique_violation THEN
-- Do nothing, and loop to try the UPDATE again.
END;
END LOOP;
END;
LANGUAGE plpgsql;
执行 sqoop 导出时出现以下错误:
Warning: /opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
20/11/09 17:28:27 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.16.2
20/11/09 17:28:27 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
20/11/09 17:28:27 INFO manager.SqlManager: Using default fetchSize of 1000
20/11/09 17:28:27 INFO manager.PostgresqlManager: We will use schema myschema
20/11/09 17:28:27 INFO tool.CodeGenTool: Beginning code generation
20/11/09 17:28:27 ERROR tool.ExportTool: Encountered IOException running export job:
java.io.IOException: No columns to generate for ClassWriter
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1663)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106)
at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:75)
at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:111)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
解决方案
推荐阅读
- node.js - 无法读取未定义节点 Js Post 请求的属性
- xml - 错误:XML 文档在最终结束标记之后需要开始标记
- swift - iOS SwiftUI:如何在视图中检测 UITapGesture 的位置?
- reactjs - 如何将 Stockfish.js 添加到我的 react (+ typescript) 应用程序中?
- javascript - 网络音频:播放立体声(2 声道)mp3,因此只能在左耳听到通道 1,在右耳听到通道 2
- python - 从数据框创建 json 文件
- azure-devops - Azure Devops 拉取请求中的 Sonarcloud 门卡在等待状态
- javascript - NEXT_LOCALE cookie 被忽略。难道我做错了什么?
- serverless - 无法在 Azure Purview 上开始扫描 Azure Synapse Analytics 无服务器池
- python - Python asyncio 取消未等待的协程