hive - java.lang.NoSuchMethodError：org.apache.hadoop.hive.ql.exec.Utilities.addToClassPath

Question

我使用 CDH 6.3.2 蜂巢作业，出现错误

24-06-2021 13:35:18 CST hive-job INFO - Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hive.ql.exec.Utilities.addToClassPath(Ljava/lang/ClassLoader;[Ljava/lang/String;)Ljava/lang/ClassLoader;
24-06-2021 13:35:18 CST hive-job INFO -     at azkaban.jobtype.HadoopSecureHiveWrapper.runHive(HadoopSecureHiveWrapper.java:123)
24-06-2021 13:35:18 CST hive-job INFO -     at azkaban.jobtype.HadoopSecureHiveWrapper$1.run(HadoopSecureHiveWrapper.java:71)
24-06-2021 13:35:18 CST hive-job INFO -     at azkaban.jobtype.HadoopSecureHiveWrapper$1.run(HadoopSecureHiveWrapper.java:68)
24-06-2021 13:35:18 CST hive-job INFO -     at java.security.AccessController.doPrivileged(Native Method)
24-06-2021 13:35:18 CST hive-job INFO -     at javax.security.auth.Subject.doAs(Subject.java:422)
24-06-2021 13:35:18 CST hive-job INFO -     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
24-06-2021 13:35:18 CST hive-job INFO -     at azkaban.jobtype.HadoopSecureHiveWrapper.main(HadoopSecureHiveWrapper.java:68)
24-06-2021 13:35:18 CST hive-job INFO - Process with id 14985 completed unsuccessfully in 2 seconds.
24-06-2021 13:35:18 CST hive-job ERROR - caught error running the job
java.lang.RuntimeException: azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 1
    at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:312)
    at azkaban.jobtype.AbstractHadoopJavaProcessJob.run(AbstractHadoopJavaProcessJob.java:50)
    at azkaban.jobtype.HadoopHiveJob.run(HadoopHiveJob.java:52)
    at azkaban.execapp.JobRunner.runJob(JobRunner.java:830)
    at azkaban.execapp.JobRunner.doRun(JobRunner.java:607)
    at azkaban.execapp.JobRunner.run(JobRunner.java:568)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 1
    at azkaban.jobExecutor.utils.process.AzkabanProcess.run(AzkabanProcess.java:125)
    at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:304)
    ... 10 more
24-06-2021 13:35:18 CST hive-job ERROR - java.io.IOException: Exception reading file:/tmp/mr-azkaban3460056546681440256.tokenFailed to cancel tokens Exception reading file:/tmp/mr-azkaban3460056546681440256.tokenjava.io.FileNotFoundException: /tmp/mr-azkaban3460056546681440256.token (Permission denied)
24-06-2021 13:35:18 CST hive-job ERROR - Job run failed!
java.lang.RuntimeException: azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 1
    at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:312)
    at azkaban.jobtype.AbstractHadoopJavaProcessJob.run(AbstractHadoopJavaProcessJob.java:50)
    at azkaban.jobtype.HadoopHiveJob.run(HadoopHiveJob.java:52)
    at azkaban.execapp.JobRunner.runJob(JobRunner.java:830)
    at azkaban.execapp.JobRunner.doRun(JobRunner.java:607)
    at azkaban.execapp.JobRunner.run(JobRunner.java:568)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 1
    at azkaban.jobExecutor.utils.process.AzkabanProcess.run(AzkabanProcess.java:125)
    at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:304)
    ... 10 more
24-06-2021 13:35:18 CST hive-job ERROR - azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 1 cause: azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 1
24-06-2021 13:35:18 CST hive-job INFO - Finishing job hive-job at 1624512918756 with status FAILED

我已经修改了 azkaban-exec/bin/internal/internal-start-executor.sh

#CLASSPATH=$CLASSPATH:$HADOOP_HOME/conf:$HADOOP_HOME/*
CLASSPATH=$CLASSPATH:/etc/hadoop/conf:/opt/cloudera/parcels/CDH/jars/*

否则我会得到一个错误“线程“主”java.lang.NoClassDefFoundError：org/apache/hadoop/conf/Configuration中的异常”

我的 commonprivate.properties

…………………………
hadoop.classpath=hadoop classpath
hadoop.home=/opt/cloudera/parcels/CDH/lib/hadoop
hive.home=/opt/cloudera/parcels/CDH/lib/hive
spark.home=/opt/cloudera/parcels/CDH/lib/spark
hadoop.conf=/etc/hadoop/conf
jobtype.global.classpath=${hadoop.conf},/opt/cloudera/parcels/CDH/jars/*,{hadoop.classpath}

标签： hiveazkaban