python - 获取错误区域名称 [us-west-1] 无效 [Kinesis 流与 spark 集成]
问题描述
我已经设置了 spark 并尝试使用 python 示例运行 spark submit 命令以获取此处的 wordcount
以下是我为 spark submit 运行的命令:-
bin/spark-submit --packages org.apache.spark:spark-streaming-kinesis-asl_2.12:2.4.0 /Users/amitdhawan/Downloads/kinesis_wordcount_asl.py [appname] [streamname] [https://kinesis.us-west-1.amazonaws.com] [us-west-1]
我得到的错误如下: -
Traceback (most recent call last):
File "/Users/amitdhawan/Downloads/kinesis_wordcount_asl.py", line 76, in <module>
ssc, appName, streamName, endpointUrl, regionName, InitialPositionInStream.LATEST, 2)
File "/Users/amitdhawan/Downloads/spark-2.2.3-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/streaming/kinesis.py", line 92, in createStream
File "/Users/amitdhawan/Downloads/spark-2.2.3-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
File "/Users/amitdhawan/Downloads/spark-2.2.3-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o28.createStream.
: java.lang.IllegalArgumentException: Region name '[us-west-1]' is not valid
at org.apache.spark.streaming.kinesis.KinesisUtils$.$anonfun$validateRegion$2(KinesisUtils.scala:561)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.streaming.kinesis.KinesisUtils$.validateRegion(KinesisUtils.scala:561)
at org.apache.spark.streaming.kinesis.KinesisUtils$.$anonfun$createStream$1(KinesisUtils.scala:75)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.streaming.StreamingContext.withNamedScope(StreamingContext.scala:274)
at org.apache.spark.streaming.kinesis.KinesisUtils$.createStream(KinesisUtils.scala:75)
at org.apache.spark.streaming.kinesis.KinesisUtils$.createStream(KinesisUtils.scala:509)
at org.apache.spark.streaming.kinesis.KinesisUtilsPythonHelper.createStream(KinesisUtils.scala:614)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:745)
不知道为什么我从 aws-console 确保我的 Kinesis 流和应用程序存在于 us-west-1 中的区域无效。
编辑:-
现在我在下面的命令中遇到错误以运行 kinesis 流
bin/run-example streaming.KinesisWordProducerASL streamname https://kinesis.us-west-1.amazonaws.com 1000 10
错误是:-
java.lang.ClassNotFoundException: org.apache.spark.examples.streaming.KinesisWordProducerASL
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:233)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:732)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
不知道如何注册这个课程。我还在 spark submit 命令中提供了 --class 参数,但这也不起作用。
解决方案
通过使用以下命令解决它: -
bin/spark-submit --packages org.apache.spark:spark-streaming-kinesis-asl_2.11:2.4.0 /Users/amitdhawan/Downloads/kinesis_wordcount_asl.py appname streamname https://kinesis.us-west-1.amazonaws.com us-west-1
观察方括号已删除,并将 scala 从 2.12 降级到 2.11
推荐阅读
- ruby-on-rails - 在表单提交之前机械化检查所有单选按钮
- java - Spring Boot 不接受括号中的列表参数
- php - 如何在 PHP 中使用 preg_replace 将表达式替换为元字符?
- php - 第二个怎么来
- 标签在我的代码中破坏了 {},而第一个
- 标签和标签没问题?
- networking - 网络爬取时的网络优化 - 使用 udp 和使用连接池?
- quarkus - Quarkus 中的 ConversationScoped
- python - 在 Python 中从另一个函数调用变量
- python - SyntaxError:解析python错误消息时出现意外的EOF
- html - 如何根据调用的内容自行制作边框大小?
- xpath - 如何获取动态加载类的 xpath 选择器