scala - Spark build.sbt 文件版本控制
问题描述
我很难理解 spark 程序的 build.sbt 文件中的多个版本号。
1. version
2. scalaVersion
3. spark version?
4. revision number.
这些版本之间也有多种兼容性。你能解释一下如何为我的项目决定这些版本吗?
解决方案
我希望以下 SBT 行及其评论足以解释您的问题。
// The version of your project itself.
// You can change this value whenever you want,
// e.g. everytime you make a production release.
version := "0.1.0"
// The Scala version your project uses for compile.
// If you use spark, you can only use a 2.11.x version.
// Also, because Spark includes its own Scala in runtime
// I recommend you use the same one;
//you can check which one your Spark instance uses in the spark-shell.
scalaVersion := "2.11.12"
// The spark version the project uses for compile.
// Because you wont generate an uber jar with Spark included,
// but deploy your jar to an spark cluster instance.
// This version must match with the remote one, unless you want weird bugs...
val SparkVersion = "2.3.1"
// Note, I use a val with the Spark version
// to make it easier to include several Spark modules in my project,
// this way, if I want/have to change the Spark version,
// I only have to modify one line,
// and avoid strange erros because I changed some versions, but not others.
// Also note the 'Provided' modifier at the end,
// it indicates SBT that it shouldn't include the Spark bits in the generated jar
// neither in package nor assembly tasks.
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % SparkVersion % Provided,
"org.apache.spark" %% "spark-sql" % SparkVersion % Provided,
)
// Exclude Scala from the assembly jar, because spark already includes it.
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
您还应该注意 SBT 版本,即项目中使用的 SBT 版本。您在“project/build.properties”文件中设置它。
sbt.version=1.2.3
注意: 我使用sbt-assembly插件来生成一个 jar,其中包含除 Spark 和 Scala 之外的所有依赖项。如果您使用其他库,例如 MongoSparkConnector,这将非常有用。
推荐阅读
- node.js - Google calendarList 未使用服务帐户返回新的日历资源
- flutter - 如何像在音乐播放器中一样在 Flutter 中创建自定义通知并处理按钮点击
- java - 无法将 ssl 密钥导入到 cacerts keytool
- java - @ComponentScan 在外部 jar 中找不到 bean。可以吗?
- c# - CalendarDatePicker 有两种阴影背景
- python - AttributeError:“zip”对象没有属性“ndim”
- asp.net-core - 卡在不显眼的客户端验证中
- sql-server - 使用子查询插入多行
- azure-devops - 在公共扩展中重命名已发布的任务
- flutter - 小部件中的颤振相机预览,而不是全屏