apache-spark - 如何在 Spark 结构化流中使用连接池覆盖 ForeachWriter
问题描述
我想在 Spark 结构化流中使用连接池,但我不想使用 writeStream 来新建连接池,我应该如何更新?
现在我想在 application.conf 中设置变量,在 main 方法中获取 application.conf 位置之前加载对象,如下所示:
class JdbcSink extends ForeachWriter[Row] with Serializable with Settings {
self: SinkStatement =>
import JdbcSink.dsPool
var connection: Connection = _
//the sql statement
var statement: Statement = _
//open
def open(partitionId: Long, version: Long): Boolean = {
connection = dsPool.getConnection()
statement = connection.createStatement()
true
}
//execute
def process(value: Row): Unit = {
//execute
statement.executeUpdate(this.make(value))
}
//close
def close(errorOrNull: Throwable): Unit = {
//close the connection
dsPool.evictConnection(connection)
}
}
object JdbcSink extends Settings {
val config = new HikariConfig
config.setDriverClassName(this.sinkDbDriver)
config.setJdbcUrl(this.sinkDbUrl)
config.setUsername(this.sinkDbUser)
config.setPassword(this.sinkDbPwd)
config.addDataSourceProperty("cachePrepStmts", "true")
config.addDataSourceProperty("prepStmtCacheSize", "250")
config.addDataSourceProperty("prepStmtCacheSqlLimit", "2048")
val dsPool = new HikariDataSource(this.HConfig)
}
我如何获得灵活的设置?像这样,“with Settings”得到“can't initiallize jdbcSink”excetption
val config = new HikariConfig
config.setDriverClassName (this.sinkDbDriver)
config.setJdbcUrl (this.sinkDbUrl)
config.setUsername (this.sinkDbUser)
config.setPassword (this.sinkDbPwd)
config.addDataSourceProperty ("cachePrepStmts", "true")
config.addDataSourceProperty ("prepStmtCacheSize", "250")
config.addDataSourceProperty ("prepStmtCacheSqlLimit", "2048")
val dsPool = new HikariDataSource (this.HConfig)
解决方案
推荐阅读
- django - 无法通过迁移将新模型添加到 Django Postgres DB
- python - 将新变量添加到 Python 中的数据集后的关键错误
- java - 对象可以具有未在类中定义的方法或变量吗?
- mysql - Spring Boot 非唯一结果异常
- haskell - 在假设 a, b 已经是 Log-instance 的情况下,如何为 (a, b) 定义一个 Log-instance?
- python - 并非所有数据都在输出 Scrapy
- maven - 如何使用“mvn test”运行 Cucumber 测试
- html - Facebook页面嵌入iframe不填充容器宽度
- cobol - 如何在 Cobol 中调用另一个程序?
- javascript - 刷新更改javascript的下拉菜单