时间戳

首页 > 解决方案 > Dremio 字符集“ISO-8859-1”

apache-calcite - Dremio 字符集“ISO-8859-1”

问题描述

在 Dremio 中运行这个简单的查询时：

SELECT 'NBC Universal – NBC News' Data

我们收到以下错误消息

Failed to encode 'NBC Universal – NBC News' in character set 'ISO-8859-1'

我们尝试使用以下 docker 命令为字符集设置 saffron 设置：

docker run -p 9047:9047 -p 31010:31010 -p 45678:45678 -e DREMIO_JAVA_SERVER_EXTRA_OPTS="-Dcalcite.default.charset=UTF-16LE -Dcalcite.default.nationalcharset=UTF-16LE -Dcalcite.default.collation.name=UTF-16LE$en_US"  dremio/dremio-oss

Dremio 版本是 4.1

是否有任何配置可以使 Dremio 默认字符集为 UTF-8 或 UTF16 ？

我们正在查看此线程以了解如何在 Dremio 中配置字符集。

来自 docker 日志的异常详细信息：

com.dremio.common.exceptions.UserException：无法在 com.dremio.common.exceptions.UserException$Builder.build(UserException.java:776) 的字符集“ISO-8859-1”中编码“NBC Universal – NBC News” ) ~[dremio-common-4.1.3-202001022113020736-53142377.jar:4.1.3-202001022113020736-53142377] 在 com.dremio.exec.planner.sql.SqlExceptionHelper.coerceException(SqlExceptionHelper.java:126) ~[dremio-java:126) sabot-kernel-4.1.3-202001022113020736-53142377.jar:4.1.3-202001022113020736-53142377] 在 com.dremio.exec.planner.sql.handlers.query.NormalHandler.getPlan(NormalHandler.java:60) ~[dremio -sabot-kernel-4.1.3-202001022113020736-53142377.jar:4.1.3-202001022113020736-53142377] 在 com.dremio.exec.planner.sql.handlers.commands.HandlerToExec.plan(HandlerToExec.java:70) ~[ dremio-sabot-kernel-4.1.3-202001022113020736-53142377.jar:4。1.3-202001022113020736-53142377] 在 com.dremio.exec.work.foreman.AttemptManager.plan(AttemptManager.java:415) ~[dremio-sabot-kernel-4.1.3-202001022113020736-53142377.jar:4.1.3-202036 -53142377] 在 com.dremio.exec.work.foreman.AttemptManager.lambda$run$0(AttemptManager.java:324) ~[dremio-sabot-kernel-4.1.3-202001022113020736-53142377.jar:4.1.3-202001022113020736 -53142377] 在 com.dremio.service.commandpool.CommandWrapper.run(CommandWrapper.java:62) ~[dremio-services-commandpool-4.1.3-202001022113020736-53142377.jar:4.1.3-202001022113020736-53142377] .util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_232] 在 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_232] 在 java .lang.Thread.run(Thread.java:748) [na:1.8.0_232] 原因：org.apache.calcite.runtime.CalciteException：无法在 sun.reflect.NativeConstructorAccessorImpl.newInstance0（本机方法）的字符集“ISO-8859-1”中编码“NBC Universal – NBC News”~[na :1.8.0_232] 在 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_232] 在 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_232 ] 在 java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_232] 在 org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463) ~[calcite -core-1.16.0-201909031936280585-3f76472.jar:1.16.0-201909031936280585-3f76472] 在 org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:572) ~[calcite-core-1.16.0-201909031936280585-3f76472.jar:1.16.0-201909031936280585-3f76472] 在 org.apache.calcite.util.NlsString.(NlsString.java:81) ~[calcite-core-1.16.0-2019090319367280585-3f7: 1.16.0-201909031936280585-3f76472] 在 org.apache.calcite.rex.RexBuilder.makeLiteral(RexBuilder.java:887) ~[calcite-core-1.16.0-201909031936280585-3f76472.jar:1.16.0-276885-3f7163 ] 在 org.apache.calcite.rex.RexBuilder.makeVarCharLiteral(RexBuilder.java:1108) ~[calcite-core-1.16.0-201909031936280585-3f76472.jar:1.16.0-201909031936280585-3f76472] 在 org.apache.calcite .sql2rel.SqlNodeToRexConverterImpl.convertLiteral(SqlNodeToRexConverterImpl.java:120) ~[calcite-core-1.16.0-201909031936280585-3f76472.jar:1.16.0-201909031936280585-3f76472] at org.elapache.calcite.sql.sql访问（SqlToRelConverter.java：4639）~[calcite-core-1.16.0-201909031936280585-3f76472.jar:1.16.0-201909031936280585-3f76472] at org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:3988) -core-1.16.0-201909031936280585-3f76472.jar:1.16.0-201909031936280585-3f76472] 在 org.apache.calcite.sql.SqlLiteral.accept(SqlLiteral.java:533) ~[calcite-core-1.16.0- 201909031936280585-3f76472.jar:1.16.0-201909031936280585-3f76472] at org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4552) ~[calcite-core-1.16.0-201909031936280585-3f76472.jar ：1.16.0-201909031936280585-3f76472] 在 org.apache.calcite.sql2rel.StandardConvertletTable$14.convertCall(StandardConvertletTable.java:292) ~[calcite-core-1.16.0-201909031936280585-3f7642705085999993628050599 -3f76472] 在 org.apache。calcite.sql2rel.SqlNodeToRexConverterImpl.convertCall(SqlNodeToRexConverterImpl.java:63)~[calcite-core-1.16.0-201909031936280585-3f76472.jar:1.16.0-201909031936280585-3f76472] at org.Sqlrelel .visit(SqlToRelConverter.java:4650) ~[calcite-core-1.16.0-201909031936280585-3f76472.jar:1.16.0-201909031936280585-3f76472] 在 org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter java:3988) ~[calcite-core-1.16.0-201909031936280585-3f76472.jar:1.16.0-201909031936280585-3f76472] 在 org.apache.calcite.sql.SqlCall.accept(SqlCall.java:138) ~[calcite -core-1.16.0-201909031936280585-3f76472.jar:1.16.0-201909031936280585-3f76472] 在 org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4552) ~[calcite-core-1.1.0-201909031936280585-3f76472.jar:1.16.0-201909031936280585-3f76472] at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectList(SqlToRelConverter.java:3831) ~[calcite-core-1.16.0-201909031936280585-3f76472.jar :1.16.0-201909031936280585-3f76472] at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:662) ~[calcite-core-1.16.0-201909031936280585-3f764702.jar:1.31930-280590-1.316.02818590- 3f76472] 在 org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:619) ~[calcite-core-1.16.0-201909031936280585-3f76472.jar:1.16.0-201909031936280585-3f76472.apache] calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3056) ~[calcite-core-1.16.0-201909031936280585-3f76472.jar:1.16.0-201909031936280585-3f76472] at org.apache.calcite.sql2SqlToRelConverter.convertQuery(SqlToRelConverter.java:555) ~[calcite-core-1.16.0-201909031936280585-3f76472.jar:1.16.0-201909031936280585-3f76472] 在 com.dremio.execvert.planner.sql.SqlConverter.to( .java:320) ~[dremio-sabot-kernel-4.1.3-202001022113020736-53142377.jar:4.1.3-202001022113020736-53142377] 在 com.dremio.exec.planner.sql.handlers.PrelTransformer.toConvertibleRelRoot(PrelTransformer.toConvertibleRelRoot(PrelTransformer.toConvertibleRelRoot) java:777) ~[dremio-sabot-kernel-4.1.3-202001022113020736-53142377.jar:4.1.3-202001022113020736-53142377] 在 com.dremio.exec.planner.sql.handlers.PrelTransformer.convertToRelRoot(PrelTransformer. :862) ~[dremio-sabot-kernel-4.1.3-202001022113020736-53142377.jar:4.1.3-202001022113020736-53142377]0-201909031936280585-3f76472] 在 com.dremio.exec.planner.sql.SqlConverter.toConvertibleRelRoot(SqlConverter.java:320) ~[dremio-sabot-kernel-4.1.3-202001022113020736-53142377.jar:4.113-30202070202020202020 -53142377] 在 com.dremio.exec.planner.sql.handlers.PrelTransformer.toConvertibleRelRoot(PrelTransformer.java:777) ~[dremio-sabot-kernel-4.1.3-202001022113020736-53142377.jar:4.1.3-2020036-0221130207 53142377] 在 com.dremio.exec.planner.sql.handlers.PrelTransformer.convertToRelRoot(PrelTransformer.java:862) ~[dremio-sabot-kernel-4.1.3-202001022113020736-53142377.jar:4.1.3-202001022113027073 ]0-201909031936280585-3f76472] 在 com.dremio.exec.planner.sql.SqlConverter.toConvertibleRelRoot(SqlConverter.java:320) ~[dremio-sabot-kernel-4.1.3-202001022113020736-53142377.jar:4.113-30202070202020202020 -53142377] 在 com.dremio.exec.planner.sql.handlers.PrelTransformer.toConvertibleRelRoot(PrelTransformer.java:777) ~[dremio-sabot-kernel-4.1.3-202001022113020736-53142377.jar:4.1.3-2020036-0221130207 53142377] 在 com.dremio.exec.planner.sql.handlers.PrelTransformer.convertToRelRoot(PrelTransformer.java:862) ~[dremio-sabot-kernel-4.1.3-202001022113020736-53142377.jar:4.1.3-202001022113027073 ]planner.sql.handlers.PrelTransformer.toConvertibleRelRoot(PrelTransformer.java:777) ~[dremio-sabot-kernel-4.1.3-202001022113020736-53142377.jar:4.1.3-202001022113020736-53142377] at com.dremio.planner. .sql.handlers.PrelTransformer.convertToRelRoot(PrelTransformer.java:862) ~[dremio-sabot-kernel-4.1.3-202001022113020736-53142377.jar:4.1.3-202001022113020736-53142377]planner.sql.handlers.PrelTransformer.toConvertibleRelRoot(PrelTransformer.java:777) ~[dremio-sabot-kernel-4.1.3-202001022113020736-53142377.jar:4.1.3-202001022113020736-53142377] at com.dremio.planner. .sql.handlers.PrelTransformer.convertToRelRoot(PrelTransformer.java:862) ~[dremio-sabot-kernel-4.1.3-202001022113020736-53142377.jar:4.1.3-202001022113020736-53142377]

标签： apache-calcitedremio

解决方案

The issue was that $ in the calcite config has to be escaped, and also since Dremio (4.1) is still using calcite 1.16 , saffron config needs to be saffron.default instead of calcite.default. 

docker run -p 9047:9047 -e DREMIO_JAVA_SERVER_EXTRA_OPTS="-Dsaffron.default.charset=UTF-16LE -Dsaffron.default.nationalcharset=UTF-16LE -Dsaffron.default.collation.name=UTF-16LE\$en_US" dremio/dremio-oss:4.1.7

For direct fix in Dremio config files, dremio-env file needs to updated with this:

# Extra Java options - server only (dremio command)
#
DREMIO_JAVA_SERVER_EXTRA_OPTS="-Dsaffron.default.charset=UTF-16LE -Dsaffron.default.nationalcharset=UTF-16LE -Dsaffron.default.collation.name=UTF-16LE\$en_US"

推荐阅读