python - 如何在 mypy 中定义隐含导入?
问题描述
我正在尝试对从 Databricks 导出的笔记本进行类型检查。笔记本是*.py
具有特殊注释格式的普通文件,用于指示单元格的开始和结束位置。mypy 没有理由不能对这些文件进行类型检查,除了一些缺少的名称:
spark
sc
dbutils
display
displayHTML
我知道在将您转储到交互模式之前,该python
命令将运行环境变量指定的文件。PYTHONSTARTUP
这就是开始定义这些名称的方式。
mypy 中是否有一个钩子可以让您在代码之外定义这样的名称?
解决方案
这是我想出的答案。它很脏,但它有效。我想要一个更好的答案,但在那之前,这就是有效的。
策略是使用 shell 脚本将“PYTHONSTARTUP”文件添加到每个笔记本,然后在最终输出中减去行号。
类型检查.sh:
#!/bin/bash
TARGET=$1
# Define the contents of "PYTHONSTARTUP" file inline. This just
# makes it easier to copy & paste this script elsewhere. You could also
# make it a separate *.py file.
PRELUDE="$(cat <<EOF
import typing
import pyspark.SparkContext
import pyspark.sql.SparkSession
spark = None # type: pyspark.sql.SparkSession
sc = None # type: pyspark.SparkContext
def display(expr):
pass
def displayHTML(expr):
pass
class dbutils:
class fs:
def help(): pass
def cp(from_: str, to: str, recurse: bool = False) -> bool: pass
def head(file: str, maxBytes: int) -> str: pass
def ls(dir: str) -> typing.List[str]: pass
def mkdirs(dir: str) -> bool: pass
def put(file: str, contents: str, overwrite: bool = False) -> bool: pass
def rm(dir: str, recurse: bool) -> bool: pass
def mount(source: str, mountPoint: str, encryptionType: str = "", owner: str = "", extraConfigs: typing.Map[str, str] = {}) -> bool: pass
def mounts() -> typing.List[str]: pass
def refreshMounts() -> bool: pass
def unmount(mountPoint: str) -> bool: pass
class notebook:
def exit(value: str): pass
def run(path: str, timeout: int, arguments: typing.Map[str, str]) -> str: pass
class widgets:
def combobox(name: str, defaultValue: str, choices: typing.List[str], label: str = ""): pass
def dropdown(name: str, defaultValue: str, choices: typing.List[str], label: str = ""): pass
def get(name: str) -> str: pass
def multiselect(name: str, defaultValue: str, choices: typing.List[str], label: str = ""): pass
def remove(name: str): pass
def removeAll(): pass
def text(name: str, defaultValue: str, label: str = ""): pass
def getArgument(name: str) -> str: pass
EOF
)"
# Remember the length of $PRELUDE so that we can subtract the line number
LEN="$(echo "$PRELUDE" | wc -l | awk '{ print $1 }')"
for file in $(find $TARGET -name '*.py'); do
# run mypy for the two files concatenated together (with a blank line
# for good measure)
OUTPUT=$(mypy -c "$(cat <<EOF
$PRELUDE
$(cat $file)
EOF
)")
# awk: Take only output where the line number is after the PRELUDE. Also, fix the file name and line number
FILE_OUTPUT="$(echo "$OUTPUT" | awk -F: '$2 > '$LEN' { line=($2-'$LEN')-1; $1=""; $2=""; print "'$file':" line ":" $0 }')"
# Remove blank lines from output before printing
if [[ $(echo "$FILE_OUTPUT" | sed '/^$/d' | wc -l) -gt 0 ]]; then
echo "$FILE_OUTPUT"
fi
# Keep track of all output, so we can decide the exit code
ALL_OUTPUT+="$FILE_OUTPUT"
done
# propagate errors to the exit code, but ignore errors in the prelude. This
# makes it easier to use in a CI pipeline.
if [[ $(echo "$ALL_OUTPUT" | wc -l) -gt 1 ]]; then
exit 1
else
exit 0
fi
用法:
./typecheck.sh notebooks/
推荐阅读
- c - 如何将井字游戏的极小极大算法修改为仅深度为 3
- ckeditor - 如何在 CKEditor 中禁用默认标头并启用自定义标头?
- php - 如何在第三个关系中传递参数,当我们从第二个关系中获取值时?
- java - 异常后Java不打印
- android - 5分钟计数器如何使进度条顺利进行?
- php - 大规模的一对多性能
- php - 是否可以将 PHP 数组转换为 POST 请求?
- docker - 如何使用新的 docker 镜像而不是旧的?
- active-directory - ASP.NET 样板:Active Directory Ldap 身份验证无效的用户名或密码
- python - 正则表达式:仅提取数字直到第一个空格