snakemake - InputFunctionException：解析时出现意外的 EOF

Question

主要编辑：由于评论修复了几个问题并编写了一个最小的可重现示例来帮助我的助手，我已将问题缩小到本地执行和使用 DRMAA 之间的差异。

这是一个最小的可重现管道，不需要任何外部文件下载，可以开箱即用或克隆以下 git 存储库：

git clone [e-mail]:kevinrue/snakemake-issue-all.git

当我使用 DRMAA 运行管道时，出现以下错误：

Building DAG of jobs...
Using shell: /bin/bash
Provided cluster nodes: 100
Singularity containers: ignored
Job counts:
    count   jobs
    1   all
    2   cat
    3
InputFunctionException in line 22 of /ifs/research-groups/sims/kevin/snakemake-issue-all/workflow/Snakefile:
SyntaxError: unexpected EOF while parsing (<string>, line 1)
Wildcards:
sample=A

但是，如果我在本地运行管道 ( --cores 1)，它可以工作：

Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Singularity containers: ignored
Job counts:
    count   jobs
    1   all
    2   cat
    3

[Sat Jun 13 08:49:46 2020]
rule cat:
    input: data/A1, data/A2
    output: results/A/cat
    jobid: 1
    wildcards: sample=A

[Sat Jun 13 08:49:46 2020]
Finished job 1.
1 of 3 steps (33%) done

[Sat Jun 13 08:49:46 2020]
rule cat:
    input: data/B1, data/B2
    output: results/B/cat
    jobid: 2
    wildcards: sample=B

[Sat Jun 13 08:49:46 2020]
Finished job 2.
2 of 3 steps (67%) done

[Sat Jun 13 08:49:46 2020]
localrule all:
    input: results/A/cat, results/B/cat
    jobid: 0

[Sat Jun 13 08:49:46 2020]
Finished job 0.
3 of 3 steps (100%) done
Complete log: /ifs/research-groups/sims/kevin/snakemake-issue-all/.snakemake/log/2020-06-13T084945.632545.snakemake.log

我的 DRMAA 个人资料如下：

jobs: 100
default-resources: 'mem_free=4G'
drmaa: "-V -notify -p -10 -l mem_free={resources.mem_free} -pe dedicated {threads} -v MKL_NUM_THREADS={threads} -v OPENBLAS_NUM_THREADS={threads} -v OMP_NUM_THREADS={threads} -R y -q all.q"
drmaa-log-dir: /ifs/scratch/kevin
use-conda: true
conda-prefix: /ifs/home/kevin/devel/snakemake/envs
printshellcmds: true
reason: true

简而言之，Snakefile 看起来像这样

# The main entry point of your workflow.
# After configuring, running snakemake -n in a clone of this repository should successfully execute a dry-run of the workflow.


report: "report/workflow.rst"

# Allow users to fix the underlying OS via singularity.
singularity: "docker://continuumio/miniconda3"

include: "rules/common.smk"
include: "rules/other.smk"

rule all:
    input:
        # The first rule should define the default target files
        # Subsequent target rules can be specified below. They should start with all_*.
        expand("results/{sample}/cat", sample=samples['sample'])


rule cat:
    input:
        file1="data/{sample}1",
        file2="data/{sample}2"
    output:
        "results/{sample}/cat"
    shell:
        "cat {input.file1} {input.file2} > {output}"

跑步snakemake -np给了我我的期望：

$ snakemake -np
          sample  condition
sample_id                  
A              A  untreated
B              B    treated
Building DAG of jobs...
Job counts:
    count   jobs
    1   all
    2   cat
    3

[Sat Jun 13 08:51:19 2020]
rule cat:
    input: data/B1, data/B2
    output: results/B/cat
    jobid: 2
    wildcards: sample=B

cat data/B1 data/B2 > results/B/cat

[Sat Jun 13 08:51:19 2020]
rule cat:
    input: data/A1, data/A2
    output: results/A/cat
    jobid: 1
    wildcards: sample=A

cat data/A1 data/A2 > results/A/cat

[Sat Jun 13 08:51:19 2020]
localrule all:
    input: results/A/cat, results/B/cat
    jobid: 0

Job counts:
    count   jobs
    1   all
    2   cat
    3
This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.

我不确定如何进一步调试它。我很乐意根据需要提供更多信息。

注意：我使用的是snakemake 5.19.2 版

提前致谢！

编辑 使用该--verbose选项，Snakemake 似乎在我的“drmaa”配置文件中定义的'default-resources: 'mem_free=4G'和/或上绊倒（见上文）。drmaa: "-l mem_free={resources.mem_free}

$ snakemake --profile drmaa --verbose
Building DAG of jobs...
Using shell: /bin/bash
Provided cluster nodes: 100
Singularity containers: ignored
Job counts:
    count   jobs
    1   all
    2   cat
    3
Resources before job selection: {'_cores': 9223372036854775807, '_nodes': 100}
Ready jobs (2):
    cat
    cat
Full Traceback (most recent call last):
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/rules.py", line 941, in apply
    res, _ = self.apply_input_function(
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/rules.py", line 684, in apply_input_function
    raise e
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/rules.py", line 678, in apply_input_function
    value = func(Wildcards(fromdict=wildcards), **_aux_params)
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/resources.py", line 10, in callable
    value = eval(
  File "<string>", line 1
    4G
     ^
SyntaxError: unexpected EOF while parsing

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/__init__.py", line 626, in snakemake
    success = workflow.execute(
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/workflow.py", line 951, in execute
    success = scheduler.schedule()
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/scheduler.py", line 394, in schedule
    run = self.job_selector(needrun)
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/scheduler.py", line 540, in job_selector
    a = list(map(self.job_weight, jobs))  # resource usage of jobs
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/scheduler.py", line 613, in job_weight
    res = job.resources
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/jobs.py", line 267, in resources
    self._resources = self.rule.expand_resources(
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/rules.py", line 977, in expand_resources
    resources[name] = apply(name, res, threads=threads)
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/rules.py", line 960, in apply
    raise InputFunctionException(e, rule=self, wildcards=wildcards)
snakemake.exceptions.InputFunctionException: SyntaxError: unexpected EOF while parsing (<string>, line 1)
Wildcards:
sample=B

InputFunctionException in line 20 of /ifs/research-groups/sims/kevin/snakemake-issue-all/workflow/Snakefile:
SyntaxError: unexpected EOF while parsing (<string>, line 1)
Wildcards:
sample=B
unlocking
removing lock
removing lock
removed all locks

标签： snakemake

Answer 1

感谢@JohannesKöster，我意识到我的个人资料设置是错误的。

--default-resources [NAME=INT [NAME=INT ...]]表示仅支持整数值，而我提供字符串（即，mem_free=4G），天真地希望这些值也能得到支持。

我已在我的个人资料中更新了以下设置，并成功运行了snakemake --cores 1和snakemake --profile drmaa.

default-resources: 'mem_free=4'
drmaa: "-V -notify -p -10 -l mem_free={resources.mem_free}G -pe dedicated {threads} -v MKL_NUM_THREADS={threads} -v OPENBLAS_NUM_THREADS={threads} -v OMP_NUM_THREADS={threads} -R y -q all.q"

请注意设置为默认资源的整数值4，以及我如何将其移至G设置drmaa: ... -l mem_free=...G。

非常感谢大家的帮助！