amazon-web-services - Snakemake 和 Tibanna：找不到字段“snakemake_main_filename”

Question

我正在尝试使用 --tibanna 选项在 AWS 上运行 snakemake。我确实按照本教程设置了 Tibanna ，并且我的工作流程在本地运行良好。但是，当我尝试在云上运行它时，在我的存储桶 'tibanna-bucket' 上使用命令snakemake --tibanna --default-remote-prefix=tibanna-bucket --use-conda，我遇到了以下错误：

(snakemake) ernestmordret@MacBook-Pro-de-Ernest workflow % snakemake --tibanna --default-remote-prefix=tibanna-bucket --use-conda --verbose
Building DAG of jobs...
sources=/Users/ernestmordret/Documents/GitHub/Snake-Activen/workflow/Snakefile
precommand=
bucket=tibanna-bucket
subdir=tibanna-bucket
Using shell: /bin/bash
Provided cloud nodes: 1
Job counts:
    count   jobs
    1   all
    1   fastp_pe
    1   get_fastq_pe
    1   kallisto_index
    1   kallisto_quant
    5
Resources before job selection: {'_cores': 9223372036854775807, '_nodes': 1}
Ready jobs (2):
    kallisto_index
    get_fastq_pe
Select jobs to execute...
Welcome to the CBC MILP Solver
Version: 2.10.5
Build Date: Oct 15 2020

command line - cbc /var/folders/xx/_135vjcd77l445nv441n8fw40000gn/T/89fbeeb095784985afbe9f4369722b4e-pulp.mps max ratio None allow None threads None presolve on strong None gomory on knapsack on probing on branch printingOptions all solution /var/folders/xx/_135vjcd77l445nv441n8fw40000gn/T/89fbeeb095784985afbe9f4369722b4e-pulp.sol (default strategy 1)
At line 2 NAME          MODEL
At line 3 ROWS
At line 7 COLUMNS
At line 18 RHS
At line 21 BOUNDS
At line 24 ENDATA
Problem MODEL has 2 rows, 2 columns and 4 elements
Coin0008I MODEL read with 0 errors
String of None is illegal for double parameter ratioGap value remains 0
String of None is illegal for double parameter allowableGap value remains 0
String of None is illegal for integer parameter threads value remains 0
String of None is illegal for integer parameter strongBranching value remains 5
Option for gomoryCuts changed from ifmove to on
Option for knapsackCuts changed from ifmove to on
Continuous objective value is 12 - 0.00 seconds
Cgl0004I processed model has 0 rows, 0 columns (0 integer (0 of which binary)) and 0 elements
Cbc3007W No integer variables - nothing to do
Cuts at root node changed objective from -12 to -1.79769e+308
Probing was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
Gomory was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
Knapsack was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
Clique was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
MixedIntegerRounding2 was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
FlowCover was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
TwoMirCuts was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
ZeroHalf was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)

Result - Optimal solution found

Objective value:                12.00000000
Enumerated nodes:               0
Total iterations:               0
Time (CPU seconds):             0.00
Time (Wallclock seconds):       0.00

Option for printingOptions changed from normal to all
Total time (CPU seconds):       0.00   (Wallclock seconds):       0.00

Selected jobs (1):
    get_fastq_pe
Resources after job selection: {'_cores': 9223372036854775801, '_nodes': 0}
running job using Tibanna...

[Wed Jan 27 21:49:21 2021]
rule get_fastq_pe:
    output: tibanna-bucket/data/SRR6663265_1.fastq, tibanna-bucket/data/SRR6663265_2.fastq
    jobid: 3
    wildcards: sample=SRR6663265
    threads: 6
    resources: mem_mb=1000, disk_mb=1000

job output tibanna-bucket/data/SRR6663265_1.fastq
job output is remote= true
 is remote default= true
job output tibanna-bucket/data/SRR6663265_2.fastq
job output is remote= true
 is remote default= true
additional tibanna config: None
Full Traceback (most recent call last):
  File "/Users/ernestmordret/opt/anaconda3/envs/snakemake/lib/python3.9/site-packages/snakemake/__init__.py", line 691, in snakemake
    success = workflow.execute(
  File "/Users/ernestmordret/opt/anaconda3/envs/snakemake/lib/python3.9/site-packages/snakemake/workflow.py", line 1007, in execute
    success = scheduler.schedule()
  File "/Users/ernestmordret/opt/anaconda3/envs/snakemake/lib/python3.9/site-packages/snakemake/scheduler.py", line 488, in schedule
    self.run(runjobs)
  File "/Users/ernestmordret/opt/anaconda3/envs/snakemake/lib/python3.9/site-packages/snakemake/scheduler.py", line 499, in run
    executor.run_jobs(
  File "/Users/ernestmordret/opt/anaconda3/envs/snakemake/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 136, in run_jobs
    self.run(
  File "/Users/ernestmordret/opt/anaconda3/envs/snakemake/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 2133, in run
    tibanna_input = self.make_tibanna_input(job)
  File "/Users/ernestmordret/opt/anaconda3/envs/snakemake/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 2108, in make_tibanna_input
    tibanna_args = ec2_utils.Args(
  File "/Users/ernestmordret/opt/anaconda3/envs/snakemake/lib/python3.9/site-packages/tibanna/ec2_utils.py", line 116, in __init__
    self.fill_default()
  File "/Users/ernestmordret/opt/anaconda3/envs/snakemake/lib/python3.9/site-packages/tibanna/ec2_utils.py", line 167, in fill_default
    raise MissingFieldInInputJsonException(errmsg_template % ('snakemake_main_filename', self.language))
tibanna.exceptions.MissingFieldInInputJsonException: field snakemake_main_filename is required in args for language snakemake

Traceback (most recent call last):
  File "/Users/ernestmordret/opt/anaconda3/envs/snakemake/lib/python3.9/site-packages/snakemake/__init__.py", line 691, in snakemake
    success = workflow.execute(
  File "/Users/ernestmordret/opt/anaconda3/envs/snakemake/lib/python3.9/site-packages/snakemake/workflow.py", line 1007, in execute
    success = scheduler.schedule()
  File "/Users/ernestmordret/opt/anaconda3/envs/snakemake/lib/python3.9/site-packages/snakemake/scheduler.py", line 488, in schedule
    self.run(runjobs)
  File "/Users/ernestmordret/opt/anaconda3/envs/snakemake/lib/python3.9/site-packages/snakemake/scheduler.py", line 499, in run
    executor.run_jobs(
  File "/Users/ernestmordret/opt/anaconda3/envs/snakemake/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 136, in run_jobs
    self.run(
  File "/Users/ernestmordret/opt/anaconda3/envs/snakemake/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 2133, in run
    tibanna_input = self.make_tibanna_input(job)
  File "/Users/ernestmordret/opt/anaconda3/envs/snakemake/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 2108, in make_tibanna_input
    tibanna_args = ec2_utils.Args(
  File "/Users/ernestmordret/opt/anaconda3/envs/snakemake/lib/python3.9/site-packages/tibanna/ec2_utils.py", line 116, in __init__
    self.fill_default()
  File "/Users/ernestmordret/opt/anaconda3/envs/snakemake/lib/python3.9/site-packages/tibanna/ec2_utils.py", line 167, in fill_default
    raise MissingFieldInInputJsonException(errmsg_template % ('snakemake_main_filename', self.language))
tibanna.exceptions.MissingFieldInInputJsonException: field snakemake_main_filename is required in args for language snakemake
unlocking
removing lock
removing lock
removed all locks

我很困惑：蛇文件应该在本地提供，还是在 s3 存储桶上提供？我应该如何组织我的 s3 存储桶以使其工作？

这是我用于此测试的蛇文件：

rule all:
    input:
        "quant_results_SRR6663265"

rule get_fastq_pe:
    output:
        # the wildcard name must be accession, pointing to an SRA number
        "data/{sample}_1.fastq",
        "data/{sample}_2.fastq"
    params:
        # optional extra arguments
        extra=""
    threads: 6  # defaults to 6
    wrapper:
        "0.70.0/bio/sra-tools/fasterq-dump"

rule fastp_pe:
    input:
        sample=["data/{sample}_1.fastq", "data/{sample}_2.fastq"]
    output:
        trimmed=["trimmed/{sample}_1.fastq", "trimmed/{sample}_2.fastq"],
        html="report/pe/{sample}.html",
        json="report/pe/{sample}.json"
    log:
        "logs/fastp/pe/{sample}.log"
    params:
        adapters="--detect_adapter_for_pe",
        extra=""
    threads: 2
    wrapper:
        "0.70.0/bio/fastp"


rule kallisto_index:
    input:
        fasta = "index/{transcriptome}.fasta"
    output:
        index = "index/{transcriptome}.idx"
    params:
        extra = "--kmer-size=31 --bias"
    log:
        "logs/kallisto_index_{transcriptome}.log"
    threads: 1
    wrapper:
        "0.70.0/bio/kallisto/index"


rule kallisto_quant:
    input:
        fastq = ["trimmed/{sample}_1.fastq","trimmed/{sample}_2.fastq"],
        index = "index/Homo_sapiens.GRCh38.cdna.all.idx"
    output:
        directory('quant_results_{sample}')
    params:
        extra = ""
    log:
        "logs/kallisto_quant_{sample}.log"
    threads: 4
    wrapper:
        "0.70.0/bio/kallisto/quant"

提前致谢！

标签： amazon-web-servicesamazon-s3snakemaketibanna