python - 让所有snakemake在SGE集群的临时暂存空间中运行

Question

在 SGE 集群上运行。直到最近我的集群人员告诉我，我需要启动“作业使存储上的 bossock 光纤超载，并且作业的 IO 配置文件使得我们的存储无法满足请求。我们希望输出文件和可能的输入文件将由snakemake在暂存空间中读取和写入，然后您将需要复制您想要的任何内容并从暂存中删除文件。”

我一直在使用以下 bash 脚本提交snakemake。

#!/bin/bash
#Submit to the cluster, give it a unique name
#$ -S /bin/bash

#$ -V
#$ -l h_vmem=1.9G,h_rt=20:00:00,tmem=1.9G
#$ -l tscratch=100G
#$ -pe smp 2

# join stdout and stderr output
#$ -j y
#$ -R y


if [ "$1" != "" ]; then
    RUN_NAME=$1
else
    RUN_NAME=$""
fi

#setup scratch space
scratch=/scratch0/annbrown/$JOB_ID



#FOLDER=submissions/$(date +"%Y%m%d%H%M")
FOLDER=${scratch}submissions/$(date +"%Y%m%d%H%M")
mkdir -p $FOLDER
# make /a symlink of my snakemake pipeline in the scratch space
ln -s /home/annbrown/pipelines/rna_seq_snakemake $FOLDER

cd $FOLDER/rna_seq_snakemake


cp config/config.yaml $FOLDER/$RUN_NAMEconfig.yaml


snakemake -s rna_seq.snakefile \
--jobscript cluster_qsub.sh \
--cluster-config config/cluster.yaml \
--cluster-sync "qsub -R y -l h_vmem={cluster.h_vmem},h_rt={cluster.h_rt} -pe {cluster.pe} -o $FOLDER" \
-j 25 \
--nolock \
--rerun-incomplete \
--latency-wait 100

#copy work out of scratch -you may need to change the destination

cp -r $FOLDER ~/annbrown

#delete scratch once you have finished (very important)
rm -rf $scratch

但是每当snakemake开始运行时，它仍然会将它的提交临时文件写入错误的位置

/SAN/vyplab/alb_projects/pipelines/rna_seq_snakemake/.snakemake/tmp.gkxzqok7/snakejob.run_star_pe.139.sh
#$ -S /bin/bash

#$ -cwd
#$ -V
#$ -l h_vmem=4G,h_rt=6:00:00,tmem=4G


# join stdout and stderr output
#$ -j y
#$ -sync y
#$ -R y




cd /SAN/vyplab/alb_projects/pipelines/rna_seq_snakemake && \
/share/apps/python-3.7.2-shared/bin/python3.7 \
-m snakemake /SAN/vyplab/alb_projects/data/muscle/analysis/STAR_aligned/12_9_5_18.Aligned.out.bam --snakefile /SAN/vyplab/alb_projects/pipelines/rna_seq_snakemake/rna_seq.snakefile \
--force -j --keep-target-files --keep-remote \
--wait-for-files /SAN/vyplab/alb_projects/pipelines/rna_seq_snakemake/.snakemake/tmp.gkxzqok7 /SAN/vyplab/vyplab_reference_genomes/STAR/human/GRCh38/star_indices_overhang150/SA /SAN/vyplab/alb_projects/data/muscle/analysis/merged_fastqs/12_9_5_18_1.merged.fastq.gz /SAN/vyplab/alb_projects/data/muscle/analysis/merged_fastqs/12_9_5_18_2.merged.fastq.gz --latency-wait 100 \
 --attempt 1 --force-use-threads \
--wrapper-prefix https://bitbucket.org/snakemake/snakemake-wrappers/raw/ \
   --allowed-rules run_star_pe --nocolor --notemp --no-hooks --nolock \
--mode 2 

有没有人让snakemake在临时的临时空间中运行？我不确定我是打算使用影子指令还是说实话。

标签： pythonbashsnakemake