r - 如何使用命令行中的 rscript 命令在 R 中运行作业数组？

Question

我想知道如何在R使用该Rscript功能时运行 500 个并行作业。我目前有一个R顶部有标题的文件：

args <- commandArgs(TRUE)
B <- as.numeric(args[1])
Num.Cores <- as.numeric(args[2])

在 R 文件之外，我希望传递要运行的 500 个作业中的哪一个，由B. 另外，我想控制每个作业可用的核心/CPU 数量，Num.Cores.

我想知道是否有软件或指南可以做到这一点。我目前有一个 CentOS 7/Linux 服务器，我知道一种方法是安装 Slurm。但是，这很麻烦，我想知道是否有办法通过队列执行 500 个作业。谢谢。

标签： rparameter-passingcluster-computinghpcslurm

Answer 1

这就是我使用SLURM调度程序在集群上设置的方式

1. slurm sbatch作业提交脚本

   #!/bin/bash
   
   #SBATCH --partition=xxx             ### Partition (like a queue in PBS)
   #SBATCH --job-name=array_example    ### Job Name
   #SBATCH -o jarray.%j.%N.out         ### File in which to store job output/error
   #SBATCH --time=00-00:30:00          ### Wall clock time limit in Days-HH:MM:SS
   #SBATCH --nodes=1                   ### Node count required for the job
   #SBATCH --ntasks=1                  ### Nuber of tasks to be launched per Node
   #SBATCH --cpus-per-task=2           ### Number of threads per task (OMP threads)
   #SBATCH --mail-type=FAIL            ### When to send mail
   #SBATCH --mail-user=[e-mail]
   #SBATCH --get-user-env              ### Import your user environment setup
   #SBATCH --requeue                   ### On failure, requeue for another try
   #SBATCH --verbose                   ### Increase informational messages
   #SBATCH --array=1-500%50            ### Array index | %50: number of simultaneously tasks
   
   echo
   echo "****************************************************************************"
   echo "*                                                                          *"
   echo "********************** sbatch script for array job *************************"
   echo "*                                                                          *"
   echo "****************************************************************************"
   echo
   
   current_dir=${PWD##*/}
   echo "Current dir: $current_dir"
   echo
   pwd
   echo
   
   # First we ensure a clean running environment:
   module purge
   
   # Load R
   module load R/R-3.5.0
   
   ### Initialization
   # Get Array ID
   i=${SLURM_ARRAY_TASK_ID}
   
   # Output file
   outFile="output_parameter_${i}.txt"
   
   # Pass line #i to a R script 
   Rscript --vanilla my_R_script.R ${i} ${outFile}
   
   echo
   echo '******************** FINISHED ***********************'
   echo
   

2. my_R_script.Rarg从sbatch脚本中获取

   args <- commandArgs(trailingOnly = TRUE)
   str(args)
   cat(args, sep = "\n")
   
   # test if there is at least one argument: if not, return an error
   if (length(args) == 0) {
     stop("At least one argument must be supplied (input file).\n", call. = FALSE)
   } else if (length(args) == 1) {
     # default output file
     args[2] = "out.txt"
   }
   
   cat("\n")
   print("Hello World !!!")
   
   cat("\n")
   print(paste0("i = ", as.numeric(args[1])))
   print(paste0("outFile = ", args[2]))
   
   ### Parallel:
   # https://hpc.nih.gov/apps/R.html
   # https://github.com/tobigithub/R-parallel/blob/gh-pages/R/code-setups/Install-doSNOW-parallel-DeLuxe.R
   
   # load doSnow and (parallel for CPU info) library
   library(doSNOW)
   library(parallel)   
   
   detectBatchCPUs <- function() { 
       ncores <- as.integer(Sys.getenv("SLURM_CPUS_PER_TASK")) 
       if (is.na(ncores)) { 
           ncores <- as.integer(Sys.getenv("SLURM_JOB_CPUS_PER_NODE")) 
       } 
       if (is.na(ncores)) { 
           return(2) # default
       } 
       return(ncores) 
   }
   
   ncpus <- detectBatchCPUs() 
   # or ncpus <- future::availableCores()
   cat(ncpus, " cores detected.")
   
   cluster = makeCluster(ncpus)
   
   # register the cluster
   registerDoSNOW(cluster)
   
   # get info
   getDoParWorkers(); getDoParName();
   
   ##### insert parallel computation here #####
   
   # stop cluster and remove clients
   stopCluster(cluster); print("Cluster stopped.")
   
   # insert serial backend, otherwise error in repetitive tasks
   registerDoSEQ()
   
   # clean up a bit.
   invisible(gc); remove(ncpus); remove(cluster); 
   
   # END
   

PS：如果要逐行读取参数文件，请在脚本中包含以下行，sbatch然后将它们传递给my_R_script.R

    ### Parameter file to read 
    parameter_file="parameter_file.txt"
    echo "Parameter file: ${parameter_file}"
    echo

    # Read line #i from the parameter file
    PARAMETERS=$(sed "${i}q;d" ${parameter_file})
    echo "Parameters are: ${PARAMETERS}"
    echo

参考：

• http://tuxette.nathalievilla.org/?p=1696
• https://hpc.nih.gov/apps/R.html
• https://github.com/tobigithub/R-parallel/blob/gh-pages/R/code-setups/Install-doSNOW-parallel-DeLuxe.R