configuration - 当工作刚刚挂起时,我在哪里可以找到 slurm 诊断信息?
问题描述
我在具有 1 个登录节点和 3 个计算节点的系统上运行 slurm 20.11.8。我可以找到的状态信息如下。
$ slurmd -V
slurm 20.11.8
$ sinfo -N
NODELIST NODES PARTITION STATE
pauli-node-01 1 normal* idle
pauli-node-02 1 normal* idle
pauli-node-03 1 normal* idle
$ systemctl status slurmctld
● slurmctld.service - Slurm controller daemon
Loaded: loaded (/usr/lib/systemd/system/slurmctld.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2021-10-05 22:04:10 CDT; 10h ago
Main PID: 11802 (slurmctld)
Tasks: 7
Memory: 6.7M
CGroup: /system.slice/slurmctld.service
└─11802 /usr/sbin/slurmctld -D
Oct 05 22:04:10 pauli.mer.utexas.edu systemd[1]: Started Slurm controller daemon.
这是我的配置文件:
$ cat /etc/slurm/slurm.conf
# slurm.conf file generated by configurator.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
SlurmctldHost=pauli
#SlurmctldHost=
#
#DisableRootJobs=NO
#EnforcePartLimits=NO
#Epilog=
#EpilogSlurmctld=
#FirstJobId=1
#MaxJobId=999999
#GresTypes=
#GroupUpdateForce=0
#GroupUpdateTime=600
#JobFileAppend=0
#JobRequeue=1
#JobSubmitPlugins=1
#KillOnBadExit=0
#LaunchType=launch/slurm
#Licenses=foo*4,bar
#MailProg=/bin/mail
#MaxJobCount=5000
#MaxStepCount=40000
#MaxTasksPerNode=128
MpiDefault=none
#MpiParams=ports=#-#
#PluginDir=
#PlugStackConfig=
#PrivateData=jobs
ProctrackType=proctrack/cgroup
#Prolog=
#PrologFlags=
#PrologSlurmctld=
#PropagatePrioProcess=0
#PropagateResourceLimits=
#PropagateResourceLimitsExcept=
#RebootProgram=
ReturnToService=1
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/spool/slurmd
SlurmUser=slurm
#SlurmdUser=root
#SrunEpilog=
#SrunProlog=
StateSaveLocation=/var/spool/slurmctld
SwitchType=switch/none
#TaskEpilog=
TaskPlugin=task/affinity
#TaskProlog=
#TopologyPlugin=topology/tree
#TmpFS=/tmp
#TrackWCKey=no
#TreeWidth=
#UnkillableStepProgram=
#UsePAM=0
#
#
# TIMERS
#BatchStartTimeout=10
#CompleteWait=0
#EpilogMsgTime=2000
#GetEnvTimeout=2
#HealthCheckInterval=0
#HealthCheckProgram=
InactiveLimit=0
KillWait=30
#MessageTimeout=10
#ResvOverRun=0
MinJobAge=300
#OverTimeLimit=0
SlurmctldTimeout=120
SlurmdTimeout=300
#UnkillableStepTimeout=60
#VSizeFactor=0
Waittime=0
#
#
# SCHEDULING
#DefMemPerCPU=0
#MaxMemPerCPU=0
#SchedulerTimeSlice=30
SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_CPU
#
#
# JOB PRIORITY
#PriorityFlags=
#PriorityType=priority/basic
#PriorityDecayHalfLife=
#PriorityCalcPeriod=
#PriorityFavorSmall=
#PriorityMaxAge=
#PriorityUsageResetPeriod=
#PriorityWeightAge=
#PriorityWeightFairshare=
#PriorityWeightJobSize=
#PriorityWeightPartition=
#PriorityWeightQOS=
#
#
# LOGGING AND ACCOUNTING
#AccountingStorageEnforce=0
AccountingStorageHost=pauli
#AccountingStoragePass=abcdef
#AccountingStoragePort=1234
AccountingStorageType=accounting_storage/none
AccountingStorageUser=slurm
AccountingStoreJobComment=YES
ClusterName=cluster
#DebugFlags=
JobCompHost=pauli
#JobCompLoc=slurm_comp_db
#JobCompPass=abcdef
#JobCompPort=1234
JobCompType=jobcomp/none
JobCompUser=slurm
#JobContainerType=job_container/none
JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/none
SlurmctldDebug=info
#SlurmctldLogFile=
SlurmdDebug=info
#SlurmdLogFile=
#SlurmSchedLogFile=
#SlurmSchedLogLevel=
#
#
# POWER SAVE SUPPORT FOR IDLE NODES (optional)
#SuspendProgram=
#ResumeProgram=
#SuspendTimeout=
#ResumeTimeout=
#ResumeRate=
#SuspendExcNodes=
#SuspendExcParts=
#SuspendRate=
#SuspendTime=
#
#
# COMPUTE NODES
NodeName=pauli-node-0[1-2] CPUs=64 Sockets=2 CoresPerSocket=16 ThreadsPerCore=2 State=UNKNOWN
NodeName=pauli-node-03 CPUs=40 Sockets=2 CoresPerSocket=10 ThreadsPerCore=2 State=UNKNOWN
PartitionName=normal Nodes=pauli-node-0[1-3] Default=YES MaxTime=INFINITE State=UP
当我尝试使用srun
(按照Slurm:快速入门用户指南)在 1 个节点上运行一个简单的作业时,该作业只是挂起。有谁知道我应该在哪里查找诊断信息以找出工作挂起的原因?
$ srun -N1 -n1 -l hostname
解决方案
首先要检查的事情之一是网络连接并确保没有防火墙阻碍。你可以检查一下
scontrol ping
在控制节点上。此外,srun
还有一个-v
选项可以告诉您它被阻止的位置(您可以添加多个此类选项以增加详细程度)。
当然,控制器和slurmd
's 的日志文件都可能包含信息。同样,可以使用scontrol setdebug增加日志级别。
除了防火墙之外,通常的嫌疑人通常是 SELinux、网络掩码、IP 路由。还要确保时钟同步并且munge
运行正常。
推荐阅读
- python - SQLAlchemy - 到 SQL 的数据框
- python - 执行涉及 tkinter 和 python-mysql 连接器的函数时没有错误,但没有输出
- firebase - 使用自定义域的 Firebase 电子邮件链接身份验证
- animation - 为什么 bodymovin 结果动画看起来比源帧更拉伸
- python - 如何从字符串中删除前缀?
- python - 车轮和 setup.py 上的 PyAudio Windows 10 安装错误
- javascript - 是否可以在浏览器返回时使用 react-router-dom 将状态发送到页面目标?
- python - Python 之禅“显式胜于隐式”
- c++ - c++ std::lock_guard 作用域到达
- node.js - testcafe 命令失败:“npm