Wiki DIFA

Strumenti Utente

Strumenti Sito

Ti trovi qui: start » oph » cluster » jobs
oph:cluster:jobs

Differenze

Queste sono le differenze tra la revisione selezionata e la versione attuale della pagina.

Link a questa pagina di confronto

Entrambe le parti precedenti la revisioneRevisione precedente
Prossima revisione		Revisione precedente
oph:cluster:jobs [2023/04/12 09:19] – [Submission via script] carlo.cintolesi@unibo.itoph:cluster:jobs [2024/11/18 09:08] (versione attuale) – diego.zuccato@unibo.it
Linea 1: Linea 1:
-====== The Frontend ======+====== Frontend ======
  
 The Frontend is the node you connect to remotely. Its primary function is to allow remote access to the calculation clusters by all users and (in limited circumstances) to edit and compile source codes. It must never be used to execute resource-intensive codes, as these will slow down the work of other users and leads to loss of cluster functionality and eventually lead to the blocking of the entire infrastructure.  The Frontend is the node you connect to remotely. Its primary function is to allow remote access to the calculation clusters by all users and (in limited circumstances) to edit and compile source codes. It must never be used to execute resource-intensive codes, as these will slow down the work of other users and leads to loss of cluster functionality and eventually lead to the blocking of the entire infrastructure. 
  
 +<WRAP center round important 60%>
 If an executable must necessarily be tested on the Frontend, the responsible user must actively monitor the job and be sure that it is not active for more than a few seconds. If an executable must necessarily be tested on the Frontend, the responsible user must actively monitor the job and be sure that it is not active for more than a few seconds.
 +</WRAP>
  
 +That includes heavy IDEs((Integrated Development Environments)) (VScode, just to cite a name). If you're used to an IDE, use it on your client and just transfer the resulting files to the frontend. If it's worth using, it supports this workflow.
  
-====== Job Management ======+To better enforce the fair use of the frontend, the memory (RAM) usage is limited to 1GB per user. 
 + 
 + 
 +====== Run a Job ======
  
 To execute serial or parallel code, it is necessary to use the [[https://slurm.schedmd.com/documentation.html|Slurm WorkLoad Manager]], which will allocate the necessary resources and manage the priority of requests. Below are some of the basic functions and operating instructions for submitting serial and parallel execution (job) via Slurm; please refer to the official documentation for further information. To execute serial or parallel code, it is necessary to use the [[https://slurm.schedmd.com/documentation.html|Slurm WorkLoad Manager]], which will allocate the necessary resources and manage the priority of requests. Below are some of the basic functions and operating instructions for submitting serial and parallel execution (job) via Slurm; please refer to the official documentation for further information.
  
-For each job, it is necessary to specify via a batch script the required resources (e.g. number of nodes, number of processors, memory, execution time) and, optionally, any other constraints (e.g. group of nodes). Optionally, other parameters may also be indicated +For each job, it is necessary to specify via a batch script the required resources (e.g. number of nodes, number of processors, memory, execution time) and, optionally, any other constraints (e.g. a group of nodes). Optionally, other parameters may also be indicated 
  
  
 ===== Submission via script ===== ===== Submission via script =====
  
-Although it is possible to provide job submission information to the WorkLoad Manager via command line parameters, it is normally preferred to create a bash script (job script) that contains the information permanently. +Although it is possible to provide job submission information to the WorkLoad Manager via command line parameters, it is usually preferred to create a bash script (job script) that contains the information permanently. 
  
 The job script is ideally divided into three sections: The job script is ideally divided into three sections:
-  * The header, consisting of commented text in which information and notes useful to the user but ignored by the system are given (the syntax of the comments is #text-for-user...);  +  * The header, consisting of commented text in which information and notes useful to the user but ignored by the system are given (the syntax of the comments is ''#text-for-user...'');  
-  * The Slurm settings, in which instructions for launching the actual job are specified (the syntax of the instructions is #SLURM --option);+  * The Slurm settings, in which instructions for launching the actual job are specified (the syntax of the instructions is ''#SLURM --option'');
   * The module loading and code execution, the structure of which varies according to the particular software each user is using.   * The module loading and code execution, the structure of which varies according to the particular software each user is using.
  
-Below is an example job script ([[https://liveunibo.sharepoint.com/:u:/s/HPCClusterbetatesters/EecW_OOm3_NCle2VM3lZtZgBUSj6IbAxR_Hmh0Faf3quCQ?e=whnblQ|runParalle.sh]]) for parallel computing:+Below is an example job script for parallel computing: 
 +<code bash runParallel.sh> 
 +#!/bin/bash 
 +#---------------------------------------------------------------------------- # 
 +#   University    |   DIFA - Dept of Physics and Astrophysics  
 +#       of        |   Open Physics Hub 
 +#    Bologna      |   (https://site.unibo.it/openphysicshub/en) 
 +#---------------------------------------------------------------------------- 
 +# 
 +# License 
 +#    This is free software: you can redistribute it and/or modify it 
 +#    under the terms of the GNU General Public License as published by 
 +#    the Free Software Foundation, either version 3 of the License, or 
 +#    (at your option) any later version. 
 +# 
 +# Author 
 +#   Carlo Cintolesi 
 +# 
 +# Application 
 +#   slurm workload manager 
 +# 
 +# Usage 
 +#   run a job:         sbatch run.sh 
 +#   check processes:   slurmtop 
 +#   delete a job:      scancel <jobID>    
 +# 
 +# Description 
 +#   Run job on the new cluster of OPH with SLURM 
 +# 
 +# --------------------------------------------------------------------------- # 
 +# SLURM setup 
 +# --------------------------------------------------------------------------- #
  
-  #!/bin/bash +#- (1) [optional] Choose the account of your research group 
-  #---------------------------------------------------------------------------- # +##SBATCH --account=oph         ## This job must be "billed" to OPH project 
-  #   University    |   DIFA - Dept of Physics and Astrophysics  +##SBATCH --reservation=prj-can ## Use the node reserved for CAN project 
-  #       of        |   Open Physics Hub +##SBATCH --qos=normal          ## Also available 'debug' (max 15', no billing)  
-  #    Bologna      |   (https://site.unibo.it/openphysicshub/en) +                               ## and 'long' (max 72h, low priority) 
-  #---------------------------------------------------------------------------- + 
-  # +#- (2) Select the subcluster partition to work on (optional), 
-  # License +#  the number of tasks to be used (or specify the number of nodes and tasks), 
-  #    This is free software: you can redistribute it and/or modify it +#  and the RAM memory available for each node 
-  #    under the terms of the GNU General Public License as published by +#- 
-  #    the Free Software Foundation, either version 3 of the License, or +#SBATCH --constraint=matrix  ## run on matrix subcluster (parallel computing) 
-  #    (at your option) any later version. +##SBATCH --constraint=blade  ## run on blade subcluster (pre/post-processing) 
-  # +#SBATCH --ntasks=56          ## total number of tasks 
-  # Author +##SBATCH --nodes=2           ## number of nodes to be allocated 
-  #   Carlo Cintolesi +##SBATCH --tasks-per-node=28 ## number of tasks per node (multiple of 28) 
-  # +#SBATCH --mem-per-cpu=2G     ## ram per cpu (to be tuned) 
-  # Application + 
-  #   slurm workload manager +#- (3) Set the name of the job, the log and error files, 
-  # +#  define the email address for communications (just UniBo) 
-  # Usage +#- 
-  #   run a job:         sbatch run.sh +#SBATCH --job-name="jobName" ## job name in the scheduler 
-  #   check processes:   slurmtop +#SBATCH --output=%N_%j.out   ## log file  
-  #   delate a job:      scancel <jobNumber>    +#SBATCH --error=%N_%j.err    ## err file 
-  # +#SBATCH --mail-type=ALL      ## send a message when the job start and end 
-  # Description +#SBATCH --mail-user=nome.cognome@unibo.it  ## email address for messages 
-  #   Run job on the new cluster of OPH with SLURM + 
-  # +# --------------------------------------------------------------------------- # 
-  # --------------------------------------------------------------------------- # +# Modules setup and applications run 
-  # SLURM setup +# --------------------------------------------------------------------------- # 
-  # --------------------------------------------------------------------------- # + 
-   +#- (4) Modules to be load  
-  #- (1) Choose the partition where launch the job, +#- 
-  #  and the account of your research group  +module load mpi/openmpi/4.1.4 
-  #- + 
-  ##SBATCH --partition=g1      ## GPU node +#- (5) Run the job: just an example. 
-  #SBATCH --partition=m5       ## Matrix node 23,24,25+#  Note that the number of processors "-np 56" must be equal to --ntasks=56 
 +#- 
 +mpirun -np 56 ./executable <params>
      
-  #- (2) Select the nodes to work on (discouraged in Matrix), +# ------------------------------------------------------------------------end # 
-  #  the number of tasks to be used (or specify the number of node and tasks), +</code>
-  #  the Infiniband constraint (encoraged in Matrix) +
-  #  the RAM memory available for each node +
-  #- +
-  #SBATCH --constraint=ib      ## infiniband, keep for all matrix node +
-  #SBATCH --ntasks=56          ## number of processors +
-  #SBATCH --mem-per-cpu=2G     ## ram per cpu (to be tuned) +
-   +
-  #- (3) Set the name of the job, the log and error files, +
-  #  define the email address for comunications (just UniBo) +
-  #- +
-  #SBATCH --job-name="jobName" ## job name in the scheduler +
-  #SBATCH --output=infoRun%j   ## log file  +
-  #SBATCH --error=err%j        ## err file +
-  #SBATCH --mail-type=ALL      ## mail to send communications +
-  #SBATCH --mail-user=nome.cognome@unibo.it +
-   +
-  # --------------------------------------------------------------------------- # +
-  # Modules setup and applications run +
-  # --------------------------------------------------------------------------- # +
-   +
-  #- (4) Modules to be load  +
-  #- +
-  ADD MODULES YOU NEED +
-   +
-  #- (5) Run the job: just an example +
-  #- +
-  mpirun --prefix $MPI_HOME -n 2 --mcapmlucx-x UCX_NET_DEVICES=mlx5_0:1 ./SCRIPT +
-   +
-  # ------------------------------------------------------------------------end #+
  
 It is possible to use several job steps (several lines that launch executables such as ''mpirun'') in a single job script if each step requires the same resource allocation as the previous one and must start when the previous one has finished. If, on the other hand, the steps are independent or sequentially dependent on different resource requests, then it is better to use separate job scripts: the execution of the job steps takes place sequentially within a single resource allocation (e.g. in a single subset of nodes), while different jobs can have different allocations (thus reducing resource wastage) and also start in parallel. It is possible to use several job steps (several lines that launch executables such as ''mpirun'') in a single job script if each step requires the same resource allocation as the previous one and must start when the previous one has finished. If, on the other hand, the steps are independent or sequentially dependent on different resource requests, then it is better to use separate job scripts: the execution of the job steps takes place sequentially within a single resource allocation (e.g. in a single subset of nodes), while different jobs can have different allocations (thus reducing resource wastage) and also start in parallel.
  
-To send the resource request defined by this job script to the WorkLoad Manager, the command must be executed:+To allocate the resource request in the job script by the WorkLoad Manager, the command must be executed: 
 + 
 +  sbatch --time hh:mm:ss runParallel.sh [other parameters] 
 + 
 +<WRAP center round info> 
 +Estimating the value to use for ''--time'' is possibly the hardest part of the request. Please **do not** always use the maximum allowed time. Using a shorter estimate usually means your job will run before others that are requesting the maximum (backfill scheduling). 
 +</WRAP> 
 +<WRAP center round tip> 
 +''--nodes'' can also be a range. 
 + 
 +While ''--nodes=2 --ntasks=56'' **always** asks for 2 nodes even if the job would run on a single 112-vCPUs node (leading to longer queue times), ''--nodes=1-4 --ntasks=56'' would happily use the bigger node, if available, or up to 4 half-nodes from mtx[00-15]. 
 +</WRAP> 
 + 
 +For the management of running jobs, please refer to section "Job Management". 
 + 
 + 
 +===== 'Interactive' jobs ===== 
 + 
 +Sometimes you have to run some heavy tasks (unsuitable for the frontend) that require interactivity. For example to compile a complex program that requires you to answer some questions, or to create a container. 
 + 
 +You have to first request a node allocation, either by sbatch (as above, possibly with 'dummy' payload, like a ''sleep 7200'' for a 2h duration) or by: 
 +  salloc -N 1 --cpus-per-task=... --time=... --mem=... --constraint=blade 
 +salloc will pause while waiting for the requested resources, so be prepared. It also tells you the value for $JOBID to be used in the following steps. 
 + 
 +Then you can connect your terminal to the running job via: 
 +  srun --pty --overlap --jobid $JOBID bash 
 +that gives you a new shell on the first allocated node for $JOBID (just like SSH-ing a node with the resources you asked for). 
 + 
 +Once you're done, remember to call: 
 +  scancel $JOBID 
 +to release resources for other users. 
 + 
 +===== Job Management =====
  
-  sbatch runParallel.sh [other parameters]+Once a job has been sent to the WorkLoad Manager via the command ''sbatch'' command, it is possible to monitor the priority and progress status of the job with a series of management functions:
  
-For the management of running jobs, please refer to section [[oph:cluster:jobs|The Frontend]]+  * ''/home/software/utils/slurmtop'', displays the status of the cluster in a 'semigraphic' fashion. Among other features, it displays the status of jobs and the allocation of jobs to nodes.  
 +  * ''squeue'', displays queue status 
 +  * ''scancel <job-ID>'', cancels the execution of a job with a given identification number (ID)  
 +  * ''scontrol show job <job-ID>'', displays the details of a job, including the queued priority
  
-==== Job Management ====+Other management functions for the job and the accounting issue include the following: 
  
 +  * ''/home/software/utils/seff <job-ID>'', informs how efficiently the required resources have been utilised by an **already completed job**.
 +  * ''sshare'' and ''/home/software/utils/showfullusage.sh'', informs on how many resources have already been used and by which user.
oph/cluster/jobs.1681291176.txt.gz · Ultima modifica: 2023/04/12 09:19 da carlo.cintolesi@unibo.it
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki