| Entrambe le parti precedenti la revisioneRevisione precedente | |
| oph:cluster:archiver [2023/10/18 07:53] – versione precedente ripristinata (2023/10/17 16:18) carlo.cintolesi@unibo.it | oph:cluster:archiver [2023/10/18 07:58] (versione attuale) – versione precedente ripristinata (2023/10/18 08:43) carlo.cintolesi@unibo.it |
|---|
| ====== Archive data ====== | ====== Archive data ====== |
| |
| Archiving data not in current use is mandatory to not compromise the entire cluster's functionality. The procedure to archive data folders is described below and it is the responsibility of each cluster user to ensure the smooth operation of the system | Archiving data not currently in use is mandatory to avoid compromising the cluster's functionality. The procedure to archive data folders is described below and it is the responsibility of each user to ensure the smooth operation of the system. |
| |
| ==== Which data ==== | ==== Which data ==== |
| |
| Archiving concerns data folders not in current use, i.e. data that you want to keep in memory but do not plan to use in the time horizon of one month. These must be compressed and stored in the memory partition ''/scratch''. | Archiving concerns data folders not currently in use, i.e. data that you want to keep in memory but do not plan to use 'soon' (like in one month, not one year). These must be compressed and stored in ''[[oph:cluster:storage#scratch|/scratch]]'' partition. |
| |
| ==== Why mandatory ==== | ==== Why mandatory ==== |
| |
| One mast compresses data folders creating a ''.tgz'' archives since the current cluster architecture handles a high number of files with great difficulty, even if they occupy a few Mb of memory. This means that it is preferable to have a single compressed folder instead of a normal folder with 1,000 small files in it. | One must compresses data folders creating ''.tgz'' archives since the current cluster architecture handles a high number of files with great difficulty, even if they occupy a few Mb of space. This means that it is preferable to have a single compressed folder instead of a normal folder with 1,000 small files in it. |
| |
| ==== How to do ==== | ==== How to do ==== |
| == Practical guide to using ''archiver'' == | == Practical guide to using ''archiver'' == |
| |
| - Create a folder in ''/scratch/yourGroup/name.surnameNumber'' (**if it is not already there!**), where //yourGroup// is the research group of affiliation and //name.surnameNumber// is the beginning of your institutional e-mail. <WRAP center round box 100%> | - Create a folder in ''/scratch/yourGroup/name.surnameNumber'' (**if it is not already there!**), where //yourGroup// is the research group of affiliation and //name.surnameNumber// is the beginning of your institutional e-mail.<code> |
| ''mkdir /scratch/yourGroup/name.surnameNumber'' | $ mkdir /scratch/yourGroup/name.surnameNumber</code> |
| </WRAP> | - Load the cluster utilities by adding ''module load utils'' to the ''.bashrc'' file in your home folder (**do this just once**).<code> |
| | cd |
| - Load the cluster utilities by adding ''module load utils'' to the ''.bashrc'' file in your home folder (**do this just once**). <WRAP center round box 100%> | echo "module load utils" >> .bashrc</code> |
| ''cd; echo "module load utils" >> .bashrc'' | - Use archiver to archive and move individual folders in one shot. You need to specify (first argument) the path to where to store the compressed folder and (second argument) the path to the original uncompressed folder.<code> |
| </WRAP> | archiver /scratch/yourGroup/name.surnameNumber/foldername.tgz /path-to-folder-to-be-archived/folderName</code> |
| - Use archiver to archive and move individual folders in one shot. You need to specify (first argument) the path to where to store the compressed folder and (second argument) the path to the original uncompressed folder. <WRAP center round box 100%> | |
| ''archiver /scratch/yourGroup/name.surnameNumber/foldername.tgz /path-to-folder-to-be-archived/folderName'' | |
| </WRAP> | |
| |
| == Important notes: == | == Important notes: == |
| |
| Archiver generates a log file in the folder where it was launched that informs us of the progress of compression and does a check at the end to verify that the folder was compressed without errors. The utility runs on bladeRunner in the background. It is advisable to store folders that are not too large in order to be manageable (easy to move and decompress if necessary). | Archiver generates a log file in the folder where it was launched that informs you of the progress of compression and does a check at the end to verify that the folder was compressed without errors. The utility runs on bladeRunner in the background. It is advisable to store folders that are not too large in order to be manageable (easy to move and decompress if necessary). |
| |
