Differenze

Queste sono le differenze tra la revisione selezionata e la versione attuale della pagina.

--- oph:cluster:storage [2024/06/03 11:31] – Aggiornato a nuova /scratch e preparato per /archive diego.zuccato@unibo.it
+++ oph:cluster:storage [2025/02/06 16:07] (versione attuale) – [/home/] nuovo sito per /home/web diego.zuccato@unibo.it
@@ Linea 1: / Linea 1: @@
 ====== Storage types ======
-The DIFA-OPH cluster offers different storage areas with very different characteristics and limitations.
+The DIFA-OPH cluster offers three different storage areas with different features and usage policies:
-==== $HOME ====
+===== /home/ =====
-  * This is a storage area available from every node. Should be **used for source files, compiling code, jobs that do not need a lot of space**.
+This is a storage area available from every node and this is the space that you access when you connect to the cluster frontend. This is meant to store non-reproducible data (e.g. source codes) and is regularly backed up. It should be **used for source files, compiling code**, or jobs that do not need a lot of space (see ''/home/work'' below).
+The /home is the area where your home folders are stored, as well as other shared areas such as ''/home/work/'' and ''/home/web/'' that are meant to be:
-  * NFS-mount via Ethernet (1Gbps): not very fast but quite responsive.
+  * **/home/work**: Used as a work area for jobs that do not need very big datasets or need to have lots of files in a single directory (not recommended since this might degrade performances for all users!). **Per-sector quota** of 1TB (soft) / 2TB (hard).
-  * Quota limit: 50GB (soft) / 100GB (hard); check how much you're using with ''quota -u''
+  * **/home/web**:
+    * This space is web-accessible at ''https://cluster.difa.unibo.it'' (and also from old ''https://apps.difa.unibo.it/files/people/Str957-cluster'' via an automatic redirect).
+    * web access is read-only and it is not possible to create dynamic pages.
+    * **Per-sector quota** of 1TB (soft) / 2TB (hard) except astro with 4.4TB/7TB.
+    * Requires either index.html or .htaccess file with ''Options: +Indexes'' to be browseable
-==== /home/temp ====
+==== Technical characteristics ====
-  * **Temporary** replacement for /scratch :!: please migrate to /scratch
+  * NFS-mount via Ethernet (1Gbps which is not very fast but quite responsive);
+  * Quota limit to 50 GB (100GB as hard limit); check how much you're using with ''quota -u''.
-==== /home/web ====
+===== /scratch/ ====
-  * This share is **web-accessible (read-only)** at ''https://apps.difa.unibo.it/files/people/Str957-cluster'' . Web access is read-only and it is not possible to create dynamic pages.
+This is the fast Input/Output area to be used for direct read/write operations from the compute nodes. There is no quota in this area, but an automatic cleaning procedure is enforced on all files older than 40 days to avoid the disk space being exhausted, as this would make running jobs crash when trying to write outputs on disk. Therefore, once your jobs are finished you are recommended to archive the relevant data to ''/archive/'' (see below) to avoid any data loss;
-==== /home/work ====
+:!: folders inside sectors areas **must** use the account as name or you won't get important mails => possibe data loss.
+==== Technical characteristics: ====
-  * Used as **work area for jobs that do not need very big datasets** or need to have lots of files in a single directory (**not recommended**, might degrade performances for all users!).
+  * Parallel filesystem, for quick (SSD-backed) access to the data you are working on;
+  * No quota, but **files older than 40 days will be automatically deleted** without further notice;
+  * No cross-server redundancy, just local RAID. Hence, if (when) a server (or two disks in the same RAID) fails, all data becomes unavailable.
-==== /archive (PLANNED) ====
-  * :!: **PLANNED** :!: => not useable (yet)
+===== /archive/ ====
-  * Main archive area. **To be used for large files, big datasets and archives**.
-  * **Readonly access** from compute nodes, read/write only from frontends and filetransfer nodes
-  * **IMPORTANT: DO NOT USE FOR LOTS OF SMALL FILES!!!**: ''/archive'' is a clustered filesystem that on one hand allows to store big files and to scale out capacity by adding more servers, but on the other hand introduces quite a lot of overhead (it must contact 3 servers to determine where to get a file from, for example) that won't get amortized with small files. Way worse are directory lookups, that nearly crawl to a halt when a directory contains more that about a thousand files. If you need some big files (say a dataset) and your codes do not allow for specifying a different path, just **use symlinks from** ''$HOME'' **to the files in** ''/archive''.
-  * Quota: **per-sector** (see ''df -kh /archive/sector''), **with extra space allocated for specific projects** or bought by individuals (''df -kh /archive/extra/project_or_name'').
-==== /scratch ====
+This is the main archive area to be used for large files, big datasets or archives; it is designed to be a distributed storage area for long-term data preservation. Data in this area should be stored in the form of compressed folders because the presence of a large number of small files will compromise its functionality. Every sector or project has a dedicated area with an associated quota on ''/archive/'', and when the quota is exceeded no further writing is possible on the sector or project area.
-  * Parallel filesystem, for **quick** (SSD-backed) access to the data you're working on.
+:!: folders inside sectors areas **must** use the account as name or you won't get important mails => possibe data loss.
-  * No quota, but **files older than TBD((Decision to be finalized by the board)) FIXME days** will be automatically deleted without further notice.
+==== Technical characteristics: ====
-  * No cross-server redundancy, just local RAID: if (when) a server (or two disks in the same RAID) fails, all data becomes unavailable -- always keep a copy of your important data archived elsewhere (maybe /archive, but for very important data **offsite is better**)
-==== $TMPDIR ====
+  * **To not be used to store a large number of small files**, this will compromise the functionality of the storage space eventually blocking all the reading/writing operations of the entire cluster.
+  * Max size for a single file is 8TB. When archiving big datasets please split them into sub-folders, compress and store them separately (preferably each compressed folder should be less than 1TB)
+  * Read-only access from compute nodes, read/write only from frontends and filetransfer nodes
+  * Quota imposed (both on file size and number of files) per sector, with extra space allocated for specific projects (in /archive/projects) or bought by individuals (in /archive/extra).
+  * Currently ACLs (setfacl) are notsupported (cephfs exported via NFS-Ganesha does not allow to set/get ACLs)
+==== Monitoring system of space usage ====
+To allow users/sectors/projects to handle their /archive/ storage and avoid their sector going over quota, a monitoring and alerting system is now in place. Some of you may have already received alerting emails from the OPH cluster last night, around midnight. These emails are meant to inform you about your usage of the /archive/ system and alert you in case your sector/project is already over quota (i.e. using more than 100% of the allowed space or number of files, so that writing is already blocked) or close to the quota limit (i.e. using more than 90% of either the available space or number of files).
+In particular:
+   * every sector/project reference person will receive an email on the first day of each month in any case (i.e. even if their sector is below 90% of disk usage) with the overview of their sector/project disk usage
+   * every sector/project reference person will receive an email when their sector/project is using more than 90% of either disk space or number of files allowed for their quota; these emails will be sent with daily frequency until the sector/project disk usage is reduced below the 90% threshold
+   * individual users will receive an email only if their sector/project is using more than 90% of either disk space or number of files allowed for their quota; these emails will be sent with daily frequency to the users that are using (individually) more than 50% of the sector/project quota until the disk usage is reduced below the 90% threshold.
+====== $TMPDIR local node space (advanced) ======
+Every node does have some available space on local storage in ''$TMPDIR''. This can be used to store temporary files that do not need to be shared between nodes. Being a local memory, latency is very low and with a limited capacity, usually around 200GB. It it automatically cleaned when the job terminates.
+==== Technical characteristics: ====
+  * local space: not shared between multiple nodes, not even for a single multi-node job
+  * quite fast
+  * automatically cleaned when job ends
-Every node does have some available space on local storage. It's useful to store temporary files that do not need to be shared between nodes. Being local, latency is very low. But local disks aren't big. $TMPDIR can usually store around 200GB of data. It gets **automatically cleaned when the job terminates**: if you need to keep some files, copy 'em to other storages.