oph:cluster:storage
Questa è una vecchia versione del documento!
Storage types
The DIFA-OPH cluster offers different storage areas with very different characteristics and limitations.
$HOME
- This is a storage area available from every node. Should be used for source files, compiling code, jobs that do not need a lot of space.
- NFS-mount via Ethernet (1Gbps): not very fast but quite responsive.
- Quota limit: 50GB (soft) / 100GB (hard)
/home/web
- This share is web-accessible (read-only) at
https://apps.difa.unibo.it/files/people/Str957-cluster
. Web access is read-only and it is not possible to create dynamic pages.
/home/work
- Used as work area for jobs that do not need very big datasets or need to have lots of files in a single directory (not recommended, might degrade performances for all users!).
/scratch
- Main archive area. To be used for large files, big datasets and archives.
- IMPORTANT: DO NOT USE FOR LOTS OF SMALL FILES!!!:
/scratch
is a clustered filesystem that on one hand allows to store big files and to scale out capacity by adding more servers, but on the other hand introduces quite a lot of overhead (it must contact 3 servers to determine where to get a file from, for example) that won't get amortized with small files. Way worse are directory lookups, that nearly crawl to a halt when a directory contains more that about a thousand files. If you need some big files (say a dataset) and your codes do not allow for specifying a different path, just use symlinks from$HOME
to the files in/scratch
.
- Quota: per-sector (see
df -kh /scratch
), with extra space allocated for specific projects or bought by individuals.
oph/cluster/storage.1683191293.txt.gz · Ultima modifica: 2023/05/04 09:08 da diego.zuccato@unibo.it