oph:cluster:storage
Questa è una vecchia versione del documento!
Storage types
The DIFA-OPH cluster offers different storage areas with very different characteristics and limitations.
$HOME
- This is a storage area available from every node. Should be used for source files, compiling code, jobs that do not need a lot of space.
- NFS-mount via Ethernet (1Gbps): not very fast but quite responsive.
- Quota limit: 50GB (soft) / 100GB (hard)
/home/temp
Temporary replacement for /fast (see below), useable to store work that doesn't fit $HOME.
/home/web
- This share is web-accessible (read-only) at
https://apps.difa.unibo.it/files/people/Str957-cluster
. Web access is read-only and it is not possible to create dynamic pages.
/home/work
- Used as work area for jobs that do not need very big datasets or need to have lots of files in a single directory (not recommended, might degrade performances for all users!).
/scratch
- Main archive area. To be used for large files, big datasets and archives.
- IMPORTANT: DO NOT USE FOR LOTS OF SMALL FILES!!!:
/scratch
is a clustered filesystem that on one hand allows to store big files and to scale out capacity by adding more servers, but on the other hand introduces quite a lot of overhead (it must contact 3 servers to determine where to get a file from, for example) that won't get amortized with small files. Way worse are directory lookups, that nearly crawl to a halt when a directory contains more that about a thousand files. If you need some big files (say a dataset) and your codes do not allow for specifying a different path, just use symlinks from$HOME
to the files in/scratch
.
- Quota: per-sector (see
df -kh /scratch
), with extra space allocated for specific projects or bought by individuals.
/fast
PLANNED
Parallel filesystem, for quick access to the data you're working on.
No quota, but files older than TBD1) days will be automatically deleted without further notice.
1)
Decision to be finalized by the board
oph/cluster/storage.1692953187.txt.gz · Ultima modifica: 2023/08/25 08:46 da diego.zuccato@unibo.it