Strumenti Utente

Strumenti Sito


oph:cluster:storage

Questa è una vecchia versione del documento!


Storage types

The DIFA-OPH cluster offers different storage areas with very different characteristics and limitations.

$HOME

  • This is a storage area available from every node. Should be used for source files, compiling code, jobs that do not need a lot of space.
  • NFS-mount via Ethernet (1Gbps): not very fast but quite responsive.
  • Quota limit: 50GB (soft) / 100GB (hard)

/home/temp

Temporary replacement for /fast (see below), useable to store work that doesn't fit $HOME.

/home/web

/home/work

  • Used as work area for jobs that do not need very big datasets or need to have lots of files in a single directory (not recommended, might degrade performances for all users!).

/scratch

  • Main archive area. To be used for large files, big datasets and archives.
  • IMPORTANT: DO NOT USE FOR LOTS OF SMALL FILES!!!: /scratch is a clustered filesystem that on one hand allows to store big files and to scale out capacity by adding more servers, but on the other hand introduces quite a lot of overhead (it must contact 3 servers to determine where to get a file from, for example) that won't get amortized with small files. Way worse are directory lookups, that nearly crawl to a halt when a directory contains more that about a thousand files. If you need some big files (say a dataset) and your codes do not allow for specifying a different path, just use symlinks from $HOME to the files in /scratch.
  • Quota: per-sector (see df -kh /scratch), with extra space allocated for specific projects or bought by individuals.

/fast

:!: PLANNED :!:

Parallel filesystem, for quick access to the data you're working on.

No quota, but files older than TBD1) FIXME days will be automatically deleted without further notice.

1)
Decision to be finalized by the board
oph/cluster/storage.1692953187.txt.gz · Ultima modifica: 2023/08/25 08:46 da diego.zuccato@unibo.it

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki