HPC and Big Data Technologies for Global Challenges
2021-02-17
Sergiy Gogolenko
Introduction to Spack for HiDALGO software developers
HiDALGOEU founded project
ℵ824115
@OITTTIO


Introduction to Spack for HiDALGO software developers
Use with HiDALGO HPC infrastructure

Sergiy Gogolenko

HiDALGO call∘2021-02-17

Why to use it at the 1st place?

  • [ALL ] make it possible to install off-line on bare metal with a full control on the installation process
  • [ALL ] consistent build customization for each platform
  • [ALL ] reproducible software environments for all use cases over all platforms
    • [T5.4] way to reproducible science with lock-files
    • [T5.4] installation matrices for benchmarks and performance studies
      • effect of compilers & different configurations
  • [T6.4] provides almost for granted such features as:
    • [WP5,T6.4] containerization of environments (both Docker & Singularity) and
    • [WP5,T6.4] GitLabCI-pipelines
    • [T6.4] documentation for the installation process

Why more than PyPI and conda?

  • advantages over PyPI:
    • full support packages with non-Python dependencies
      • compile non-Python dependencies
      • can build cythonized versions of a package
      • can link to an optimized libraries (e.g., MKL in case of BLAS/LAPACK)
  • advantage over conda:
    • ability to choose a specific compiler
    • can link to an specific libraries (BLAS/LAPACK, MPI,…)
    • better platform support for supercomputers (builds optimized binaries for specific microarchitectures)
  • disadvantages of Spack:
    • PyPI: incredible amount of packages that are not yet in Spack
    • conda: Windows support far better than in Spack

How it works

Key mechanisms of Spack:

Key parts of Spack:

  • Specs: expressions for describing builds of software, and
  • Packages: Python codes describing how to build software according to a spec.

Making specs more specific

Anatomy of specs

  • package name 

    spack list pyth*

  • version 

    spack versions python # check for particular versions before requesting them

  • compiler 

    spack compilers # or spack compiler list
spack compiler info clang # to know better what is it
  • flags (for compilers & linkers)

  • boolean variants and dependencies 

    spack info python
  • hash (concretize or find -l)

Specify platform

  • list platforms defined in spack 

    spack arch --known-targets

  • check current platform 

    spack arch -f # frontend
spack arch -b # backend

  • check current platform: parts of specification 

    spack arch -b -t # only the target
spack arch -b -p # only platform
spack arch -b -o # only the operating system

Specify platform: Use your knowledge

  • check core michroarchitecture from HiDALGO table
  Hawk Vulcan Eagle
Nodes 5632 96 1087
CPU model EPYC 7742 Xeon 6248 E5-2697V3
Cores 2 x 64 x zen2 2 x 20 x cascadelake 2 x 14 x haswell
RAM per node 256GB 384GB 256 GB
Interconnect IB HDR200 IB HDR IB FDR56
OS CentOS 8.2 CentOS 7.9 CentOS 7.6
MPI backend MPT 2.23 OpenMPI 4.0.5 OpenMPI 4.0.1
Interpreter Python 3.8.3 Python 3.6.6 Python 3.6.6

Going off-line

Deploying spack

  • download your favorite release or clone spack and patch (if necessary) 

    curl https://github.com/spack/spack/releases/download/v0.16.0/spack-0.16.0.tar.gz |
    ssh hawk 'cat > ~/spack.tar.gz'
wget https://github.com/spack/spack/releases/download/v0.16.0/spack-0.16.0.tar.gz &&
    scp ./spack-0.16.0.tar.gz hawk:~/spack.tar.gz && rm -rf ./spack-0.16.0.tar.gz

  • allocate disk space for your experiments at remote system and unpack there 

    ws_allocate -d 30 spack2
export SPACK_ROOT=$(ws_find spack2)/spack-0.16
tar xvf ~/spack.tar.gz -C $(dirname $SPACK_ROOT) && rm ~/spack.tar.gz

  • activate 

    . $SPACK_ROOT/share/spack/setup-env.sh

Mirrors

Mirrors: Analyse environment

  • find deps 

    spack concretize -f

spack find -c --no-groups | grep @ # may miss external packages
spack find -c -d --no-groups | grep @ | tr -d " " | sort | uniq

Mirrors: Create mirror at local workstation

  • create mirror for a subset of specs at your local PC 

    spack mirror create # mirror everything
spack mirror create -f specs.txt # mirror specs from file
spack mirror create -n10 -d ~/hidalgo_boost boost # get 10 safe versions of boost (including develop)
    • note: downloads safe versions only

    • minimalistic mirror for the active environment 

      spack mirror create -n 1 -D -d ~/hidalgo_mirror-$(date +%Y%m%d) \
  $(spack find -c -d --no-groups | grep @ | tr -d " " | sort | uniq)

      ==> Skipping cmake@3.5.1%gcc@5.4.0~doc+ncurses+openssl+ownlibs~qt \
 arch=linux-ubuntu16.04-broadwell/begrqoj as it is an external spec.

      • dirty workaround

        • add this line to environment config
        packages:: {}
        • don't forget to call spack concretize -f

Mirrors: Effect of packages:: {}

spack graph -d libosrm | dot -Tpdf | zathura - 
spack graph -d libosrm | dot -Tsvg -o/tmp/spack.stdin.svg && eog /tmp/spack.stdin.svg
spack graph -i -d | dot -Tsvg -o./full_installation.svg # all installed (not only in env)

Sorry, your browser does not support SVG.

Sorry, your browser does not support SVG.

Mirrors: Copy and register mirror

  • copy mirror to the remote system 

    scp -r $HOME/hidalgo_mirror-$(date +%Y%m%d) \
    hawk:$(ssh hawk '/opt/hlrs/non-spack/system/ws/1.3.1/bin/ws_find spack')/hidalgo_mirror

  • register mirror 

    spack mirror add hidalgo_mirror file://$(ws_find spack)/hidalgo_mirror-$(date +%Y%m%d)

  • check environment config 

    spack config get

    mirrors:
  hidalgo_mirror: file:///lustre/cray/ws9/2/ws/username-spack/hidalgo_mirror-20210211

Mirrors: Removing ineligible mirrors

  • list existing mirrors 

    spack mirror list

    hidalgo_mirror    file:///lustre/cray/ws9/2/ws/username-spack/hidalgo_mirror-20210211
spack-public      https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/

  • hard remove of undesirable/ineligible mirrors 

    spack mirror remove spack-public

    ==> Error: No mirror with name spack-public

    • workaround: define scope 

      less $SPACK_ROOT/etc/spack/defaults/mirrors.yaml
spack mirror remove --scope defaults spack-public

  • safer to fix environment config 

    mirrors::
  hidalgo_mirror: file:///lustre/cray/ws9/2/ws/username-spack/hidalgo_mirror-20210211

Repos

  • copy your specific repos to the remote host 

    scp -r $HOME/dev/hidalgo-spack-repo hawk:~/hidalgo-spack-repo
  • register with Spack

Summary

  • create mirror for the environment with the software versions of choice and externals turned off
  • copy mirror and register (within environment or scope of choice)
  • copy and register repos with packages if necessary
  • there are other more hacky ways too use packages off-line

Reuse of installed packages

Upstreams

  • default location of spack installations at Hawk 

    ll /opt/hlrs/spack/current
# lrwxrwxrwx 1 hpcoft28 hpc43203 18 Jul  3  2020 /opt/hlrs/spack/current -> rev-004_2020-06-17

  • more installations 

    ll /opt/hlrs/spack/rev*
# rev-008_2020-10-03

  • register it as an upstream 

    upstreams:
  spack-hawk-new:
    install_tree:
      /opt/hlrs/spack/rev-008_2020-10-03
  spack-hawk-default:
    install_tree:
      /opt/hlrs/spack/current

  • check packages compiled for the given microarchitecture 

    spack find target=zen2

Externals

spack external find --not-buildable

  • limited to finding a small subset of build-only dependencies (e.g., Python/Lua, etc are not there) 

    spack external list
  • spack does not
    • collects and examines beyond executable files
    • search through module files
    • overwrite existing entries in the package configuration

Externals: steal configs

  • tempting solution: on Hawk, reuse compilers.yaml and packages.yaml from upstreamed spack 

    include:
- /opt/hlrs/non-spack/misc/spack/spack-hlrs/spack-config/compilers.yaml
- /opt/hlrs/non-spack/misc/spack/spack-hlrs/spack-config/packages.yaml

  • issue 

    ==> Warning: detected deprecated properties in $HOME/hidalgo_configs/packages.yaml
Activate the debug flag to have more information on the deprecated parts or run:

	$ spack config update packages

to update the file to the new format

  • try to fix packages.yaml automatically 

    spack config update packages

    ==> Warning: included configuration files *should be updated manually* \
 [files=$HOME/hidalgo_configs/compilers.yaml, $HOME/hidalgo_configs/packages.yaml]

Externals: fix configs manually?

  • deprecated syntax from /opt/hlrs/non-spack/misc/spack/spack-hlrs/spack-config/packages.yaml 

    openmpi:
  buildable: False
  modules:
    openmpi@4.0.5: openmpi/4.0.5
    openmpi@4.0.4: openmpi/4.0.4

  • correct syntax 

    spack config blame packages

    ~/hidalgo_configs/packages.yaml:34      openmpi:
~/hidalgo_configs/packages.yaml:35        buildable: False
~/hidalgo_configs/packages.yaml:35        externals:
~/hidalgo_configs/packages.yaml:35        - spec: openmpi@4.0.5
~/hidalgo_configs/packages.yaml:35          modules:
~/hidalgo_configs/packages.yaml:35          - openmpi/4.0.5
~/hidalgo_configs/packages.yaml:35        - spec: openmpi@4.0.4
~/hidalgo_configs/packages.yaml:35          modules:
~/hidalgo_configs/packages.yaml:35          - openmpi/4.0.4

Externals: fix configs manually?

  • one-liner 

    spack config blame config | python -c \
    "from sys import stdin, stdout; i=stdin.readline().rfind(' '); map(lambda _: stdout.write(_[i:]), stdin)"

  • more careful script with filtering configs for the given file 

    import re,sys
from fileinput import FileInput
re_spack_blame=re.compile(r'(?P<fileneme>.+):(?P<lineno>[0-9]+)\w+')
with FileInput() as input:
    i=stdin.readline().rfind(' ')
    for line in input:
	m=re_spack_blame.match(line[:i])
	if m and m.group('filename') == sys.argv[1]:
	    print(line[i:])

Externals: fix configs manually!

  • better to make MPI non-buildable at all! 

    mpi:
  buildable: False
openmpi:
  # buildable: False
  externals:
  - spec: openmpi@4.0.5
    modules: [openmpi/4.0.5]
  - spec: openmpi@4.0.4
    modules: [openmpi/4.0.4]

  • alternative short format which is also fine 

    mpi:
  buildable: False
    openmpi@4.0.5: /path/to/openmpi/4.0.5
    openmpi@4.0.4: /path/to/openmpi/4.0.4

Externals: be more concrete

  • define in specs as mush as possible 

    - spec: openmpi@4.0.5%gcc@10.2.0 arch=linux-centos8-zen2
  modules: [openmpi/4.0.5]
- spec: openmpi@4.0.4%gcc@9.2.0 arch=linux-centos8-zen2
  modules: [openmpi/4.0.4]

  • not always easy or feasible with modulefiles 

    module avail openmpi

    ---------------- Compiler-dependent modules -----------------
   openmpi/4.0.4    openmpi/4.0.5 (D)

    ls /opt/hlrs/non-spack/mpi/openmpi

    4.0.4-gcc-9.2.0     4.0.5-aocc-2.2.0  4.0.5-gcc-9.2.0     4.0.5-intel-19.1.2
4.0.4-intel-19.1.0  4.0.5-gcc-10.2.0  4.0.5-intel-19.1.0  4.0.5-intel-19.1.3
4.1.0-aocc-2.2.0  4.1.0-intel-19.1.3  4.1.0-gcc-10.2.0    hcoll
  • sometimes we have better control (on expence of extra-work) with prefix

Externals: make package discoverable

  • define executables and implement determine_version method 

    executables = [...]
@classmethod
def determine_version(cls, exe): ...
  • optionally add
    • detection of variants and custom attributes
    • extra logic to filtered the executables (e.g., architecture-specific logic)

    • spec validation logic 

      @classmethod
def determine_variants(cls, exes, version_str): ...
@classmethod
def filter_detected_exes(cls, prefix, exes_in_prefix): ...
@classmethod
def validate_detected_spec(cls, spec, extra_attributes): ...

  • OR (last resort) completely redefine the detection logic for a package 

    @classmethod
def determine_spec_details(cls, prefix, exes_in_prefix): ...

Summary: Step-Wise Instructions

  • upstream spack-installed packages
  • reuse configs carefully
  • use modules for non-spack installations
  • use external find
  • use manual specification of externals via prefix for the rest

Build customization

  • configuration scopes

    • common (lower-precedence scopes first):

      Scope Specification Description
      defaults $SPACK_ROOT/etc/spack/defaults "factory" settings
      system /etc/spack settings for this machine
      site $SPACK_ROOT/etc/spack settings for Spack instance
      user ~/.spack all instances of Spack for user
      custom options --config-scope or -C </path/to/scope> custom location
    • platform-specific: <base-scope>/<platform>) (darwin, linux,…)

Build customization: Sections

  • configuration sections 

    spack config list

    compilers mirrors repos packages modules config upstreams
Section Description
config basic configuration options
packages build customization
compilers compiler definitions
mirrors list of local and remote mirrors
repos list of repos with build instructions
upstreams installation trees of external Spack instances
modules set up for mudulefile generation 
spack -C /path/to/hawk/scope config edit packages
spack -C /path/to/hawk/scope config get upstreams
spack --insecure -C /path/to/hawk/scope -C /path/to/hidalgo/scope config blame config

More on packages section

  • permissions 

    all:
  permissions:
    read: world # read-world is default, can remove
    write: group
    group: spack
uap_simulator:
  permissions:
    read: group
    group: hid_uap

  • concretization preferences 

    packages:
  python:
    compiler: [gcc@10.2.0]
    variants: +optimization
    version: [3.6, 3.9.0, 3.7]
  all:
    compiler: [gcc@10.2.0, 'clang@10:', 'gcc', intel]
    target: [zen2]
    providers:
      mpi: [mpe, openmpi]

    all providers can be found in the defaults scope

Once again on compilers

  • excerpt from /opt/hlrs/non-spack/misc/spack/spack-hlrs/spack-config/compilers.yaml 

    - compiler:
    spec: clang@10.0.0
    paths:
      cc: /opt/hlrs/non-spack/compiler/aocc/2.2.0/bin/aocc-clang
      cxx: /opt/hlrs/non-spack/compiler/aocc/2.2.0/bin/aocc-clang++
      f77: /opt/hlrs/non-spack/compiler/aocc/2.2.0/bin/aocc-flang
      fc: /opt/hlrs/non-spack/compiler/aocc/2.2.0/bin/aocc-flang
    flags: {}
    operating_system: centos8
    target: x86_64
    modules: [aocc/2.2.0]
    environment: {}
    extra_rpaths: []

Ideally sites must provide their configs

More on Environments

Reproducing Environments

  • create environment from

    • a set of abstract specs 

      spack env create abstract spack.yaml

    • a set of all fully concretized specs 

      spack env create concrete spack.lock

  • concretize 

    spack concretize -f

Build Environments: Pitfalls on Hawk (externals)

  • sometimes fail to concretize with the "stolen" packages.yaml

    • e.g., in my case, Lua could not resolve ncurses dependency 

      ncurses:
  buildable: False
  paths:
    ncurses@6.1.20180224: /usr
  # providers: {}
  version: [6.1.20180224]
    • What's wrong: (1) old format, (2) non-buildable
    • Solution for me: remove ncurses record or make buildable
    • Conclusion:
      • a good reason to keep only build dependencies external by default
      • packages.yaml should be prepared with a good care
        • ❓❗ be precise in defining specs if feasible
        • know what to make non-buildable

Build Environments: Pitfalls on Hawk (folders)

  • try to install 

    spack install
spack install -v
spack -d install
spack -d install --no-cache --overwrite --yes-to-all

  • issue originates in config.yaml from defaults scope 

    spack config blame config

    $SPACK_ROOT/etc/spack/defaults/config.yaml:69     build_stage:
$SPACK_ROOT/etc/spack/defaults/config.yaml:69     - $spack/var/spack/stage
$SPACK_ROOT/etc/spack/defaults/config.yaml:69     - ~/.spack/stage
$SPACK_ROOT/etc/spack/defaults/config.yaml:69     # - $spack/var/spack/stage

  • Solution: move stage and other folders to another places 

    spack config --scope defaults edit config

    build_stage: [$spack/var/spack/stage]

Build Environments: Speedup build

  • parallel builds

    • package-level parallelism 

      spack install -j 32

      config:
  build_jobs: 32

    • install-level parallelism

      • several installation processes (can be cross-node)
      • only one process being allowed to install a given package at a time
      qsub -l select=2:node_type=rome:mpiprocs=4,walltime=00:20:00 -V -I
mpirun -np 8 spack install -j 32

Build Environments: dev-build as a Last Resort

  • build with verbose output 

    spack dev-build libosrm

  • configure once and iteratively change/re-build sources 

    spack dev-build -u

    • dropping in 

      spack build-env hwloc@master -- bash

Stacks

Allow to define "a set of packages we want to install across a wide range of compilers".

  • constraints resolution for matrices:
    • dependencies and variants can be used regardless of whether they apply to every package
  • instruments
    • keywords for environment definitions
      • matrix: cartesian product for lists of spec parts
      • exclude: excludes specific combinations from matrix
    • conditional definitions (e.g., when: arch.satisfies('x86_64:'))
    • view descriptors

Environment vs. BundlePackage

  • basic bundle packages: opencl-headers, fastmath
  • advanced bundles: Xsdk
  • PythonPackage bundles: py-exodus-bundler, py-jupyter
class Fastmath(BundlePackage):
    homepage = "https://fastmath-scidac.org/"
    version('latest')

    depends_on('amrex')  # default is 3 dimensions
    depends_on('chombo@3.2')
    depends_on('hypre~internal-superlu')
    depends_on('mpi')
    depends_on('arpack-ng')
    depends_on('petsc')
    depends_on('phasta')
    depends_on('pumi')
    depends_on('sundials')
    depends_on('superlu-dist')
    depends_on('trilinos')
    depends_on('zoltan')
  • bundle packages in the context of HiDALGO:
    • definition of benchmark suites and sets of miniapps (e.g., Ceed)

Summary

  • use the same Spack versions on local PC and remote system (if feasible)
  • reproduce environment with either spack.yaml (abstract) or spack.lock (concrete) depending on your needs
  • prepare scope files (e.g., packages.yaml, config.yaml) with a good care
    • aim to be precise in defining specs for externals
    • think twice before marking package as non-buildable
    • redefine paths config.yaml if have acces permission troubles (e.g., size restrictions for /tmp and $HOIME)
  • speed up installation by parallel builds if necessary
  • use stacks for installation matrices, e.g., in profound benchmarks
  • think of BundlePackage's if you are certain in frequent reuse and/or installation logis is non-trivial

Other topics

Spack as a tool for documenting installation process

  • on-line documentation for builtin packages
  • package.py is the best possible documentation
    • easily readable
    • absolut complete

  • spack info makes summary of package.py readable for everyone 

    spack info libosrm

    CMakePackage:   libosrm

Description:
    libOSRM is a high performance routing library written in C++14 designed
    to run on OpenStreetMap data.

Homepage: http://project-osrm.org/

Tags: 
    None

Preferred version:  
    5.24.0    https://github.com/Project-OSRM/osrm-backend/archive/v5.24.0.tar.gz

Safe versions:  
    5.24.0    https://github.com/Project-OSRM/osrm-backend/archive/v5.24.0.tar.gz

Variants:
    Name [Default]          Allowed values    Description
    ====================    ==============    =====================================

    build_type [Release]    Debug, Release    The build type to build
    ipo [off]               on, off           CMake interprocedural optimization
    lib_only [on]           on, off           Install OSRM in a library only mode
    osmium [off]            on, off           Install with libosmium
    shared [off]            on, off           Enables the build of shared libraries

Installation Phases:
    cmake    build    install

Build Dependencies:
    boost  bzip2  cmake  git  intel-tbb  libosmium  libxml2  libzip  lua  pkg-config

Link Dependencies:
    boost  bzip2  intel-tbb  libosmium  libxml2  libzip  lua

Run Dependencies:
    None

Virtual Packages:

Making containers from environments

  • as simple as this
spack cd -e hidalgo && spack containerize > hidalgo.def

  • base images: supports both 

    container:
  images:
    os: centos/7
    spack: 0.16

  • default format is Docker but one can put format: singularity 

    container:
  format: singularity
  • all configuration options can be found here
  • caution: avoid double-installing CUDA by using externals
  • alternative: build caches

GitLab-CI

TBA

Modules

  • modules vs spack load

    spack load --only package libosrm module load libosrm
    spack unload libosrm module unload libosrm
    spack find --loaded module av
    spack find -lp libosrm module show libosrm
    spack load --sh libosrm -//-
    spack load -r libosrm -//-
    spack info libosrm module whatis libosrm
  • Spack suports generation of modulefiles for tcl and lmod (see manual)

    • default notion for module names: package-version-compiler-version-hash 

      spack module tcl refresh libosrm

Extending spack

Further information