HPC and Big Data Technologies for Global Challenges
2021-02-03
Sergiy Gogolenko
Introduction to Spack for HiDALGO software developers
HiDALGOEU founded project
ℵ824115
@OITTTIO


Introduction to Spack for HiDALGO software developers
Basics

Sergiy Gogolenko

HiDALGO call∘2021-02-03

Installation

  • Clone sources 

    export SPACK_ROOT=~/dev/spack

mkdir -p $SPACK_ROOT
rm -rf $SPACK_ROOT ~/.spack

git clone https://github.com/spack/spack $SPACK_ROOT

  • Switch to a required version and patch it (if needed) 

    cd $SPACK_ROOT &&
    git checkout releases/v0.16
git apply ../spack-0.16-ml.patch

  • Activate 

    . $SPACK_ROOT/share/spack/setup-env.sh

  • Set up favorite editor 

    export EDITOR=emacsclient

Specs

spack install python@3.9
spack install python %clang
spack install python %gcc@6.5.0
spack install python@3 cppflags=-O3
spack install python ^/h6i

Specs: Simple Installation

  • check out available packages 

    spack list
spack list pyth*

  • installation specification 

    spack install python
    • installs python2 by default

Specs: Version

  • sigil: @

  • check out available versions 

    spack versions python # check for particular versions before requesting them

  • installation specification 

    spack install python@3
spack install python@3.9.0

Specs: Compiler

  • sigil: %

  • check out available compilers 

    spack compiler list
spack compilers

  • installation specification 

    spack install python@3%gcc
spack install python@3%clang@8.0.0

Specs: Compiler - external (2)

  • get info about compiler 

    spack compiler info clang
  • register one if absend

    • automatic 

      module load gcc
spack compiler find gcc
# ==> Found no new compilers
# ==> Compilers are defined in the following files:
#   /home/xyz/.spack/linux/compilers.yaml
    • edit config (e.g., ~/.spack/<os>/compilers.yaml)

Specs: Compiler - config entries (3)

  • open config with editor (same effect to $EDITOR ~/.spack/linux/compilers.yaml) 

    spack config edit compilers

  • example of entry (mix clang@8.0.0 with gfortran) 

    - compiler:
    spec: clang-gfortran@3.8.0
    paths:
      cc: /usr/bin/clang-3.8
      cxx: /usr/bin/clang++-3.8
      f77: /usr/bin/gfortran
      fc: /usr/bin/gfortran
    operating_system: ubuntu16.04
    flags:
      cflags: -O3 -fPIC
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []

Specs: Compiler - spack-installed (4)

  • use external compilers 

    module load gcc
spack external find gcc # not recommended since registers only package
spack compiler find gcc # recommended since registers compiler

spack compilers
spack compiler info clang

  • use spack-installed compilers 

    spack install clang@8.0.0
spack compiler add $(spack location -i clang@8.0.0)
spack compiler remove clang@8.0.0

spack config edit compilers

Specs: Flags

  • markers:
    • compilers: cppflags, cflags, cxxflags, and fflags
    • linkers: ldflags, and ldlibs

  • check out available options

    you must rely on package scripts and its variants (but can always modify them)

  • installation specification 

    spack install python@3 cppflags=-O3

Specs: Boolean Variants

What about other options and flags?

  • sigils: +, ~, -

  • check out available variants (with description) 

    spack info python

    Variants:
  Name [Default]         Allowed values    Description
  ===================    ==============    ================================================================
  debug [off]            on, off           debug build with extra checks (this is high overhead)
  optimizations [off]    on, off           Enable expensive build-time optimizations, if available
  pic [on]               on, off           Produce position-independent code (for shared libs)
...

  • installation specification 

    spack install python@3 +optimizations+pic~debug
spack install python@3 +optimizations # equivalent to former

Specs: Dependencies (1)

  • sigil: ^
    • also is used for virtual dependencies (depend on abstract interfaces) e.g., on MPI ^mpi@3, while the real package can be either MPIv3 distribution

  • check out feasible dependencies 

    spack info python

  • installation specification 

    spack install python@3 ^openssl@1.1
spack install py-h5py ^hdf5+hl+mpi ^openmpi

Specs: Dependencies (2)

  • try to play :) 

    spack spec py-h5py ^hdf5~mpi ^openmpi
spack spec py-h5py~mpi ^hdf5 ^openmpi
spack spec py-h5py~mpi ^hdf5
spack spec py-h5py~mpi ^hdf5~mpi
spack spec py-h5py~mpi ^hdf5~mpi ^openmpi

  • check full dependencies tree (with compilers) of installed package 

    spack find -df python

Specs: Hash

How to distinguish between 2 installations with the same version but different specs?

  • spack generates a unique hash for each new spec
  • sigil: /

  • installation specification with package dependency specified by hash 

    spack find -l openssl
# ext6i2u openssl@1.0.2g
# rzymlhl openssl@1.1.1g 
spack install python@3 ^/rzy

Specs: Summary

  • common sigil and options:
    • @: version
    • %: compiler
    • +, ~, -: variants
    • ^: explicitly request dependencies
    • /: referred to package by its hash
    • cppflags, cflags, cxxflags, fflags, ldflags, and ldlibs (only if really needed, e.g., to force optimizations)

  • last but not least: check resolved specs and dependencies before installation 

    spack spec python@3.9%clang@3.8+optimizations~debug ^openssl@1.1
    • it is also a way to estimate efforts for installation

Basic commands

We met already

spack spec python@3.9%clang@3.8+optimizations~debug ^openssl@1.1
spack install python@3.9%clang@3.8+optimizations~debug ^openssl@1.1
spack list python
spack list -d MPI # --search-description (also search the description)
spack list -v mpi # --virtuals (also include virtual packages like ~mpi~)

spec and graph

  • display full spec with dependencies 

    spack spec -I python@3
spack --color always spec python@3 | less -R

  • show all dependencies as a graph 

    spack graph python@3.9
spack graph --dot python@3.9 | dot

uninstall

Do not uninstall packages that are not sufficiently specified

  • -f force
  • -R remove dependents as well
  • -a multiple packages at once
spack uninstall zlib %gcc@6.5.0
spack uninstall -R zlib/pdf

find

  • -x explicit installs only
  • -X implicit installs only
  • -p show the path to which a package was installed
  • -d show dependency information
spack find ^mpich # every installed package that depends on mpich
spack find cppflags="-O3" # built with cppflags="-O3"
spack find -ldf

Summary

Externals

spack external find python
spack external list

Register detectable externals

  • check out packages detectable by external (list by repository and name) 

    spack external list

  • add external packages to packages.yaml 

    spack external find --not-buildable git
spack external find --not-buildable automake
spack external find --not-buildable autoconf
spack external find python

Register external by editing packages.yaml

  • get version, location and installation details of binutils 

    dirname $(dirname $(which ld))
ld --version
apt-cache show binutils
dpkg -l | grep binutils
dpkg -l | grep binutils
apt-cache policy binutils

  • put this information to packages.yaml 

      buildable: False
  modules:
    binutils+libiberty~nls: binutils/2.26.1
  version: [2.26.1]

binutils:
  externals:
  - spec: binutils@2.26.1+libiberty~nls
    prefix: /usr
  buildable: False

Environments

Initialization

  • Create environment 

    spack env create python3
spack env activate python3

  • Tell to use single consistent environment (all of the specs concretized together) 

    spack cd -e python3
sed -i '$a \ \ concretization: together' ./spack.yaml

Define further installations

  • Add installations 

    spack add python@3.9.0+optimizations
spack add py-numpy ^python@3+optimizations
spack add py-scipy ^python@3+optimizations
spack add py-matplotlib ^python@3+optimizations
spack add py-scikit-learn ^python@3+optimizations

  • Check 

    spack concretize

Environment config

  • view env config 

    spack config get

  • expected output 

    # This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  # add package specs to the `specs` list
  specs: [python@3.9.0+optimizations, py-numpy ^python@3.9.0+optimizations, py-scipy ^python@3.9.0+optimizations, py-matplotlib ^python@3.9.0+optimizations, py-scikit-learn ^python@3.9.0+optimizations, py-jupyter ^cairo+X+fc+ft ^python@3.9.0+optimizations]
  view: true
  concretization: together

Editing Config: Basic Improvements

  • open environment config in editor 

    spack config edit

  • 1st iteration of improvements: matrices 

    # Spack Environment file for Python3 ML stack
spack:
  specs:
  - python@3.9.0+optimizations
  - matrix:
    - [py-numpy, py-scipy, py-matplotlib, py-scikit-learn, py-jupyter ^cairo+X+fc+ft]
    - [^python@3.9.0+optimizations]
  concretization: together
  view: true

Editing Config: Further Improvements

  • 2nd iteration of improvements: definitions 

    # Spack Environment file for Python3 ML stack
spack:
  definitions:
  - packages:
    - py-pip
    - py-virtualenv
    - py-numpy
    - py-scipy
    - py-matplotlib
    - py-scikit-learn
    - py-jupyter ^cairo+X+fc+ft
    - py-keras
  - pythons: [python@3.9.0+optimizations]
  # - compilers: [gcc@8.1.0]
  specs:
  - $pythons
  - matrix:
    - [$packages]
    - [$^pythons]
    # - [$%compilers]
  concretization: together
  view: true

Installation

spack install

Combining with Python venv

  • create virtual environment and install required packages 

    virtualenv --system-site-packages ~/dev/python/envs/plmr
. ~/dev/spack/share/spack/setup-env.sh && spack env activate python3 && . ~/dev/python/envs/plmr/bin/activate
pip install altair
pip install ipykernel --upgrade

  • install kernel for the given virtual environment 

    python -m ipykernel install --user --name=hidalgo

  • deactivate environments 

    deactivate && despacktivate

Package Creation

    • where to find and how to retrieve the software
    • its dependencies
    • options for building the software from source
    • build commands
  • spack-installation scripts \(\equiv\) Python recipes including this info

Preparing environment for the experiments

  • create environment 

    spack env create hidalgo_lge
spack env activate hidalgo_lge

  • tell to use single consistent environment (all of the specs concretized together) 

    spack cd -e hidalgo_lge
sed -i '$a \ \ concretization: together' ./spack.yaml

  • create repo 

    export MY_SPACK_REPO=$(dirname $SPACK_ROOT)/my-spack-repo
spack repo create $MY_SPACK_REPO/
spack repo add $MY_SPACK_REPO/

Creating the Package File

  • fetch the code and create a package skeleton 

    spack create --force https://github.com/osmcode/libosmium/archive/v2.16.0.tar.gz
spack create --force https://github.com/Project-OSRM/osrm-backend/archive/v5.24.0.tar.gz

  • rename package if default name is not satisfactory 

    mv $MY_SPACK_REPO/packages/osrm-backend $MY_SPACK_REPO/packages/libosrm

  • edit the skeleton 

    spack edit libosrm &

Editing the Package Skeleton

  • where to find and how to retrieve the software
  • its dependencies
  • options for building the software from source
  • build commands

Package Skeleton: Where to Find and How to Retrieve SW

class Libosrm(CMakePackage): # name must correspond to package name
    """libOSRM is a high performance routing library written in C++14
    designed to run on OpenStreetMap data."""

    homepage = "http://project-osrm.org/"
    url      = "https://github.com/Project-OSRM/osrm-backend/archive/v5.24.0.tar.gz"

    maintainers = ['sgo-go']

    version('master', branch='master') #
    version('5.24.0',                   sha256='a66b20e7ffe83e5e5fe12324980320e12a6ec2b05f2befd157de5c60c665613c')
    # version('5.23.0',                   sha256='8527ce7d799123a9e9e99551936821cc0025baae6f2120dbf2fbc6332c709915')
    # version('5.22.0-customsnapping.3',  sha256='414922ec383f9cbfcb10f2ced80688359f1ee5e87b920b0d00b3d6eda9b5925b')
  • naming (module \(\mapsto\) class): foo-bar \(\mapsto\) FooBar, 3proxy \(\mapsto\) _3proxy
  • versions: besides branches, they can also be tags & commits

Package Skeleton: Options for Building from Source

variant('build_type', default='Release',
	description='The build type to build',
	values=('Debug', 'Release'))

# ---- See about library OSRM at:
# https://github.com/Project-OSRM/osrm-backend/blob/master/docs/libosrm.md
variant('lib_only', default=True,
	description='Install OSRM in a library only mode')
  • variants: can be boolean or multi-valued with validation logic

Package Skeleton: Dependencies and Conflicts

depends_on('bzip2')
depends_on('libxml2')
depends_on('libzip')
depends_on('boost@1.69.0:')
depends_on('lua@5.3.0:')
depends_on('intel-tbb')

# Build-time dependencies:
depends_on('pkg-config', type='build')
depends_on('cmake@3.1:', type='build', when='@5.21:')
depends_on('git', type='build', when='@master')
depends_on('libosmium', when='+osmium')

conflicts('%gcc', when='@:5', msg='libOSRM needs C++14 support (GCC >= 5)')
conflicts('@:5.23', when='target=aarch64:') # incompatibility with ARM64
  • dependencies: supports
    • virtual & real dependencies
    • dependency patching & intervention to dependents build process

Package Skeleton: Commands

  • example configuration 

    CXX=$CXX CC=$CC cmake .. -DCMAKE_INSTALL_PREFIX=$OSRM_INSTALL \
   -DLUA_INCLUDE_DIR=$LUA_INCLUDE_DIR \
   -DBoost_USE_STATIC_LIBS=ON \
   -DCMAKE_BUILD_TYPE=Release

    def cmake_args(self):
    variant_bool = lambda feature: str(feature in self.spec)
    cmake_args = []

    cmake_args.append('-DLUA_INCLUDE_DIR=%s' % self.spec['lua'].headers.directories[0])
    cmake_args.append('-DBUILD_SHARED_LIBS:BOOL=%s' % variant_bool('+shared'))
    cmake_args.append('-DBoost_USE_STATIC_LIBS=ON') # %s' % variant_bool('+shared')

    return cmake_args

Package Skeleton: Where to dig?

  • official documentation: packaging guide, build systems API 

    spack info libosrm

    Installation Phases:
  cmake    build    install

  • check examples in the default repo 

    grep -r $SPACK_ROOT/var/spack/repos/builtin -e "CMakePackage" *
spack edit cgal
    • CMakePackage (cgal)
    • AutotoolsPackage (gcc, gdal, hdf5, mvapich2),
    • Package (python, lua, julia)
      • extensions (e.g., Python packages like py-numpy)
    • …

Installation of Dependencies

  • check and install dependencies 

    spack spec -I libosrm
spack add libosrm
spack install --only=dependencies libosrm

       46    /home/hpcgogol/dev/spack/lib/spack/env/gcc/gcc -std=gnu99 -fPIC -o luac  -L/home/hpcgogol/dev/spack/opt/spack/linux-ubuntu16.04-broad
	 well/gcc-5.4.0/readline-8.0-feqfow6hrrf5blkasduwuwnuap6s5h7n/lib -L/usr/lib luac.o liblua.a -lm -Wl,-E -ldl -lreadline -lncursesw -lt
	 infow
>> 47    /usr/bin/ld: cannot find -ltinfow
>> 48    collect2: error: ld returned 1 exit status

Installation of Dependencies: Fix

  • libtinfow is a part of ncurses 

    ncurses:
  externals:
  - spec: ncurses@6.0.20160213+symlinks+termlib
    prefix: /usr
  buildable: false

  • comment it out and use the spec libosrm^ncurses+termlib 

    spack add libosrm^ncurses+termlib
spack install --only=dependencies libosrm^ncurses+termlib

Typical Package Workflow

spack edit libosrm
spack install --fail-fast libosrm
# ... build breaks! ...
spack clean libosrm
spack edit libosrm
spack install --fail-fast libosrm
# ...
spack uninstall -ay libosrm
spack repo remove my_spack_repo

Typical Package Workflow: Useful Tricks

  • extract build environment 

    spack build-env libosrm

  • enter to build environment for experiments 

    spack build-env libosrm bash

  • explore folders 

    spack cd -e hidalgo_lge
spack cd -p libosrm
echo $(spack location --build-dir libosrm)
spack cd --install-dir libosrm

Further information