Installing parallel I/O libraries

High-resolution simulations (e.g. modeling the entire Greenland ice sheet using the 900 m resolution and a grid of 1700 by 3000 points) can produce huge amounts of data and I/O can become a bottleneck.

PISM supports several parallel I/O approaches that take advantage of parallel NetCDF, PnetCDF, and NCAR ParallelIO. The administrators of your HPC system should be able to help you install these libraries, but it may be easier to install them in the “home” directory instead.

This section describes the steps needed to build

  • NetCDF with parallel I/O based on HDF5 (needed to use PISM’s option -o_format netcdf4_parallel),

  • PNetCDF (needed to use PISM’s option -o_format pnetfdf),

  • ParallelIO (needed to use options -o_format pio_netcdf4p, -o_format pio_netcdf4c, -o_format pio_pnetcdf, -o_format pio_netcdf).

Scripts below install libraries in ~/local/library_name, using ~/local/build/library_name to build them.

Section Building PISM with libraries in non-standard locations explains how build PISM with these libraries.

Section PISM’s I/O performance explains how to use them in PISM.

Installing HDF5-based parallel NetCDF

Installing HDF5

 1# Install HDF5 1.12.0 with parallel I/O in ~/local/hdf5,
 2# using ~/local/build/hdf5 as the build directory.
 3
 4version=1.12.0
 5prefix=$HOME/local/hdf5
 6build_dir=~/local/build/hdf5
 7hdf5_site=https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.12
 8url=${hdf5_site}/hdf5-${version}/src/hdf5-${version}.tar.gz
 9
10mkdir -p ${build_dir}
11pushd ${build_dir}
12
13wget -nc ${url}
14tar xzf hdf5-${version}.tar.gz
15
16pushd hdf5-${version}
17
18CC=mpicc CFLAGS=-w ./configure \
19  --disable-static \
20  --enable-parallel \
21  --prefix=${prefix} 2>&1 | tee hdf5_configure.log
22
23make all 2>&1 | tee hdf5_compile.log
24make install 2>&1 | tee hdf5_install.log
25
26popd
27popd

To compile parallel HDF5 one should use the MPI compiler wrapper mpicc and run configure with the option --enable-parallel. The flag -w is not important: it hides numerous compiler warnings emitted when building HDF5.

Installing NetCDF

 1# Install parallel NetCDF using parallel HDF5 in ~/local/hdf5 and
 2# ~/local/build/netcdf as a build directory.
 3
 4hdf5=~/local/hdf5
 5
 6version=4.7.4
 7prefix=$HOME/local/netcdf
 8build_dir=~/local/build/netcdf
 9url=https://github.com/Unidata/netcdf-c/archive/refs/tags/v${version}.tar.gz
10
11mkdir -p ${build_dir}
12pushd ${build_dir}
13
14wget -nc ${url}
15tar zxf v${version}.tar.gz
16
17pushd netcdf-c-${version}
18
19export CFLAGS="-g -O0"
20unset CFLAGS
21
22CC=mpicc CPPFLAGS=-I${hdf5}/include LDFLAGS=-L${hdf5}/lib ./configure \
23        --enable-netcdf4 \
24        --disable-dap \
25        --prefix=${prefix} 2>&1 | tee netcdf_configure.log
26
27make all 2>&1 | tee netcdf_compile.log
28make install 2>&1 | tee netcdf_install.log
29
30popd
31popd

Here we use the same compiler wrapper and set CPPFLAGS and LDFLAGS to select the parallel HDF5 library installed earlier. The option --enable-netcdf4 is required for parallel I/O; --disable-dap is not required (it disables a NetCDF feature not used by PISM).

Installing PnetCDF

 1# Install PnetCDF 1.12.1 in ~/local/pnetcdf,
 2# using ~/local/build/pnetcdf as a build directory.
 3
 4version=1.12.1
 5prefix=$HOME/local/pnetcdf
 6build_dir=~/local/build/pnetcdf/
 7url=https://parallel-netcdf.github.io/Release/pnetcdf-${version}.tar.gz
 8
 9mkdir -p ${build_dir}
10pushd ${build_dir}
11
12wget -nc ${url}
13tar xzf pnetcdf-${version}.tar.gz
14
15pushd pnetcdf-${version}
16
17./configure \
18      --prefix=${prefix} \
19      --enable-shared \
20      --disable-static \
21      --disable-cxx \
22      --disable-fortran
23
24make all
25make install
26
27popd
28popd

Here we disable PnetCDF’s C++ and Fortran APIs and build the shared library.

Installing NCAR ParallelIO

 1# Install NCAR ParallelIO in ~/local/parallelio
 2# using parallel NetCDF and PnetCDF installed in ~/local/netcdf
 3# and ~/local/pnetcdf. Uses ~/local/build/parallelio
 4# as a build directory.
 5
 6netcdf_prefix=~/local/netcdf
 7pnetcdf_prefix=~/local/pnetcdf
 8
 9url=https://github.com/NCAR/ParallelIO.git
10build=~/local/build/parallelio
11prefix=$HOME/local/parallelio
12
13rm -rf ${build}
14mkdir -p ${build}/build ${build}/sources
15
16git clone ${url} ${build}/sources
17
18pushd ${build}/sources
19git checkout -b 2_5_7 pio2_5_7
20popd
21
22pushd ${build}/build
23
24CC=mpicc cmake \
25  -DCMAKE_C_FLAGS="-fPIC" \
26  -DCMAKE_INSTALL_PREFIX=${prefix} \
27  -DNetCDF_PATH=${netcdf_prefix} \
28  -DPnetCDF_PATH=${pnetcdf_prefix} \
29  -DPIO_ENABLE_FORTRAN=0 \
30  -DPIO_ENABLE_TIMING=0 \
31  ${build}/sources
32
33make install
34
35popd

Here we use CMake’s variable CMAKE_FIND_ROOT_PATH to tell CMake to use libraries in ~/local/netcdf and ~/local/pnetcdf, to install in ~/local/parallelio, and to disable ParallelIO features not used by PISM.

Previous Up Next