Edison

Table of Contents Build Setup the environment Create a toolchain file Create a build directory Run CMake Build Run Setup the environment Create a single run job script Submit the job

The following instructions are for execution on the NERSC Edison System.

See the Edison page for system details https://www.nersc.gov/users/computational-systems/testbeds/babbage/

Build

Setup the environment

module load cmake/3.0.0

Create a toolchain file

Create 'Edison.cmake' with the following contents

set(CMAKE_SYSTEM_NAME Catamount)
set(CMAKE_C_COMPILER cc)
set(CMAKE_CXX_COMPILER CC)
set(CMAKE_Fortran_COMPILER ftn)
set(MPI_C_COMPILER cc)
set(MPI_CXX_COMPILER CC)
set(MPI_Fortran_COMPILER CC)

Note, setting MPI_XXX_COMPILER seems to be a bit of a hack to prevent cmake FindMPI from finding the wrong headers/libs.

Create a build directory

mkdir buildEdison
cd buildEdison

Run CMake

cmake \
  -DCMAKE_TOOLCHAIN_FILE=../Edison.cmake \
  -DCMAKE_BUILD_TYPE=Release \
  -DPHASTA_INCOMPRESSIBLE=OFF \
  -DPHASTA_COMPRESSIBLE=ON \
  ..

Build

make VERBOSE=1

Run

Setup the environment

see above

Create a single run job script

Create 'runEdison.sh' with the following contents

#!/bin/bash 
#pin processes to cores
export I_MPI_PIN_DOMAIN=core

Submit the job

Note, '-n' specifies the total number of host processes. Since we are not running on the host processes we will set '-n' to equal the number of nodes; a value of zero is not valid.

cd path/to/case/directory # should contain a N-procs_case sub-directory
qsub -l nodes=numNodes -l walltime=HH:MM:SS ./runPhi.sh