Skip to content

Global Arrays

Jump to bottom
Jeff Hammond edited this page Jan 10, 2020 · 1 revision

Users are strongly encouraged to use Global Arrays 5.2 with ARMCI-MPI.

Table of Contents

Global Arrays 5.2

As of GA 5.2, an external build of ARMCI is supported. This has been verified to work perfectly on Mac and Blue Gene/Q using the MPICH trunk (3.0.4+) and MPICH2 1.5, respectively.

First, build ARMCI-MPI normally: 

git clone git://git.mpich.org/armci-mpi.git
cd armci-mpi
mkdir build
 ../configure CC=mpicc --prefix=$ARMCI_MPI_INSTALL_DIR --enable-g
make -j8
make -j8 check
make install

Second, build Global Arrays 5.2 like this: 

svn co https://svn.pnl.gov/svn/hpctools/branches/ga-5-2
cd ga-5-2
mkdir build
cd build
../configure MPICC=mpicc CC=mpicc MPIF77=mpif77 F77=mpif77 MPICXX=mpicxx CXX=mpicxx \
--with-mpi --with-armci=$ARMCI_MPI_INSTALL_DIR MPIEXEC="mpiexec -np %NP%" NPROCS=4
make -j8
make -j8 check

This might be fixed For some reason, the simple_groups_comm test build is broken in automake and thus one has to remove it and all related pieces for make check to succeed. Otherwise, you can just run the test suite manually.

NWChem 6.3

Add these to your environment (in addition to the other settings you use that don't relate to ARMCI): 

export ARMCI_NETWORK=ARMCI
export EXTERNAL_ARMCI_PATH=$ARMCI_MPI_INSTALL_DIR

Global Arrays 5.1

ARMCI-MPI does not work with GA 5.1. This is unlikely to change in the future. The fix requires ARMCI-MPI to provide the full TCGMSG interface and writing that code is not our idea of a good time, particularly now that GA 5.2 is working fine.

Global Arrays 5.0

These directions are not correct any more but they were used when the NWChem trunk was using GA 5.0. They are a close enough approximation to what is required that savvy users can figure things out. Non-savvy users should just use GA 5.2.

Details 

# ARMCI_MPI_ROOT needs to be on a filesystem which is big enough to hold the NWChem source, 
# which means not your home directory on e.g. Jaguar or Hopper.

mkdir $ARMCI_MPI_ROOT && cd $ARMCI_MPI_ROOT
# this is only required if you're building from the SVN source
svn co https://svn.pnl.gov/svn/nwchem nwchem-trunk
# wait a very long time
export NWCHEM_TOP=$ARMCI_MPI_ROOT/nwchem-trunk
cd $NWCHEM_TOP/src/tools
# this is only required if you're building from the SVN source
./get-tools 
# this is unnecessary clutter
rm -rf tcgmsg-mpi tcgmsg pario ma LinAlg global ga-4-3 ga++ examples doc config cca armci-portals armci 
cd ga-dev
mv armci armci-pnnl # or just delete it
svn co https://svn.mcs.anl.gov/repos/armci-mpi/trunk armci-mpi
ln -s armci-mpi armci
cd armci
# if necessary, build latest m4, automake, autoconf, libtool somewhere
./autogen.sh
# these steps are only necessary on Cray XT and XE
ln -s src src-gemini
ln -s src src-portals

# because the GA build system in NWChem 6.0 is imperfect, do this in the armci-mpi directory
mkdir config
touch config/makemp.h config/makecoms.h
# configure ARMCI-MPI in the base directory
# change references to armci/src in tools/GNUmakefile to armci

# at this point, you should be done with the GA/ARMCI part

cd $NWCHEM_TOP/src

# comment out these lines from $NWCHEM_TOP/src/GNUmakefile:
include ./tools/armci/config/makecoms.h  
include ./tools/armci/config/makemp.h  

# choose one of the following, depending on if you are on BGP or not
export NWCHEM_TARGET=LINUX64 
export NWCHEM_TARGET=BGP

# even though ARMCI isn't using a native channel, this variable may have other side effects.
export ARMCI_NETWORK=DCMFMPI
export ARMCI_NETWORK=GEMINI
export ARMCI_NETWORK=OPENIB
export ARMCI_NETWORK=PORTALS

Global Arrays 4.3

I recall that we got this to work but I don't recommend it.

Clone this wiki locally