VASP CPU AOCL compilation

Compile VASP (6.4.1) within WSL2 Ubuntu on CPU parallelized using OpenMPI + OpenMP, combined with AOCL and supporting HDF5

Compiled version: VASP 6.4.1
(10/11/2023)
Contact: lebedmi2@cvut.cz

Official documentation for compiling VASP: https://www.vasp.at/wiki/index.php/Installing_VASP.6.X.X
Official documentation for makefiles: https://www.vasp.at/wiki/index.php/Makefile.include

Tested PC:
  • AMD 3950x
  • Native Ubuntu version: 22.04
Steps:
1) Compile The AMD Optimizing C/C++ and Fortran Compilers (AOCC)
2) Compile AMD Optimizing CPU Libraries (AOCL).
3) Compile OpenMPI using AOCC compilers.
4) Compile HDF5 using AOCC compilers to support h5 output format from VASP (useful for example when post-processing data with py4vasp).
5) Compile VASP using AOCC and AOCL libraries.
 
Prerequisites:
(Not all are necessary but might be useful): 
sudo apt update
sudo apt upgrade

sudo apt-get install build-essential cmake cmake-curses-gui libopenmpi-dev openmpi-bin libfftw3-dev
sudo apt install python3-pip
pip3 install Pygments
 

1) Compile AOCC

Download the newest version from https://www.amd.com/en/developer/aocc.html#downloads, then compile with:

tar xvf aocc-compiler-4.1.0.tar
cd aocc-compiler-4.1.0
./install.sh

The compilers can be temporarily set with:

source /home/lebedmi2/SOFTWARE/AOCC/setenv_AOCC.sh

Check if there are no major errors with the following script:

source /home/lebedmi2/SOFTWARE/AOCC/setenv_AOCC.sh
./AOCC-prerequisites-check.sh

I needed to solve one error with (suggested by the runned script):

sudo ln -s /lib/x86_64-linux-gnu/libtinfo.so.6 /lib/x86_64-linux-gnu/libtinfo.so.5

Check if clang compiler is successfully loaded:

which clang
clang -v
 

2) Compile AOCL

Download the newest version from https://www.amd.com/en/developer/aocl.html#downloads, then compile with:
 
tar xvf aocl-linux-aocc-4.1.0.tar.gz
mkdir build
cd aocl-linux-aocc-4.1.0
./install.sh -t /home/lebedmi2/SOFTWARE/AOCL/build

Setup AOCL environment by executing the below command

source /home/lebedmi2/SOFTWARE/AOCL/build/4.1.0/aocc/amd-libs.cfg
 

3) OpenMPI compiled with AOCC

Download OpenMPI (use wget command or download from here https://www.open-mpi.org/software/ompi/v5.0/) and compile:

wget https://download.open-mpi.org/release/open-mpi/v5.0/openmpi-5.0.0.tar.gz
mkdir OpenMPI_AOCC
tar xvzf openmpi-5.0.0.tar.gz -C OpenMPI_AOCC
cd OpenMPI_AOCC/openmpi-5.0.0

mkdir build
./configure CC=clang CXX=clang++ FC=flang F77=flang OMPI_CC=clang OMPI_CXX=clang++ OMPI_FC=flang OMPI_F77=flang --prefix=/home/lebedmi2/SOFTWARE/OpenMPI/OpenMPI_AOCC/openmpi-5.0.0/build
make install -j

Activate it (either write the following into the last line of bashrc for permanent activation or write it directly into the console for temporally activation):

export PATH=/home/lebedmi2/SOFTWARE/OpenMPI/OpenMPI_AOCC/openmpi-5.0.0/build/bin:$PATH
export LD_LIBRARY_PATH=/home/lebedmi2/SOFTWARE/OpenMPI/OpenMPI_AOCC/openmpi-5.0.0/build/lib:$LD_LIBRARY_PATH

Note: On Native Ubuntu, I needed to solve the following error by:

 sudo apt install g++-12
checking if C and C++ are link compatible... no
**********************************************************************
* It appears that your C++ compiler is unable to link against object
* files created by your C compiler.  This generally indicates either
* a conflict between the options specified in CFLAGS and CXXFLAGS
* or a problem with the local compiler installation.  More
* information (including exactly what command was given to the
* compilers and what error resulted when the commands were executed) is
* available in the config.log file in this directory.
**********************************************************************
configure: error: C and C++ compilers are not link compatible.  Can not continue.


4) HDF5 compiled with AOCC

To use HDF5 for VASP AOCC, it must be compiled with AOCC clang and flang compilators. Create a new folder where HDF5 will be compiled:

mkdir HDF5_aocc
cd HDF5_aocc

Install ZLIB:

sudo apt install zlib1g

Download into HDF5_Intel folder the newest version of HDF5 from https://www.hdfgroup.org/downloads/hdf5/source-code/# , extract it, and create a new folder called ‚build‘ into which the compilation will be made. 

tar xvzf hdf5-1.14.4-2.tar.gz
cd hdf5-1.14.4-2/

mkdir build

 Configurate HDF5 with compilers from oneAPI:

CC=clang CXX=clang++ ./configure --prefix=/home/lebedmi2/SOFTWARE/TEST_New_HDF5/HDF5_aocc/hdf5-1.14.4-2/build --enable-fortran --enable-cxx --enable-shared 

 Then build and install it:

make -j
make install -j

 Before compiling VASP or running the simulations with this version, make sure to either permanently or temporally export HDF5 directory to system variables:

export PATH=/home/lebedmi2/SOFTWARE/TEST_New_HDF5/HDF5_aocc/hdf5-1.14.4-2/build/bin:$PATH
export LD_LIBRARY_PATH=/home/lebedmi2/SOFTWARE/TEST_New_HDF5/HDF5_aocc/hdf5-1.14.4-2/build/lib:$LD_LIBRARY_PATH

 OLDER VERSION of HDF5 with Cmake:

Create a new folder where HDF5 will be compiled:

mkdir HDF5_AOCC_compiler
cd HDF5_AOCC_compiler

Download into this folder Cmake version of HDF5 from https://www.hdfgroup.org/downloads/hdf5/source-code/#:

tar xvzf CMake-hdf5-1.14.3.tar.gz
cd CMake-hdf5-1.14.3/

Compile LIBAEC and ZLib downloaded with HDF5:

tar xvzf LIBAEC.tar.gz
tar xvzf ZLib.tar.gz
cd libaec-v1.0.6
mkdir build
cd build
cmake ..
sudo make install
cd ..
cd ..
cd zlib-1.3
mkdir build
cd build
cmake ..
sudo make install

Add libsz.so to path:

export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
cd ..
cd ..

Copy hdf5-1.14.3 source to the HDF5_AOCC_compiler dir:

cp -r hdf5-1.14.3/ ..

In HDF5_GCC_compiler dir, make a new directory called “myhdfstuff”. Without this called folder, compilation is problematic:

cd ..
mkdir myhdfstuff

Copy the source files in hdf5-1.14.3 to myhdfstuff:

cp -r hdf5-1.14.3/ myhdfstuff/

Build HDF5 (flang should be automatically switch on as well when switching to clang):

cd myhdfstuff
mkdir build
cd build
cmake -G "Unix Makefiles" -DBUILD_SHARED_LIBS:BOOL=ON -DHDF5_BUILD_FORTRAN=YES -DHDF5_BUILD_TOOLS:BOOL=ON -DCMAKE_C_COMPILER=clang ../hdf5-1.14.3
cmake --build . --config Release -j12
cpack -C Release CPackConfig.cmake
./HDF5-1.14.3-Linux.sh

Scroll down with enter, accept license with “y”, and write “n” to have it installed in the build directory.
Activate it (either write the following into the last line of bashrc for permanent activation or write it directly into the console for temporally activation):

export PATH=/home/lebedmi2/SOFTWARE/HDF5_GCC_compiler/myhdfstuff/build/HDF_Group/HDF5/1.14.3:$PATH
export LD_LIBRARY_PATH=/home/lebedmi/SOFTWARE/HDF5_GCC_compiler/myhdfstuff/build/HDF_Group/HDF5/1.14.3/lib:$LD_LIBRARY_PATH

Note: HDF5 must be compiled with the same compiler as VASP. For example, when using VASP compiled with NVIDIA HPC sdk, nvc compiler for HDF5 is required. On the other hand when using GNU for VASP, HDF5 must be compiled with gcc as well.

5) VASP compiled with AOCC
Before running the compilation (also before running the simulations with this version), make sure to load AOCC and AOCL:

source /home/lebedmi2/SOFTWARE/AOCC/setenv_AOCC.sh
source /home/lebedmi2/SOFTWARE/AOCL/build/4.1.0/aocc/amd-libs.cfg


Load OpenMPI you want to use for the compilation (must be the one compiled with AOCC) and for running the simulations (write to the console or add to the last line of bashrc).

export PATH=/home/lebedmi2/SOFTWARE/OpenMPI/openmpi-5.0.0/build/bin:$PATH
export LD_LIBRARY_PATH=/home/lebedmi2/SOFTWARE/OpenMPI/openmpi-5.0.0/build/lib:$LD_LIBRARY_PATH

Prepare VASP:

tar xvzf vasp.6.4.1.tgz
mv vasp.6.4.1 vasp.6.4.1_AOCC_AOCL
cd vasp.6.4.1_AOCC_AOCL/vasp.6.4.1

To run commands without sudo (otherwise you could face problems with paths, see below this section), give rights to the currect user for the vasp.6.4.1 folder (change only the username ‚lebedmi2‘ in the following command. You can check name of the user with command ‚whoami‚):

sudo chown -R lebedmi2 .

Into the makefile.include, copy the content from /arch/makefile.include.aocc_ompi_aocl_omp (as of 17/12/2023, this file cannot be find here https://www.vasp.at/wiki/index.php/Makefile.include):

cd arch
cp makefile.include.aocc_ompi_aocl_omp ../makefile.include

In makefile.include, set the paths to to the AOCL and HDF5 (HDF5_ROOT):

cd ..
nano makefile.include
AMDBLIS_ROOT ?= /home/lebedmi2/SOFTWARE/AOCL/build/4.1.0/aocc
AMDLIBFLAME_ROOT ?= /home/lebedmi2/SOFTWARE/AOCL/build/4.1.0/aocc
AMDSCALAPACK_ROOT ?= /home/lebedmi2/SOFTWARE/AOCL/build/4.1.0/aocc
AMDFFTW_ROOT  ?= /home/lebedmi2/SOFTWARE/AOCL/build/4.1.0/aocc
HDF5_ROOT  ?= /home/lebedmi2/SOFTWARE/TEST_New_HDF5/HDF5_aocc/hdf5-1.14.4-2/build

Compile VASP:

make DEPS=1 -j

It will take some time. If no error appeared, try to run the test: 

make test -j

You might also want to check if all libraries are correctly set and none are missing:

cd bin
ldd vasp_std

Export number of threads to bashrc:

export OMP_NUM_THREADS=2

If more than one VASP compilation will be installed on the computer, rename the binary:

mv vasp_std vasp_std_aocl

Add path to the vasp_std_aocl to bashrc:

export PATH=/home/lebedmi2/SOFTWARE/VASP/vasp.6.4.1_AOCL/bin:$PATH
 
Run the simulations with VASP AOCC, AOCL:

Before running the simulations, all the following should be exported (if they are not set permanently in ~/.bashrc), e.g.:

#AOCC, AOCL
source /home/lebedmi2/SOFTWARE/ALL_VASP_AOCC/AOCC/setenv_AOCC.sh

source /home/lebedmi2/SOFTWARE/ALL_VASP_AOCC/AOCL/build/4.1.0/aocc/amd-libs.cfg
#OpenMPI AOCC
export PATH=/home/lebedmi2/SOFTWARE/ALL_VASP_AOCC/OpenMPI/OpenMPI_AOCC/openmpi-5.0.0/build/bin:$PATH
export LD_LIBRARY_PATH=/home/lebedmi2/SOFTWARE/ALL_VASP_AOCC/OpenMPI/OpenMPI_AOCC/openmpi-5.0.0/build/lib:$LD_LIBRARY_PATH
#HDF5 AOCC
export PATH=/home/lebedmi2/SOFTWARE/TEST_New_HDF5/HDF5_aocc/hdf5-1.14.4-2/build/bin:$PATH
export LD_LIBRARY_PATH=/home/lebedmi2/SOFTWARE/TEST_New_HDF5/HDF5_aocc/hdf5-1.14.4-2/build/lib:$LD_LIBRARY_PATH

Then run with: mpirun -np 4 vasp_std_aocl 

 

Encountered problems

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
WARNING: CNORMN: search vector ill defined
#0 0x7fe184342960 in ???
#1 0x7fe184341ac5 in ???
#0 0x7fa18edb9960 in ???
#1 0x7fa18edb8ac5 in ???
#0 0x7f6fd4b7b960 in ???
#1 0x7f6fd4b7aac5 in ???

#14 0x5648add477b4 in ???
#15 0xffffffffffffffff in ???

  • Compiled version makefile.include.aocc_ompi_aocl from /arch folder within unpacked VASP works well for the lower number of atoms (tested on 36 atoms), but for higher number (tested on 72), it gives the following error:
VERY BAD NEWS! internal error in subroutine SGRGEN: Too many elements 49
  • OpenMPI not compiled with AOCC leads to errors, e.g.:
gfortran: error: unrecognized command-line option ‘-fno-fortran-main’
gfortran: error: unrecognized command-line option ‘-fno-fortran-main’
gfortran: error: unrecognized command-line option ‘-Mbackslash’; did you mean ‘-fbackslash’?
gfortran: error: unrecognized command-line option ‘-fno-fortran-main’
  • Before starting the compilation, it might be necessary to use:
make veryclean
  • even if the VASP was just extracted. One time, I received the following error, which I solve by ‚make veryclean‚ and by starting the compilation again. 
ld.lld: error: undefined symbol: _gfortran_st_backspace
>>> referenced by diolib.f90
>>>               diolib.o:(append_) in archive lib/libdmy.a
clang-16: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [makefile:132: vasp] Error 1
make[2]: Leaving directory

Solving possible problem with PATHs

In my ‚makefile‘ (it is in the same location where makefile.include is), I added the following at the first line to check which environmental paths are accessible during compilation:

show-path:
@echo "Current PATH: $$PATH"

When executing ‚make‚, it shows the correct paths (including the path to nvfortran).
When executing ‚sudo make‚, it shows:
Current PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
These are pre-set secure PATHs used by sudo, not the ones I have set in ~/.bashrc needed for the compilation. The other way how to check the secure paths is to write ‚sudo env‚ and see the row starting with ‚PATH‘.

To have access to paths in ~/.bashrc, I would need to run the compilation without sudo, but doing so results in a ‚permission denied‘ error for some files.

You can address this by one of the following options (the first one is the most straightforward):

1) Change the ownership of the folder with VASP and all its content to the user to be able to run ‚make‘ without sudo:

sudo chown -R lebedmi ~/SOFTWARE/VASP/v.6.4.1/vasp.6.4.1

Change the ‚lebedmi‚ for your username (you can check it with command ‚whoami‘) and modify ‚~/SOFTWARE/VASP/v.6.4.1/vasp.6.4.1‚ to the extracted directory with your VASP. Then you can run the compilation as ‚make DEPS=1 -j4‚).

2) In case of WSL2 Ubuntu (not needed for native Ubuntu), first remove preset /mnt/ folders from the environment variables by writing the following command to the last line of bashrc:

nano ~/.bashrc
export PATH=$(echo "$PATH" | tr ':' '\n' | grep -v '/mnt/c' | tr '\n' ':')

Save and exit, then

source ~/.bashrc

Now you can compile by running the sudo command with ‚env PATH=$PATH‘, e.g.:

sudo env PATH=$PATH make DEPS=1 -j4

3) Write ‚sudo visudo‘ and on the row starting with ‚Defaults secure_path=“…..‘ add the paths you need to have access when running command with ‚sudo‘

Content of makefile.include for CPU AOCC, AOCL compilation:
# Default precompiler options
CPP_OPTIONS = -DHOST=\"LinuxGNU\" \
              -DMPI -DMPI_BLOCK=8000 -Duse_collective \
              -DscaLAPACK \
              -DCACHE_SIZE=4000 \
              -Davoidalloc \
              -Dvasp6 \
              -Duse_bse_te \
              -Dtbdyn \
              -Dfock_dblbuf \
              -D_OPENMP

CPP         = flang -E -ffree-form -C -w $*$(FUFFIX) >$*$(SUFFIX) $(CPP_OPTIONS)

FC          = mpif90 -fopenmp
FCL         = mpif90 -fopenmp

FREE        = -ffree-form -ffree-line-length-none

FFLAGS      = -w -fno-fortran-main -Mbackslash

OFLAG       = -O2
OFLAG_IN    = $(OFLAG)
DEBUG       = -O0

OBJECTS     = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o
OBJECTS_O1 += fftw3d.o fftmpi.o fftmpiw.o
OBJECTS_O2 += fft3dlib.o

# For what used to be vasp.5.lib
CPP_LIB     = $(CPP)
FC_LIB      = $(FC)
CC_LIB      = clang
CFLAGS_LIB  = -O
FFLAGS_LIB  = -O1
FREE_LIB    = $(FREE)

OBJECTS_LIB = linpack_double.o

# For the parser library
CXX_PARS    = clang++
LLIBS       = -lstdc++

##
## Customize as of this point! Of course you may change the preceding
## part of this file as well if you like, but it should rarely be
## necessary ...
##

# When compiling on the target machine itself, change this to the
# relevant target when cross-compiling for another architecture
VASP_TARGET_CPU ?= -march=native
FFLAGS     += $(VASP_TARGET_CPU)

# BLAS (mandatory)
AMDBLIS_ROOT ?= /home/lebedmi2/SOFTWARE/AOCL/build/4.1.0/aocc
BLAS        = -L${AMDBLIS_ROOT}/lib -lblis-mt

# LAPACK (mandatory)
AMDLIBFLAME_ROOT ?= /home/lebedmi2/SOFTWARE/AOCL/build/4.1.0/aocc
LAPACK      = -L${AMDLIBFLAME_ROOT}/lib -lflame

# scaLAPACK (mandatory)
AMDSCALAPACK_ROOT ?= /home/lebedmi2/SOFTWARE/AOCL/build/4.1.0/aocc
SCALAPACK   = -L${AMDSCALAPACK_ROOT}/lib -lscalapack

LLIBS      += $(SCALAPACK) $(LAPACK) $(BLAS)

# FFTW (mandatory)
AMDFFTW_ROOT  ?= /home/lebedmi2/SOFTWARE/AOCL/build/4.1.0/aocc
LLIBS      += -L$(AMDFFTW_ROOT)/lib -lfftw3 -lfftw3_omp
INCS       += -I$(AMDFFTW_ROOT)/include

# HDF5-support (optional but strongly recommended)
CPP_OPTIONS+= -DVASP_HDF5
HDF5_ROOT  ?= /home/lebedmi2/SOFTWARE/HDF5_aocl/myhdfstuff/build/HDF_Group/HDF5/1.14.3
LLIBS      += -L$(HDF5_ROOT)/lib -lhdf5_fortran
INCS       += -I$(HDF5_ROOT)/include

# For the VASP-2-Wannier90 interface (optional)
#CPP_OPTIONS    += -DVASP2WANNIER90
#WANNIER90_ROOT ?= /path/to/your/wannier90/installation
#LLIBS          += -L$(WANNIER90_ROOT)/lib -lwannier

# For the fftlib library (recommended)
#CPP_OPTIONS+= -Dsysv
#FCL        += fftlib.o
#CXX_FFTLIB  = clang++ -fopenmp -std=c++11 -DFFTLIB_THREADSAFE
#INCS_FFTLIB = -I./include -I$(AMDFFTW_ROOT)/include
#LIBS       += fftlib
#LLIBS      += -ldl