VASP GPU compilation

Compile VASP (6.4.1) within WSL2 Ubuntu on GPU parallelized using OpenMPI + OpenMP, combined with MKL and supporting HDF5

Full description for the video: https://youtu.be/i-spfjcEXMk

Compiled version: VASP 6.4.1
(10/11/2023)

Contact: lebedmi2@cvut.cz

Official documentation for compiling VASP: https://www.vasp.at/wiki/index.php/Installing_VASP.6.X.X
Official documentation for makefiles: https://www.vasp.at/wiki/index.php/Makefile.include

Tested PC:

Intel(R) Core(TM) i5-14600K 3.50 GHz
RTX 2060 SUPER
Kingston FURY 32GB KIT DDR5 6000MHz CL32 Renegade
WSL2 Ubuntu version: 22.04

Steps:

1) Compile Nvidia HPC SDK, which provides the necessary compilers and libraries for VASP compilation on GPU.
2) Compile oneAPI Base Toolkit for Intel’s Math Kernel Library (MKL), which improves the performance of VASP on Intel processors (otherwise compile VASP without MKL).

3) Compile HDF5 using Nvidia HPC SDK compilers to support h5 output format from VASP (useful for example when post-processing data with py4vasp)
4) Compile VASP.

Prerequisites:

Not all are necessary but might be useful:

sudo apt update
sudo apt upgrade
sudo apt-get install build-essential cmake cmake-curses-gui libopenmpi-dev openmpi-bin libfftw3-dev

Nvidia HPC SDK

Download the newest version from https://developer.nvidia.com/hpc-sdk-downloads:

wget https://developer.download.nvidia.com/hpc-sdk/23.11/nvhpc_2023_2311_Linux_x86_64_cuda_12.3.tar.gz

Install it:

tar xpzf nvhpc_2023_2311_Linux_x86_64_cuda_12.3.tar.gz
sudo nvhpc_2023_2311_Linux_x86_64_cuda_12.3/install

Export the following directories into the PATH system environments in order to successfully connect all libraries and binaries when compiling VASP using Nvidia HPC:

nano ~/.bashrc

Scroll to the bottom. Insert there the following (if you changed the installation folder of NVIDIA HPC SDK or you downloaded the newer version with a newer Cuda, modify the path accordingly), then save (control + s), exit nano (control + x), and write “source ~/.bashrc” to refresh the console.

#NVIDIA HPC SDK
export PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/23.11/comm_libs/mpi/bin:$PATH
export MANPATH=$MANPATH:/opt/nvidia/hpc_sdk/Linux_x86_64/23.11/comm_libs/mpi/man
MANPATH=$MANPATH:/opt/nvidia/hpc_sdk/Linux_x86_64/23.11/compilers/man; export MANPATH
PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/23.11/compilers/bin:$PATH; export PATH
export PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/23.11/compilers/compilers/extras:$PATH
export LD_LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/23.11/compilers/extras/qd/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/23.11/cuda/12.3/targets/x86_64-linux/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/23.11/comm_libs/12.3/openmpi4/openmpi-4.1.5/lib:$LD_LIBRARY_PATH
export PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/23.11/comm_libs/12.3/openmpi4/openmpi-4.1.5/bin:$PATH

Check if nvfortran is correctly set:

which nvfortran
nvfortran --version

It should give its path if correct, e.g. /opt/nvidia/hpc_sdk/Linux_x86_64/23.11/compilers/bin/nvfortran.

oneAPI Base Toolkit

Download the oneAPI Base Toolkit (https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html?operatingsystem=linux&distributions=online):

wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/992857b9-624c-45de-9701-f6445d845359/l_BaseKit_p_2023.2.0.49397.sh

Install it:

sudo sh ./l_BaseKit_p_2023.2.0.49397.sh

Continue with the installation according to the instructions (accept terms, recommended installation, ignore warnings about GUI and continue, skip Enclipse configuration).

Add MKL to PATH:

#oneAPI MKL
export PATH=/opt/intel/oneapi/mkl/2023.2.0:$PATH
export LD_LIBRARY_PATH=/opt/intel/oneapi/mkl/2023.2.0/lib/intel64:$LD_LIBRARY_PATH

If you downloaded newer version, modify the paths accordingly.

HDF5

Create a new folder where HDF5 will be compiled:

mkdir HDF5_nvc_compiler
cd HDF5_nvc_compiler

Download into this folder Cmake version of HDF5 from https://www.hdfgroup.org/downloads/hdf5/source-code/#:

tar xvzf CMake-hdf5-1.14.3.tar.gz
cd CMake-hdf5-1.14.3/

Compile LIBAEC and ZLib downloaded with HDF5:

tar xvzf LIBAEC.tar.gz
tar xvzf ZLib.tar.gz
cd libaec-v1.0.6
mkdir build
cd build
cmake ..
sudo make install
cd ..
cd ..
cd zlib-1.3
mkdir build
cd build
cmake ..
sudo make install

Add libsz.so to path:

export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

cd ..
cd ..

Copy hdf5-1.14.3 source to the HDF5_nvc_compiler dir:

cp -r hdf5-1.14.3/ ..

In HDF5_nvc_compiler dir, make a new directory called “myhdfstuff”. Without this called folder, compilation is problematic:

cd ..
mkdir myhdfstuff

Copy the source files in hdf5-1.14.3 to myhdfstuff:

cp -r hdf5-1.14.3/ myhdfstuff/

Build HDF5 using nvc compiler by Nvidia HPC SDK:

cd myhdfstuff
mkdir build
cd build
cmake -G "Unix Makefiles" -DBUILD_SHARED_LIBS:BOOL=ON -DHDF5_BUILD_FORTRAN=YES -DHDF5_BUILD_TOOLS:BOOL=ON -DCMAKE_C_COMPILER=nvc ../hdf5-1.14.3 
cmake --build . --config Release -j12
cpack -C Release CPackConfig.cmake
./HDF5-1.14.3-Linux.sh

When building HDF5 for compilation with GNU, use gcc instead of nvc. For running the last .sh command, you might need to first needed add permissions (chmod +x HDF5-1.14.3-Linux.sh)

Scroll down with enter, accept license with “y”, and write “n” to have it installed in the build directory. Export HDF5 directory to system variables in bashrc:

export PATH=/home/lebedmi2/SOFTWARE/HDF5_nvc_compiler/myhdfstuff/build/HDF_Group/HDF5/1.14.3:$PATH
export LD_LIBRARY_PATH=/home/lebedmi/SOFTWARE/HDF5_nvc_compiler/myhdfstuff/build/HDF_Group/HDF5/1.14.3/lib:$LD_LIBRARY_PATH

VASP

tar xvzf vasp.6.4.1.tgz
mv vasp.6.4.1 vasp.6.4.1_GPU_MKL_2
cd vasp.6.4.1_GPU_MKL_2
cd vasp.6.4.1

To run commands without sudo (otherwise you could face problems with paths, see below this section), give rights to the currect user for the vasp.6.4.1 folder (change only the username ‚lebedmi2‘ in the following command. You can check name of the user with command ‚whoami‚):

sudo chown -R lebedmi2 .

Create makefile.include file:

nano makefile.include

Into makefile.include, copy here from: makefile.include.nvhpc_ompi_mkl_omp_acc (https://www.vasp.at/wiki/index.php/Makefile.include.nvhpc_ompi_mkl_omp_acc or find it in arch folder of your VASP). On the first line of makefile.include, add your path to the MKL folder “export MKLROOT=/opt/intel/oneapi/mkl/2023.2.0” to solve this error:

nvfortran-Fatal-MKLROOT not found. Please set the environment variable MKLROOT to the location of Intel’s Math Kernel Libraries (the part of the path to the *.so files that precedes ‚lib/<arch>‘.).

In makefile.include, set the paths to to MKL (MKLROOT) and HDF5 (HDF5_ROOT). Also uncomment all the options for HDF5 and change the CUDA version to match the one installed with your NVIDIA HPC SDK. I have version 12.3. Additionally, verify the architecture code of your graphics card. For an RTX 2060, use the code cc=70 (cc=75). If you are using an RTX 4090, the appropriate code is either cc=89 or cc=90.

FC = mpif90 -acc -gpu=cc70,cc80,cuda12.3 -mp 
FCL = mpif90 -acc -gpu=cc70,cc80,cuda12.3 -mp -c++libs

MKLROOT    ?= /opt/intel/oneapi/mkl/2023.2.0
HDF5_ROOT  ?= /home/lebedmi2/SOFTWARE/HDF5_nvc_compiler/myhdfstuff/build/HDF_Group/HDF5/1.14.3

Compile vasp:

make DEPS=1 -j

It will take some time. If no error appeared, check if all libraries are correctly set and none are missing:

cd bin
ldd vasp_std

Export number of threads to bashrc:

export OMP_NUM_THREADS=1

If more than one vasp compilation will be installed on the computer, rename the binary:

mv vasp_std vasp_std_gpu_2

Add path vasp_std_gpu_2 to bashrc:

export PATH=/home/lebedmi2/SOFTWARE/VASP/vasp.6.4.1_GPU_MKL_2/bin:$PATH

Note: If you simply run “make test” without any modifications in the testsuite, it will not work on GPU. You need to change the number of mpirun ranks to 1, change the name of the binary for vasp_std (including the absolute path to it), and disable tests which are not GPU supported. For that, write the following to the console:

export VASP_TESTSUITE_EXE_STD="mpirun -np 1 /home/lebedmi2/SOFTWARE/VASP/vasp.6.4.1_GPU_MKL_2/bin/vasp_std_gpu_2"
export VASP_TESTSUITE_EXE_GAM="mpirun -np 1 /home/lebedmi2/SOFTWARE/VASP/vasp.6.4.1_GPU_MKL_2/bin/vasp_std_gpu_2"
export VASP_TESTSUITE_EXE_NCL="mpirun -np 1 /home/lebedmi2/SOFTWARE/VASP/vasp.6.4.1_GPU_MKL_2/bin/vasp_std_gpu_2"
export VASP_TESTSUITE_CUDA=Y
export VASP_TESTSUITE_SKIP_NOCUDA=Y

Then you can run the tests to see if GPU VASP was compiled correctly (write this command within the directory where ‚makefile.include‘ file is):

make test

Run calculations

Make sure that the openMPI used for the compilation is loaded, otherwise source it:

export PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/23.11/comm_libs/12.3/openmpi4/openmpi-4.1.5/bin:$PATH
export LD_LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/23.11/comm_libs/12.3/openmpi4/openmpi-4.1.5/bin:$LD_LIBRARY_PATH

Then within a folder with INCAR, POSCAR, POTCAR, KPOINTS files run the following to start the calculations using the GPU VASP:

mpirun -np 1 vasp_std_gpu_2

Compile CPU version of VASP using NVIDIA HPC-SDK next to the GPU version

Note: When compiling CPU VASP next to the GPU, it is better to use the same compiler: NVIDIA HPC-SDK. Otherwise you could have problems with necessity to have two versions of HDF5. One compiled with nvc and one compiled with gcc/oneAPI/AOCC. Also you could have problem with the openmpi (which you could probably solve if you compile the used openmpi for CPU version as CUDA aware)

Extract the VASP, rename it as you want and create there the makefile.include file with the content from: makefile.include.nvhpc_ompi_mkl_omp makefile.include.nvhpc_ompi_mkl_omp – Vaspwiki:

In makefile.include, change again the roots for MKL and HDF5 and on the first line, write “export MKLROOT=/opt/intel/oneapi/mkl/2023.2.0”.

Then continue as for the GPU compilation.

“make test -j 10” with CPU should work. Try after the compilation is completed. You can wait until it is fully completed. Then modify the binary:

mv vasp_std vasp_std_cpu_2

Do not forget to add its path to bashrc:

cd /home/lebedmi2/DATA/VASP_DATA/test/test_CPU

mpirun -np 1 vasp_std_cpu_2

If HDF5 was successfully incorporated, the output from VASP calculations will include file with .h5 extension.

Solving 'nvfortran not found'

In my ‚makefile‘ (it is in the same location where makefile.include is), I added the following at the first line to check which environmental paths are accessible during compilation:

show-path:
      @echo "Current PATH: $$PATH"

When executing ‚make‚, it shows the correct paths (including the path to nvfortran).
When executing ‚sudo make‚, it shows:
Current PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
These are pre-set secure PATHs used by sudo, not the ones I have set in ~/.bashrc needed for the compilation. The other way how to check the secure paths is to write ‚sudo env‚ and see the row starting with ‚PATH‘.

To have access to paths in ~/.bashrc, I would need to run the compilation without sudo, but doing so results in a ‚permission denied‘ error for some files.

You can address this by one of the following options (the first one is the most straightforward):

1) Change the ownership of the folder with VASP and all its content to the user to be able to run ‚make‘ without sudo:

sudo chown -R lebedmi ~/SOFTWARE/VASP/v.6.4.1/vasp.6.4.1

Change the ‚lebedmi‚ for your username (you can check it with command ‚whoami‘) and modify ‚~/SOFTWARE/VASP/v.6.4.1/vasp.6.4.1‚ to the extracted directory with your VASP. Then you can run the compilation as ‚make DEPS=1 -j4‚).

2) First remove preset /mnt/ folders from the environment variables by writingthe following command to the last line of bashrc (otherwise I was getting error: env: ‘Files/NVIDIA’: No such file or directory):

nano ~/.bashrc
export PATH=$(echo "$PATH" | tr ':' '\n' | grep -v '/mnt/c' | tr '\n' ':')

Save and exit, then

source ~/.bashrc

Now you can compile by running the sudo command with ‚env PATH=$PATH‘, e.g.:

sudo env PATH=$PATH make DEPS=1 -j4

3) Write ‚sudo visudo‘ and on the row starting with ‚Defaults secure_path=“…..‘ add the paths you need to have access when running command with ‚sudo‘

Calculation speed comparison

---> HERE <---

Content of makefile.include for GPU compilation:

export MKLROOT=/opt/intel/oneapi/mkl/2023.2.0
# Default precompiler options
CPP_OPTIONS = -DHOST=\"LinuxNV\" \
-DMPI -DMPI_BLOCK=8000 -Duse_collective \
-DscaLAPACK \
-DCACHE_SIZE=4000 \
-Davoidalloc \
-Dvasp6 \
-Duse_bse_te \
-Dtbdyn \
-Dqd_emulate \
-Dfock_dblbuf \
-D_OPENMP \
-D_OPENACC \
-DUSENCCL -DUSENCCLP2P

CPP = nvfortran -Mpreprocess -Mfree -Mextend -E $(CPP_OPTIONS) $*$(FUFFIX) > $*$(SUFFIX)

# N.B.: you might need to change the cuda-version here
# to one that comes with your NVIDIA-HPC SDK
FC = mpif90 -acc -gpu=cc70,cc80,cuda12.2 -mp 
FCL = mpif90 -acc -gpu=cc70,cc80,cuda12.2 -mp -c++libs

FREE = -Mfree

FFLAGS = -Mbackslash -Mlarge_arrays

OFLAG = -fast

DEBUG = -Mfree -O0 -traceback

OBJECTS = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o

LLIBS = -cudalib=cublas,cusolver,cufft,nccl -cuda

# Redefine the standard list of O1 and O2 objects
SOURCE_O1 := pade_fit.o
SOURCE_O2 := pead.o

# For what used to be vasp.5.lib
CPP_LIB = $(CPP)
FC_LIB = nvfortran
CC_LIB = nvc -w
CFLAGS_LIB = -O
FFLAGS_LIB = -O1 -Mfixed
FREE_LIB = $(FREE)

OBJECTS_LIB = linpack_double.o

# For the parser library
CXX_PARS = nvc++ --no_warnings

##
## Customize as of this point! Of course you may change the preceding
## part of this file as well if you like, but it should rarely be
## necessary ...
##

# Specify your NV HPC-SDK installation (mandatory)
#... first try to set it automatically
NVROOT =$(shell which nvfortran | awk -F /compilers/bin/nvfortran '{ print $$1 }')

# If the above fails, then NVROOT needs to be set manually
#NVHPC ?= /opt/nvidia/hpc_sdk
#NVVERSION = 21.11
#NVROOT = $(NVHPC)/Linux_x86_64/$(NVVERSION)

# Software emulation of quadruple precsion (mandatory)
QD ?= $(NVROOT)/compilers/extras/qd
LLIBS += -L$(QD)/lib -lqdmod -lqd
INCS += -I$(QD)/include/qd

# Intel MKL (FFTW, BLAS, LAPACK, and scaLAPACK)
MKLROOT ?= /opt/intel/oneapi/mkl/2023.2.0
LLIBS += -Mmkl -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64
INCS += -I$(MKLROOT)/include/fftw

# HDF5-support (optional but strongly recommended)
CPP_OPTIONS+= -DVASP_HDF5
HDF5_ROOT ?= /home/lebedmi2/SOFTWARE/HDF5_nvc_compiler/myhdfstuff/build/HDF_Group/HDF5/1.14.3
LLIBS += -L$(HDF5_ROOT)/lib -lhdf5_fortran
INCS += -I$(HDF5_ROOT)/include

# For the VASP-2-Wannier90 interface (optional)
#CPP_OPTIONS += -DVASP2WANNIER90
#WANNIER90_ROOT ?= /path/to/your/wannier90/installation
#LLIBS += -L$(WANNIER90_ROOT)/lib -lwannier

# For the fftlib library (experimental)
#CPP_OPTIONS+= -Dsysv
#FCL += fftlib.o
#CXX_FFTLIB = nvc++ -mp --no_warnings -std=c++11 -DFFTLIB_USE_MKL -DFFTLIB_THREADSAFE
#INCS_FFTLIB = -I./include -I$(MKLROOT)/include/fftw
#LIBS += fftlib
#LLIBS += -ldl

Content of makefile.include for CPU compilation:

export MKLROOT=/opt/intel/oneapi/mkl/2023.2.0
# Default precompiler options
CPP_OPTIONS = -DHOST=\"LinuxNV\" \
-DMPI -DMPI_BLOCK=8000 -Duse_collective \
-DscaLAPACK \
-DCACHE_SIZE=4000 \
-Davoidalloc \
-Dvasp6 \
-Duse_bse_te \
-Dtbdyn \
-Dqd_emulate \
-Dfock_dblbuf \
-D_OPENMP

CPP = nvfortran -Mpreprocess -Mfree -Mextend -E $(CPP_OPTIONS) $*$(FUFFIX) > $*$(SUFFIX)

FC = mpif90 -mp 
FCL = mpif90 -mp -c++libs

FREE = -Mfree

FFLAGS = -Mbackslash -Mlarge_arrays

OFLAG = -fast

DEBUG = -Mfree -O0 -traceback

OBJECTS = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o

# Redefine the standard list of O1 and O2 objects
SOURCE_O1 := pade_fit.o
SOURCE_O2 := pead.o

# For what used to be vasp.5.lib
CPP_LIB = $(CPP)
FC_LIB = nvfortran
CC_LIB = nvc -w
CFLAGS_LIB = -O
FFLAGS_LIB = -O1 -Mfixed
FREE_LIB = $(FREE)

OBJECTS_LIB = linpack_double.o

# For the parser library
CXX_PARS = nvc++ --no_warnings

##
## Customize as of this point! Of course you may change the preceding
## part of this file as well if you like, but it should rarely be
## necessary ...
##

# Specify your NV HPC-SDK installation (mandatory)
#... first try to set it automatically
NVROOT =$(shell which nvfortran | awk -F /compilers/bin/nvfortran '{ print $$1 }')

# If the above fails, then NVROOT needs to be set manually
#NVHPC ?= /opt/nvidia/hpc_sdk
#NVVERSION = 21.11
#NVROOT = $(NVHPC)/Linux_x86_64/$(NVVERSION)

# Software emulation of quadruple precsion (mandatory)
QD ?= $(NVROOT)/compilers/extras/qd
LLIBS += -L$(QD)/lib -lqdmod -lqd
INCS += -I$(QD)/include/qd

# Intel MKL (FFTW, BLAS, LAPACK, and scaLAPACK)
MKLROOT ?= /opt/intel/oneapi/mkl/2023.2.0
LLIBS += -Mmkl -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64
INCS += -I$(MKLROOT)/include/fftw

# HDF5-support (optional but strongly recommended)
CPP_OPTIONS+= -DVASP_HDF5
HDF5_ROOT ?= /home/lebedmi2/SOFTWARE/HDF5_nvc_compiler/myhdfstuff/build/HDF_Group/HDF5/1.14.3
LLIBS += -L$(HDF5_ROOT)/lib -lhdf5_fortran
INCS += -I$(HDF5_ROOT)/include

# For the VASP-2-Wannier90 interface (optional)
#CPP_OPTIONS += -DVASP2WANNIER90
#WANNIER90_ROOT ?= /path/to/your/wannier90/installation
#LLIBS += -L$(WANNIER90_ROOT)/lib -lwannier

# For the fftlib library (experimental)
#CPP_OPTIONS+= -Dsysv
#FCL += fftlib.o
#CXX_FFTLIB = nvc++ -mp --no_warnings -std=c++11 -DFFTLIB_USE_MKL -DFFTLIB_THREADSAFE
#INCS_FFTLIB = -I./include -I$(MKLROOT)/include/fftw
#LIBS += fftlib
#LLIBS += -ldl