Compile VASP (6.4.1) within WSL2 Ubuntu on GPU parallelized using OpenMPI + OpenMP, combined with MKL and supporting HDF5
Compiled version: VASP 6.4.1
(10/11/2023)
Contact: lebedmi2@cvut.cz
Official documentation for compiling VASP: https://www.vasp.at/wiki/index.php/Installing_VASP.6.X.X
Official documentation for makefiles: https://www.vasp.at/wiki/index.php/Makefile.include
Tested PC:
- Intel(R) Core(TM) i5-14600K 3.50 GHz
- RTX 2060 SUPER
- Kingston FURY 32GB KIT DDR5 6000MHz CL32 Renegade
- WSL2 Ubuntu version: 22.04
Steps:
2) Compile oneAPI Base Toolkit for Intel’s Math Kernel Library (MKL), which improves the performance of VASP on Intel processors (otherwise compile VASP without MKL).
4) Compile VASP.
Prerequisites:
sudo apt update
sudo apt upgrade
sudo apt-get install build-essential cmake cmake-curses-gui libopenmpi-dev openmpi-bin libfftw3-dev
1) Nvidia HPC SDK
Download the newest version from https://developer.nvidia.com/hpc-sdk-downloads:
wget https://developer.download.nvidia.com/hpc-sdk/23.11/nvhpc_2023_2311_Linux_x86_64_cuda_12.3.tar.gz
Install it:
tar xpzf nvhpc_2023_2311_Linux_x86_64_cuda_12.3.tar.gz
sudo nvhpc_2023_2311_Linux_x86_64_cuda_12.3/install
Export the following directories into the PATH system environments in order to successfully connect all libraries and binaries when compiling VASP using Nvidia HPC:
nano ~/.bashrc
Scroll to the bottom. Insert there the following (if you changed the installation folder of NVIDIA HPC SDK or you downloaded the newer version with a newer Cuda, modify the path accordingly), then save (control + s), exit nano (control + x), and write “source ~/.bashrc” to refresh the console.
#NVIDIA HPC SDK
export PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/23.11/comm_libs/mpi/bin:$PATH
export MANPATH=$MANPATH:/opt/nvidia/hpc_sdk/Linux_x86_64/23.11/comm_libs/mpi/man
MANPATH=$MANPATH:/opt/nvidia/hpc_sdk/Linux_x86_64/23.11/compilers/man; export MANPATH
PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/23.11/compilers/bin:$PATH; export PATH
export PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/23.11/compilers/compilers/extras:$PATH
export LD_LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/23.11/compilers/extras/qd/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/23.11/cuda/12.3/targets/x86_64-linux/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/23.11/comm_libs/12.3/openmpi4/openmpi-4.1.5/lib:$LD_LIBRARY_PATH
export PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/23.11/comm_libs/12.3/openmpi4/openmpi-4.1.5/bin:$PATH
Check if nvfortran is correctly set:
which nvfortran
nvfortran --version
It should give its path if correct, e.g. /opt/nvidia/hpc_sdk/Linux_x86_64/23.11/compilers/bin/nvfortran.
2) oneAPI Base Toolkit
Download the oneAPI Base Toolkit (https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html?operatingsystem=linux&distributions=online):
wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/992857b9-624c-45de-9701-f6445d845359/l_BaseKit_p_2023.2.0.49397.sh
Install it:
sudo sh ./l_BaseKit_p_2023.2.0.49397.sh
Continue with the installation according to the instructions (accept terms, recommended installation, ignore warnings about GUI and continue, skip Enclipse configuration).
Add MKL to PATH:
#oneAPI MKL
export PATH=/opt/intel/oneapi/mkl/2023.2.0:$PATH
export LD_LIBRARY_PATH=/opt/intel/oneapi/mkl/2023.2.0/lib/intel64:$LD_LIBRARY_PATH
If you downloaded newer version, modify the paths accordingly.
3) HDF5 compiled with HPC SDK
Create a new folder where HDF5 will be compiled:
mkdir HDF5_nvc
cd HDF5_nvc
Install ZLIB:
sudo apt install zlib1g
Download into HDF5_nvc folder the newest version of HDF5 from https://www.hdfgroup.org/downloads/hdf5/source-code/# , extract it, and create a new folder called ‚build‘ into which the compilation will be made.
tar xvzf hdf5-1.14.4-2.tar.gz
cd hdf5-1.14.4-2/
mkdir build
Configurate HDF5 with compilers from oneAPI:
CC=nvc CXX=nvc++ ./configure --prefix=/home/lebedmi2/SOFTWARE/TEST_New_HDF5/HDF5_intel/hdf5-1.14.4-2/build --enable-fortran --enable-cxx --enable-shared
Then build and install it:
make -j
make install -j
Before compiling VASP or running the simulations with this version, make sure to either permanently or temporally export HDF5 directory to system variables:
export PATH=/home/lebedmi2/SOFTWARE/TEST_New_HDF5/HDF5_nvc/hdf5-1.14.4-2/build/bin:$PATH
export LD_LIBRARY_PATH=/home/lebedmi2/SOFTWARE/TEST_New_HDF5/HDF5_nvc/hdf5-1.14.4-2/build/lib:$LD_LIBRARY_PATH
OLDER VERSION of HDF5 with Cmake:
Create a new folder where HDF5 will be compiled:
mkdir HDF5_nvc_compilercd HDF5_nvc_compiler
Download into this folder Cmake version of HDF5 from https://www.hdfgroup.org/downloads/hdf5/source-code/#:
tar xvzf CMake-hdf5-1.14.3.tar.gzcd CMake-hdf5-1.14.3/
Compile LIBAEC and ZLib downloaded with HDF5:
tar xvzf LIBAEC.tar.gztar xvzf ZLib.tar.gzcd libaec-v1.0.6mkdir buildcd buildcmake ..sudo make installcd ..cd ..cd zlib-1.3mkdir buildcd buildcmake ..sudo make install
Add libsz.so to path:
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
cd ..
cd ..
Copy hdf5-1.14.3 source to the HDF5_nvc_compiler dir:
cp -r hdf5-1.14.3/ ..
In HDF5_nvc_compiler dir, make a new directory called “myhdfstuff”. Without this called folder, compilation is problematic:
cd ..mkdir myhdfstuff
Copy the source files in hdf5-1.14.3 to myhdfstuff:
cp -r hdf5-1.14.3/ myhdfstuff/
Build HDF5 using nvc compiler by Nvidia HPC SDK:
cd myhdfstuffmkdir buildcd buildcmake -G "Unix Makefiles" -DBUILD_SHARED_LIBS:BOOL=ON -DHDF5_BUILD_FORTRAN=YES -DHDF5_BUILD_TOOLS:BOOL=ON -DCMAKE_C_COMPILER=nvc ../hdf5-1.14.3cmake --build . --config Release -j12cpack -C Release CPackConfig.cmake
./HDF5-1.14.3-Linux.sh
When building HDF5 for compilation with GNU, use gcc instead of nvc. For running the last .sh command, you might need to first needed add permissions (chmod +x HDF5-1.14.3-Linux.sh)
Scroll down with enter, accept license with “y”, and write “n” to have it installed in the build directory. Export HDF5 directory to system variables in bashrc:
export PATH=/home/lebedmi2/SOFTWARE/HDF5_nvc_compiler/myhdfstuff/build/HDF_Group/HDF5/1.14.3:$PATH
export LD_LIBRARY_PATH=/home/lebedmi/SOFTWARE/HDF5_nvc_compiler/myhdfstuff/build/HDF_Group/HDF5/1.14.3/lib:$LD_LIBRARY_PATH
4) VASP
tar xvzf vasp.6.4.1.tgz
mv vasp.6.4.1 vasp.6.4.1_GPU_MKL_2
cd vasp.6.4.1_GPU_MKL_2
cd vasp.6.4.1
To run commands without sudo (otherwise you could face problems with paths, see below this section), give rights to the currect user for the vasp.6.4.1 folder (change only the username ‚lebedmi2‘ in the following command. You can check name of the user with command ‚whoami‚):
sudo chown -R lebedmi2 .
Create makefile.include file:
nano makefile.include
Into makefile.include, copy here from: makefile.include.nvhpc_ompi_mkl_omp_acc (https://www.vasp.at/wiki/index.php/Makefile.include.nvhpc_ompi_mkl_omp_acc or find it in arch folder of your VASP). On the first line of makefile.include, add your path to the MKL folder “export MKLROOT=/opt/intel/oneapi/mkl/2023.2.0” to solve this error:
nvfortran-Fatal-MKLROOT not found. Please set the environment variable MKLROOT to the location of Intel’s Math Kernel Libraries (the part of the path to the *.so files that precedes ‚lib/<arch>‘.).
In makefile.include, set the paths to to MKL (MKLROOT) and HDF5 (HDF5_ROOT). Also uncomment all the options for HDF5 and change the CUDA version to match the one installed with your NVIDIA HPC SDK. I have version 12.3. Additionally, verify the architecture code of your graphics card. For an RTX 2060, use the code cc=70 (cc=75). If you are using an RTX 4090, the appropriate code is either cc=89 or cc=90.
FC = mpif90 -acc -gpu=cc70,cc80,cuda12.3 -mp
FCL = mpif90 -acc -gpu=cc70,cc80,cuda12.3 -mp -c++libs
MKLROOT ?= /opt/intel/oneapi/mkl/2023.2.0
HDF5_ROOT ?= /home/lebedmi2/SOFTWARE/TEST_New_HDF5/HDF5_nvc/hdf5-1.14.4-2/build/
Compile vasp:
make DEPS=1 -j
It will take some time. If no error appeared, check if all libraries are correctly set and none are missing:
cd bin
ldd vasp_std
Export number of threads to bashrc:
export OMP_NUM_THREADS=1
If more than one vasp compilation will be installed on the computer, rename the binary:
mv vasp_std vasp_std_gpu_2
Add path vasp_std_gpu_2 to bashrc:
export PATH=/home/lebedmi2/SOFTWARE/VASP/vasp.6.4.1_GPU_MKL_2/bin:$PATH
Note: If you simply run “make test” without any modifications in the testsuite, it will not work on GPU. You need to change the number of mpirun ranks to 1, change the name of the binary for vasp_std (including the absolute path to it), and disable tests which are not GPU supported. For that, write the following to the console:
export VASP_TESTSUITE_EXE_STD="mpirun -np 1 /home/lebedmi2/SOFTWARE/VASP/vasp.6.4.1_GPU_MKL_2/bin/vasp_std_gpu_2"
export VASP_TESTSUITE_EXE_GAM="mpirun -np 1 /home/lebedmi2/SOFTWARE/VASP/vasp.6.4.1_GPU_MKL_2/bin/vasp_std_gpu_2"
export VASP_TESTSUITE_EXE_NCL="mpirun -np 1 /home/lebedmi2/SOFTWARE/VASP/vasp.6.4.1_GPU_MKL_2/bin/vasp_std_gpu_2"
export VASP_TESTSUITE_CUDA=Y
export VASP_TESTSUITE_SKIP_NOCUDA=Y
Then you can run the tests to see if GPU VASP was compiled correctly (write this command within the directory where ‚makefile.include‘ file is):
make test
Run calculations
Make sure that the openMPI used for the compilation is loaded, otherwise source it:
export PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/23.11/comm_libs/12.3/openmpi4/openmpi-4.1.5/bin:$PATH
export LD_LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/23.11/comm_libs/12.3/openmpi4/openmpi-4.1.5/bin:$LD_LIBRARY_PATH
Then within a folder with INCAR, POSCAR, POTCAR, KPOINTS files run the following to start the calculations using the GPU VASP:
mpirun -np 1 vasp_std_gpu_2
Compile CPU version of VASP using NVIDIA HPC-SDK next to the GPU version
Note: When compiling CPU VASP next to the GPU, it is better to use the same compiler: NVIDIA HPC-SDK. Otherwise you could have problems with necessity to have two versions of HDF5. One compiled with nvc and one compiled with gcc/oneAPI/AOCC. Also you could have problem with the openmpi (which you could probably solve if you compile the used openmpi for CPU version as CUDA aware)
Extract the VASP, rename it as you want and create there the makefile.include file with the content from: makefile.include.nvhpc_ompi_mkl_omp makefile.include.nvhpc_ompi_mkl_omp – Vaspwiki:
In makefile.include, change again the roots for MKL and HDF5 and on the first line, write “export MKLROOT=/opt/intel/oneapi/mkl/2023.2.0”.
Then continue as for the GPU compilation.
“make test -j 10” with CPU should work. Try after the compilation is completed. You can wait until it is fully completed. Then modify the binary:
mv vasp_std vasp_std_cpu_2
Do not forget to add its path to bashrc:
cd /home/lebedmi2/DATA/VASP_DATA/test/test_CPU
mpirun -np 1 vasp_std_cpu_2
If HDF5 was successfully incorporated, the output from VASP calculations will include file with .h5 extension.
Solving 'nvfortran not found'
In my ‚makefile‘ (it is in the same location where makefile.include is), I added the following at the first line to check which environmental paths are accessible during compilation:
show-path:
@echo "Current PATH: $$PATH"
When executing ‚make‚, it shows the correct paths (including the path to nvfortran).
When executing ‚sudo make‚, it shows:
Current PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
These are pre-set secure PATHs used by sudo, not the ones I have set in ~/.bashrc needed for the compilation. The other way how to check the secure paths is to write ‚sudo env‚ and see the row starting with ‚PATH‘.
To have access to paths in ~/.bashrc, I would need to run the compilation without sudo, but doing so results in a ‚permission denied‘ error for some files.
You can address this by one of the following options (the first one is the most straightforward):
1) Change the ownership of the folder with VASP and all its content to the user to be able to run ‚make‘ without sudo:
sudo chown -R lebedmi ~/SOFTWARE/VASP/v.6.4.1/vasp.6.4.1
Change the ‚lebedmi‚ for your username (you can check it with command ‚whoami‘) and modify ‚~/SOFTWARE/VASP/v.6.4.1/vasp.6.4.1‚ to the extracted directory with your VASP. Then you can run the compilation as ‚make DEPS=1 -j4‚).
2) First remove preset /mnt/ folders from the environment variables by writingthe following command to the last line of bashrc (otherwise I was getting error: env: ‘Files/NVIDIA’: No such file or directory):
nano ~/.bashrc
export PATH=$(echo "$PATH" | tr ':' '\n' | grep -v '/mnt/c' | tr '\n' ':')
Save and exit, then
source ~/.bashrc
Now you can compile by running the sudo command with ‚env PATH=$PATH‘, e.g.:
sudo env PATH=$PATH make DEPS=1 -j4
3) Write ‚sudo visudo‘ and on the row starting with ‚Defaults secure_path=“…..‘ add the paths you need to have access when running command with ‚sudo‘
Calculation speed comparison
Content of makefile.include for GPU compilation:
export MKLROOT=/opt/intel/oneapi/mkl/2023.2.0
# Default precompiler options
CPP_OPTIONS = -DHOST=\"LinuxNV\" \
-DMPI -DMPI_BLOCK=8000 -Duse_collective \
-DscaLAPACK \
-DCACHE_SIZE=4000 \
-Davoidalloc \
-Dvasp6 \
-Duse_bse_te \
-Dtbdyn \
-Dqd_emulate \
-Dfock_dblbuf \
-D_OPENMP \
-D_OPENACC \
-DUSENCCL -DUSENCCLP2P
CPP = nvfortran -Mpreprocess -Mfree -Mextend -E $(CPP_OPTIONS) $*$(FUFFIX) > $*$(SUFFIX)
# N.B.: you might need to change the cuda-version here
# to one that comes with your NVIDIA-HPC SDK
FC = mpif90 -acc -gpu=cc70,cc80,cuda12.2 -mp
FCL = mpif90 -acc -gpu=cc70,cc80,cuda12.2 -mp -c++libs
FREE = -Mfree
FFLAGS = -Mbackslash -Mlarge_arrays
OFLAG = -fast
DEBUG = -Mfree -O0 -traceback
OBJECTS = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o
LLIBS = -cudalib=cublas,cusolver,cufft,nccl -cuda
# Redefine the standard list of O1 and O2 objects
SOURCE_O1 := pade_fit.o
SOURCE_O2 := pead.o
# For what used to be vasp.5.lib
CPP_LIB = $(CPP)
FC_LIB = nvfortran
CC_LIB = nvc -w
CFLAGS_LIB = -O
FFLAGS_LIB = -O1 -Mfixed
FREE_LIB = $(FREE)
OBJECTS_LIB = linpack_double.o
# For the parser library
CXX_PARS = nvc++ --no_warnings
##
## Customize as of this point! Of course you may change the preceding
## part of this file as well if you like, but it should rarely be
## necessary ...
##
# Specify your NV HPC-SDK installation (mandatory)
#... first try to set it automatically
NVROOT =$(shell which nvfortran | awk -F /compilers/bin/nvfortran '{ print $$1 }')
# If the above fails, then NVROOT needs to be set manually
#NVHPC ?= /opt/nvidia/hpc_sdk
#NVVERSION = 21.11
#NVROOT = $(NVHPC)/Linux_x86_64/$(NVVERSION)
# Software emulation of quadruple precsion (mandatory)
QD ?= $(NVROOT)/compilers/extras/qd
LLIBS += -L$(QD)/lib -lqdmod -lqd
INCS += -I$(QD)/include/qd
# Intel MKL (FFTW, BLAS, LAPACK, and scaLAPACK)
MKLROOT ?= /opt/intel/oneapi/mkl/2023.2.0
LLIBS += -Mmkl -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64
INCS += -I$(MKLROOT)/include/fftw
# HDF5-support (optional but strongly recommended)
CPP_OPTIONS+= -DVASP_HDF5
HDF5_ROOT ?= /home/lebedmi2/SOFTWARE/HDF5_nvc_compiler/myhdfstuff/build/HDF_Group/HDF5/1.14.3
LLIBS += -L$(HDF5_ROOT)/lib -lhdf5_fortran
INCS += -I$(HDF5_ROOT)/include
# For the VASP-2-Wannier90 interface (optional)
#CPP_OPTIONS += -DVASP2WANNIER90
#WANNIER90_ROOT ?= /path/to/your/wannier90/installation
#LLIBS += -L$(WANNIER90_ROOT)/lib -lwannier
# For the fftlib library (experimental)
#CPP_OPTIONS+= -Dsysv
#FCL += fftlib.o
#CXX_FFTLIB = nvc++ -mp --no_warnings -std=c++11 -DFFTLIB_USE_MKL -DFFTLIB_THREADSAFE
#INCS_FFTLIB = -I./include -I$(MKLROOT)/include/fftw
#LIBS += fftlib
#LLIBS += -ldl
Content of makefile.include for CPU compilation:
export MKLROOT=/opt/intel/oneapi/mkl/2023.2.0
# Default precompiler options
CPP_OPTIONS = -DHOST=\"LinuxNV\" \
-DMPI -DMPI_BLOCK=8000 -Duse_collective \
-DscaLAPACK \
-DCACHE_SIZE=4000 \
-Davoidalloc \
-Dvasp6 \
-Duse_bse_te \
-Dtbdyn \
-Dqd_emulate \
-Dfock_dblbuf \
-D_OPENMP
CPP = nvfortran -Mpreprocess -Mfree -Mextend -E $(CPP_OPTIONS) $*$(FUFFIX) > $*$(SUFFIX)
FC = mpif90 -mp
FCL = mpif90 -mp -c++libs
FREE = -Mfree
FFLAGS = -Mbackslash -Mlarge_arrays
OFLAG = -fast
DEBUG = -Mfree -O0 -traceback
OBJECTS = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o
# Redefine the standard list of O1 and O2 objects
SOURCE_O1 := pade_fit.o
SOURCE_O2 := pead.o
# For what used to be vasp.5.lib
CPP_LIB = $(CPP)
FC_LIB = nvfortran
CC_LIB = nvc -w
CFLAGS_LIB = -O
FFLAGS_LIB = -O1 -Mfixed
FREE_LIB = $(FREE)
OBJECTS_LIB = linpack_double.o
# For the parser library
CXX_PARS = nvc++ --no_warnings
##
## Customize as of this point! Of course you may change the preceding
## part of this file as well if you like, but it should rarely be
## necessary ...
##
# Specify your NV HPC-SDK installation (mandatory)
#... first try to set it automatically
NVROOT =$(shell which nvfortran | awk -F /compilers/bin/nvfortran '{ print $$1 }')
# If the above fails, then NVROOT needs to be set manually
#NVHPC ?= /opt/nvidia/hpc_sdk
#NVVERSION = 21.11
#NVROOT = $(NVHPC)/Linux_x86_64/$(NVVERSION)
# Software emulation of quadruple precsion (mandatory)
QD ?= $(NVROOT)/compilers/extras/qd
LLIBS += -L$(QD)/lib -lqdmod -lqd
INCS += -I$(QD)/include/qd
# Intel MKL (FFTW, BLAS, LAPACK, and scaLAPACK)
MKLROOT ?= /opt/intel/oneapi/mkl/2023.2.0
LLIBS += -Mmkl -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64
INCS += -I$(MKLROOT)/include/fftw
# HDF5-support (optional but strongly recommended)
CPP_OPTIONS+= -DVASP_HDF5
HDF5_ROOT ?= /home/lebedmi2/SOFTWARE/HDF5_nvc_compiler/myhdfstuff/build/HDF_Group/HDF5/1.14.3
LLIBS += -L$(HDF5_ROOT)/lib -lhdf5_fortran
INCS += -I$(HDF5_ROOT)/include
# For the VASP-2-Wannier90 interface (optional)
#CPP_OPTIONS += -DVASP2WANNIER90
#WANNIER90_ROOT ?= /path/to/your/wannier90/installation
#LLIBS += -L$(WANNIER90_ROOT)/lib -lwannier
# For the fftlib library (experimental)
#CPP_OPTIONS+= -Dsysv
#FCL += fftlib.o
#CXX_FFTLIB = nvc++ -mp --no_warnings -std=c++11 -DFFTLIB_USE_MKL -DFFTLIB_THREADSAFE
#INCS_FFTLIB = -I./include -I$(MKLROOT)/include/fftw
#LIBS += fftlib
#LLIBS += -ldl