Standard VASP compilation with HDF5 support

Standard VASP (6.4.1) compilation using GCC compilers with HDF5 support in WSL2 Ubuntu

Full description for the video: https://youtu.be/QfAZ5QERZ30

Compiled version: VASP 6.4.1
(10/11/2023)

Contact: lebedmi2@cvut.cz

Official documentation for compiling VASP: https://www.vasp.at/wiki/index.php/Installing_VASP.6.X.X
Official documentation for makefiles: https://www.vasp.at/wiki/index.php/Makefile.include

Tested PC:

Intel(R) Core(TM) i5-14600K 3.50 GHz
RTX 2060 SUPER
Kingston FURY 32GB KIT DDR5 6000MHz CL32 Renegade
WSL2 Ubuntu version: 22.04

Steps:

1) Standardly compile OpenMPI using GCC compilers.

2) Standardly compile HDF5 using GCC compilers.
3) Compile VASP using GCC compilers.

Prerequisites:

Not all are necessary but might be useful:

sudo apt update
sudo apt upgrade
sudo apt-get install build-essential cmake cmake-curses-gui libopenmpi-dev openmpi-bin libfftw3-dev libopenblas-dev libscalapack-mpi-dev

1) Compile OpenMPI standardly (using GCC compilers)

Download OpenMPI (use wget command or download from here https://www.open-mpi.org/software/ompi/v5.0/) and compile:

mkdir OpenMPI_GCC
cd OpenMPI_GCC

wget https://download.open-mpi.org/release/open-mpi/v5.0/openmpi-5.0.0.tar.gz
tar xvzf openmpi-5.0.0.tar.gz
cd openmpi-5.0.0
mkdir build
./configure --prefix=/home/lebedmi2/SOFTWARE/OpenMPI/OpenMPI_GCC/openmpi-5.0.0/build
make install -j

Activate it (either write the following into the last line of bashrc for permanent activation or write it directly into the console for temporal activation):

export PATH=/home/lebedmi2/SOFTWARE/OpenMPI/OpenMPI_GCC/openmpi-5.0.0/build/bin:$PATH
export LD_LIBRARY_PATH=/home/lebedmi2/SOFTWARE/OpenMPI/OpenMPI_GCC/openmpi-5.0.0/build/lib:$LD_LIBRARY_PATH

2) Compile HDF5 standardly (using GCC compilers)

Create a new folder where HDF5 will be compiled:

mkdir HDF5_gcc
cd HDF5_gcc

Install ZLIB:

sudo apt install zlib1g

Download into HDF5_gcc folder the newest version of HDF5 from https://www.hdfgroup.org/downloads/hdf5/source-code/# , extract it, and create a new folder called ‚build‘ into which the compilation will be made.

tar xvzf hdf5-1.14.4-2.tar.gz
cd hdf5-1.14.4-2/
mkdir build

Configurate HDF5 with compilers from oneAPI:

CC=gcc CXX=gcc++ ./configure --prefix=/home/lebedmi2/SOFTWARE/TEST_New_HDF5/HDF5_intel/hdf5-1.14.4-2/build --enable-fortran --enable-cxx --enable-shared

Then build and install it:

make -j
make install -j

Before compiling VASP or running the simulations with this version, make sure to either permanently or temporally export HDF5 directory to system variables:

export PATH=/home/lebedmi2/SOFTWARE/TEST_New_HDF5/HDF5_gcc/hdf5-1.14.4-2/build/bin:$PATH
export LD_LIBRARY_PATH=/home/lebedmi2/SOFTWARE/TEST_New_HDF5/HDF5_gcc/hdf5-1.14.4-2/build/lib:$LD_LIBRARY_PATH

~~OLDER VERSION of HDF5 with Cmake:~~

~~Create a new folder where HDF5 will be compiled:~~

mkdir HDF5_GCC
cd HDF5_GCC

~~Download into this folder a Cmake version of HDF5 from https://www.hdfgroup.org/downloads/hdf5/source-code/#:~~

tar xvzf CMake-hdf5-1.14.3.tar.gz
cd CMake-hdf5-1.14.3/

~~Compile LIBAEC and ZLib downloaded with HDF5:~~

tar xvzf LIBAEC.tar.gz
tar xvzf ZLib.tar.gz
cd libaec-v1.0.6
mkdir build
cd build
cmake ..
sudo make install
cd ..
cd ..
cd zlib-1.3
mkdir build
cd build
cmake ..
sudo make install

~~Add libsz.so to path:~~

export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

cd ..
cd ..

~~Copy hdf5-1.14.3 source to the HDF5_GCC dir:~~

cp -r hdf5-1.14.3/ ..

~~In HDF5_GCC dir, make a new directory called “myhdfstuff”. Without this called folder, compilation is problematic:~~

cd ..
mkdir myhdfstuff

~~Copy the source files in hdf5-1.14.3 to myhdfstuff:~~

cp -r hdf5-1.14.3/ myhdfstuff/

~~Build HDF5:~~

cd myhdfstuff
mkdir build
cd build
cmake -G "Unix Makefiles" -DBUILD_SHARED_LIBS:BOOL=ON -DHDF5_BUILD_FORTRAN=YES -DHDF5_BUILD_TOOLS:BOOL=ON ../hdf5-1.14.3 
cmake --build . --config Release -j
cpack -C Release CPackConfig.cmake
./HDF5-1.14.3-Linux.sh

Scroll down with enter, accept license with “y”, and write “n” to have it installed in the build directory.
Before running the simulations with this version, make sure to either permanently or temporally export HDF5 directory to system variables:

export PATH=/home/lebedmi2/SOFTWARE/HDF5_GCC/myhdfstuff/build/HDF_Group/HDF5/1.14.3/bin:$PATH
export LD_LIBRARY_PATH=/home/lebedmi2/SOFTWARE/HDF5_GCC/myhdfstuff/build/HDF_Group/HDF5/1.14.3/lib:$LD_LIBRARY_PATH

Compile VASP standardly (using GCC compilers)

tar xvzf vasp.6.4.1.tgz
mv vasp.6.4.1 vasp.6.4.1_GCC
cd vasp.6.4.1_GCC
cd vasp.6.4.1

To run commands without sudo (otherwise you could face problems with paths, see below this section), give rights to the currect user for the vasp.6.4.1 folder (change only the username ‚lebedmi2‘ in the following command. You can check name of the user with command ‚whoami‚):

sudo chown -R lebedmi2 .

Create makefile.include file:

nano makefile.include

Into makefile.include, copy here from: makefile.include.gnu_omp (https://www.vasp.at/wiki/index.php/Makefile.include.gnu_omp or find it in arch folder of your VASP).

In makefile.include, set the path to OpenBLAS, scaLAPACK, FFTW3, and HDF5 (also uncomment all the options for HDF5). The lines that I needed to change are shown below:

OPENBLAS_ROOT ?= /usr
SCALAPACK_ROOT ?= /usr
SCALAPACK   = -L$(SCALAPACK_ROOT)/lib/x86_64-linux-gnu -lscalapack-openmpi
FFTW_ROOT  ?= /usr
HDF5_ROOT  ?= /home/lebedmi2/SOFTWARE/TEST_New_HDF5/HDF5_gcc/hdf5-1.14.4-2/build/

Compile vasp:

make DEPS=1 -j

It will take some time. If no error appeared, check if all libraries are correctly set and none are missing:

cd bin
ldd vasp_std

Export number of threads to bashrc:

export OMP_NUM_THREADS=1

Try if the compiled version is working (write the following command in the same directory where the makefile.include is. It should give computed values and no errors):

make test -j

If more than one vasp compilation will be installed on the computer, rename the binary:

mv vasp_std vasp_gcc

Add path vasp_gcc to bashrc:

export PATH=/home/lebedmi2/SOFTWARE/VASP/vasp.6.4.1_GCC/vasp.6.4.1/bin:$PATH

Run the simulations with VASP:

Before running the simulations, all the following should be exported (if they are not set permanently in ~/.bashrc). In my case:

#OpenMPI standardly compiled using GCC
export PATH=/home/lebedmi2/SOFTWARE/OpenMPI/OpenMPI_GCC/openmpi-5.0.0/build/bin:$PATH
export LD_LIBRARY_PATH=/home/lebedmi2/SOFTWARE/OpenMPI/OpenMPI_GCC/openmpi-5.0.0/build/lib:$LD_LIBRARY_PATH
#HDF5 standardly compiled using GCC
export PATH=/home/lebedmi2/SOFTWARE/TEST_New_HDF5/HDF5_gcc/hdf5-1.14.4-2/build//bin:$PATH
export LD_LIBRARY_PATH=/home/lebedmi2/SOFTWARE/TEST_New_HDF5/HDF5_gcc/hdf5-1.14.4-2/build//lib:$LD_LIBRARY_PATH

Then run with, e.g.:

mpirun -np 4 vasp_gcc

To run on all processors and threads:

mpirun --use-hwthread -np N vasp_gcc

Encountered problems

When running the computation on smaller number of cores, I needed to solve the following error by writting into console:

ulimit -s unlimited

It removes the maximum size restriction on the stack memory for programs in a Unix-like system, allowing them to use as much stack space as needed.

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
libc.so.6 00007F0D603C9520 Unknown Unknown Unknown
vasp_intel 000000000099B52F Unknown Unknown Unknown
vasp_intel 0000000001209389 Unknown Unknown Unknown
vasp_intel 00000000012A28BB Unknown Unknown Unknown
vasp_intel 0000000001DE5F26 Unknown Unknown Unknown
vasp_intel 0000000001DBD0EA Unknown Unknown Unknown
vasp_intel 000000000041CCED Unknown Unknown Unknown
libc.so.6 00007F0D603B0D90 Unknown Unknown Unknown
libc.so.6 00007F0D603B0E40 __libc_start_main Unknown Unknown
vasp_intel 000000000041CC05 Unknown Unknown Unknown

Solving possible problem with PATHs

In my ‚makefile‘ (it is in the same location where makefile.include is), I added the following at the first line to check which environmental paths are accessible during compilation:

show-path:
      @echo "Current PATH: $$PATH"

When executing ‚make‚, it shows the correct paths (including the path to nvfortran).
When executing ‚sudo make‚, it shows:
Current PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
These are pre-set secure PATHs used by sudo, not the ones I have set in ~/.bashrc needed for the compilation. The other way how to check the secure paths is to write ‚sudo env‚ and see the row starting with ‚PATH‘.

To have access to paths in ~/.bashrc, I would need to run the compilation without sudo, but doing so results in a ‚permission denied‘ error for some files.

You can address this by one of the following options (the first one is the most straightforward):

1) Change the ownership of the folder with VASP and all its content to the user to be able to run ‚make‘ without sudo:

sudo chown -R lebedmi ~/SOFTWARE/VASP/v.6.4.1/vasp.6.4.1

Change the ‚lebedmi‚ for your username (you can check it with command ‚whoami‘) and modify ‚~/SOFTWARE/VASP/v.6.4.1/vasp.6.4.1‚ to the extracted directory with your VASP. Then you can run the compilation as ‚make DEPS=1 -j4‚).

2) First remove preset /mnt/ folders from the environment variables by writingthe following command to the last line of bashrc (otherwise I was getting error: env: ‘Files/NVIDIA’: No such file or directory):

nano ~/.bashrc
export PATH=$(echo "$PATH" | tr ':' '\n' | grep -v '/mnt/c' | tr '\n' ':')

Save and exit, then

source ~/.bashrc

Now you can compile by running the sudo command with ‚env PATH=$PATH‘, e.g.:

sudo env PATH=$PATH make DEPS=1 -j4

3) Write ‚sudo visudo‘ and on the row starting with ‚Defaults secure_path=“…..‘ add the paths you need to have access when running command with ‚sudo‘

Calculation speed comparison

---> HERE <---

Content of makefile.include for VASP Intel MKL compilation:

# Default precompiler options
CPP_OPTIONS = -DHOST=\"LinuxGNU\" \
              -DMPI -DMPI_BLOCK=8000 -Duse_collective \
              -DscaLAPACK \
              -DCACHE_SIZE=4000 \
              -Davoidalloc \
              -Dvasp6 \
              -Duse_bse_te \
              -Dtbdyn \
              -Dfock_dblbuf \
              -D_OPENMP

CPP         = gcc -E -C -w $*$(FUFFIX) >$*$(SUFFIX) $(CPP_OPTIONS)

FC          = mpif90 -fopenmp
FCL         = mpif90 -fopenmp

FREE        = -ffree-form -ffree-line-length-none

FFLAGS      = -w -ffpe-summary=none

OFLAG       = -O2
OFLAG_IN    = $(OFLAG)
DEBUG       = -O0

OBJECTS     = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o
OBJECTS_O1 += fftw3d.o fftmpi.o fftmpiw.o
OBJECTS_O2 += fft3dlib.o

# For what used to be vasp.5.lib
CPP_LIB     = $(CPP)
FC_LIB      = $(FC)
CC_LIB      = gcc
CFLAGS_LIB  = -O
FFLAGS_LIB  = -O1
FREE_LIB    = $(FREE)

OBJECTS_LIB = linpack_double.o

# For the parser library
CXX_PARS    = g++
LLIBS       = -lstdc++

##
## Customize as of this point! Of course you may change the preceding
## part of this file as well if you like, but it should rarely be
## necessary ...
##

# When compiling on the target machine itself, change this to the
# relevant target when cross-compiling for another architecture
VASP_TARGET_CPU ?= -march=native
FFLAGS     += $(VASP_TARGET_CPU)

# For gcc-10 and higher (comment out for older versions)
FFLAGS     += -fallow-argument-mismatch

# BLAS and LAPACK (mandatory)
OPENBLAS_ROOT ?= /usr
BLASPACK    = -L$(OPENBLAS_ROOT)/lib -lopenblas

# scaLAPACK (mandatory)
SCALAPACK_ROOT ?= /usr
SCALAPACK   = -L$(SCALAPACK_ROOT)/lib/x86_64-linux-gnu -lscalapack-openmpi

LLIBS      += $(SCALAPACK) $(BLASPACK)

# FFTW (mandatory)
FFTW_ROOT  ?= /usr
LLIBS      += -L$(FFTW_ROOT)/lib -lfftw3 -lfftw3_omp
INCS       += -I$(FFTW_ROOT)/include

# HDF5-support (optional but strongly recommended)
CPP_OPTIONS+= -DVASP_HDF5
HDF5_ROOT  ?= /home/lebedmi2/SOFTWARE/HDF5_GCC_compiler/myhdfstuff/build/HDF_Group/HDF5/1.14.3
LLIBS      += -L$(HDF5_ROOT)/lib -lhdf5_fortran
INCS       += -I$(HDF5_ROOT)/include

# For the VASP-2-Wannier90 interface (optional)
#CPP_OPTIONS    += -DVASP2WANNIER90
#WANNIER90_ROOT ?= /path/to/your/wannier90/installation
#LLIBS          += -L$(WANNIER90_ROOT)/lib -lwannier

# For the fftlib library (recommended)
#CPP_OPTIONS+= -Dsysv
#FCL        += fftlib.o
#CXX_FFTLIB  = g++ -fopenmp -std=c++11 -DFFTLIB_THREADSAFE
#INCS_FFTLIB = -I./include -I$(FFTW_ROOT)/include
#LIBS       += fftlib
#LLIBS      += -ldl