Comparing Solid State DFT Codes, Basis Sets and Potentials

This web page offers all necessary information to determine the Δ-value between two solid state DFT codes within the PBE formalism. Δ is defined as the root-mean-square energy difference between the equations of state of the two codes, averaged over all crystals in a purely elemental benchmark set. This quantity can act as an accuracy-based guideline when selecting a solid state DFT code for a specific task. A README has been provided in the zip-file (see below), as well as the required input and script files. In addition, the code comparison database has been implemented in ASE. Further information and a discussion of results can be found in the following papers:

Kurt Lejaeghere, Veronique Van Speybroeck, Guido Van Oost and Stefaan Cottenier, Error estimates for solid-state density-functional theory predictions: an overview by means of the ground-state elemental crystals, Critical Reviews in Solid State and Materials Sciences 39, 1-24 (2014). (Open Access)

Kurt Lejaeghere, Gustav Bihlmayer, Torbjörn Björkman, Peter Blaha, Stefan Blügel, Volker Blum, Damien Caliste, Ivano E. Castelli, Stewart J. Clark, Andrea Dal Corso, Stefano de Gironcoli, Thierry Deutsch, John Kay Dewhurst, Igor Di Marco, Claudia Draxl, Marcin Dułak, Olle Eriksson, José A. Flores-Livas, Kevin F. Garrity, Luigi Genovese, Paolo Giannozzi, Matteo Giantomassi, Stefan Goedecker, Xavier Gonze, Oscar Grånäs, E. K. U. Gross, Andris Gulans, François Gygi, D. R. Hamann, Phil J. Hasnip, N. A. W. Holzwarth, Diana Iuşan, Dominik B. Jochym, François Jollet, Daniel Jones, Georg Kresse, Klaus Koepernik, Emine Küçükbenli, Yaroslav O. Kvashnin, Inka L. M. Locht, Sven Lubeck, Martijn Marsman, Nicola Marzari, Ulrike Nitzsche, Lars Nordström, Taisuke Ozaki, Lorenzo Paulatto, Chris J. Pickard, Ward Poelmans, Matt I. J. Probert, Keith Refson, Manuel Richter, Gian-Marco Rignanese, Santanu Saha, Matthias Scheffler, Martin Schlipf, Karlheinz Schwarz, Sangeeta Sharma, Francesca Tavazza, Patrik Thunström, Alexandre Tkatchenko, Marc Torrent, David Vanderbilt, Michiel J. van Setten, Veronique Van Speybroeck, John M. Wills, Jonathan R. Yates, Guo-Xu Zhang, Stefaan Cottenier, Reproducibility in density functional theory calculations of solids, Science 351 (6280), aad3000 (2016).
Click here to freely access a copy of the manuscript: abstract, full text, pdf.

All codes that have been assessed up until now, are mentioned in the following table. Where available, the full computatonal data and settings can be accessed by means of the file icon on the right. Please click the row of the code that you wish to see as a reference (WIEN2k is the default). Note that clicking a hyperlink takes you to the corresponding website rather than reorder the table.

Code developers and/or experts are invited to report the Δ-value of their code to us. We will try to keep this list up to date.

Code	Version	Basis	Electron treatment	Δ-value	References & Data
WIEN2k	13.1	LAPW/APW+lo	all-electron	0 meV/atom	S. Cottenier [16]
FHI-aims	081213	tier2 numerical orbitals	all-electron (relativistic atomic_zora scalar)	0.2 meV/atom	ASE [2,16] (data)
Exciting	development version	LAPW+xlo	all-electron	0.2 meV/atom	Exciting [10,16] (data)
VASP	5.2.12	plane waves	PAW 2015 GW-ready (5.4)	0.3 meV/atom	K. Lejaeghere [16]
FHI-aims	081213	tier2 numerical orbitals	all-electron (relativistic zora scalar 1e-12)	0.3 meV/atom	ASE [2] (data)
Quantum ESPRESSO	5.1	plane waves	SSSP Accuracy (mixed NC/US/PAW potential library)	0.3 meV/atom	QuantumESPRESSO [12,16] (data)
Elk	3.1.5	APW+lo	all-electron	0.3 meV/atom	Elk [14,16]
ABINIT	7.8.2	plane waves	PAW JTH v1.0	0.4 meV/atom	F. Jollet and M. Torrent
FLEUR	0.26	LAPW (+lo)	all-electron	0.4 meV/atom	FLEUR [9,16]
Quantum ESPRESSO	5.1	plane waves	SSSP Efficiency (mixed NC/US/PAW potential library)	0.4 meV/atom	QuantumESPRESSO [12] (data)
CASTEP	19.1.1	plane waves	OTFG CASTEP 19.1	0.4 meV/atom	CASTEP [7,16]
CASTEP	9.0	plane waves	OTFG CASTEP 9.0	0.5 meV/atom	CASTEP [7,16]
ABINIT	7.7.3	plane waves	PAW JTH v0.2	0.5 meV/atom	F. Jollet and M. Torrent [16]
FHI-aims	081213	tight numerical orbitals	all-electron (relativistic atomic_zora scalar)	0.5 meV/atom	ASE [2,16] (data)
VASP	5.2.12	plane waves	PAW 2012	0.6 meV/atom	K. Lejaeghere [16]
ABINIT	7.11.8	plane waves	pseudo_dojo_ONCVPSP 0.1 norm-conserving	0.6 meV/atom	ABINIT [13,16]
Questaal	7.14.1	LMTO	all-electron	0.6 meV/atom	Questaal [18]
VASP	5.2.12	plane waves	PAW 2012 GW-ready	0.7 meV/atom	K. Lejaeghere
VASP	5.2.12	plane waves	PAW 2015 (5.4)	0.7 meV/atom	K. Lejaeghere
RSPt	1672	LMTO	all-electron	0.8 meV/atom	RSPt [6,16]
ABINIT	7.10.2	plane waves	PAW GBRV 1.2	0.8 meV/atom	ASE [2,16]
Quantum ESPRESSO	5.1	plane waves	PAW PSLibrary 1.0.0	0.8 meV/atom	QuantumESPRESSO [12,16]
Quantum ESPRESSO	5.1	plane waves	GBRV 1.2 ultrasoft	0.9 meV/atom	ASE [2,16]
ABINIT	7.2.0	plane waves	PAW GBRV 1.0 (v1.01 for O and N)	0.9 meV/atom	F. Jollet et al. [3]
FPLO	14.00	enhanced local orbitals	all-electron	0.9 meV/atom	FPLO [8,16]
FPLO	14.00	enhanced local orbitals + fixed compact support radius	all-electron	0.9 meV/atom	FPLO [8,16]
Quantum ESPRESSO	5.1	plane waves	GBRV 1.4 ultrasoft	0.9 meV/atom	ASE [2,16]
CASTEP	17.2.1	plane waves	GBRV 1.5 ultrasoft	0.9 meV/atom	CASTEP [7]
BigDFT	1.7.6	Daubechies wavelets	HGHk-semicore and NLCC 2015 norm-conserving	1.0 meV/atom	BigDFT [11,16]
CASTEP	9.0	plane waves	GBRV 1.4 ultrasoft	1.0 meV/atom	CASTEP [7,16]
CASTEP	19.1.1	plane waves	OTFG ONCVPSP norm-conserving 19.1	1.1 meV/atom	CASTEP [7,16]
ABINIT	7.5.3	plane waves	PAW JTH	1.2 meV/atom	F. Jollet et al. [3,16]
BigDFT	1.7.6	Daubechies wavelets	HGHk-semicore and NLCC 2013 norm-conserving	1.2 meV/atom	BigDFT [11]
ABINIT	7.10.2	plane waves	GPAW PAW 0.9 (80 Ha cut-off)	1.3 meV/atom	ASE [2,16]
Quantum ESPRESSO	5.1	plane waves	Schlipf-Gygi ONCVPSP 2015-01-24 norm-conserving	1.3 meV/atom	ASE [2,16]
CASTEP	9.0	plane waves	Schlipf-Gygi ONCVPSP 2015-05-20 norm-conserving	1.4 meV/atom	CASTEP [7,16]
CASTEP	9.0	plane waves	Schlipf-Gygi ONCVPSP 2015-01-24 norm-conserving	1.4 meV/atom	CASTEP [7,16]
GPAW	0.10.0	plane waves	PAW 0.9	1.5 meV/atom	ASE [2,16]
GPAW	0.10.0	grid-based	PAW 0.9	1.5 meV/atom	ASE [2]
Octopus	8.0	grid-based	Schlipf-Gygi ONCVPSP 2015-01-24 norm-conserving	1.6 meV/atom	Octopus [17]
Quantum ESPRESSO	5.0.2	plane waves	PAW PSLibrary 0.3.1	1.8 meV/atom	Küçükbenli et al. [5,16]
ATK	2016	LCAO High	Schlipf-Gygi ONCVPSP 2015-05-20 norm-conserving	1.8 meV/atom	ATK/QuantumWise [15]
CASTEP	19.1.1	plane waves	OTFG ultrasoft for high-throughput QC5	1.9 meV/atom	CASTEP [7,16]
OpenMX	3.7	pseudo-atomic orbitals	Morrison-Bylander-Kleinman norm-conserving (2013)	2.0 meV/atom	OpenMX [4,16]
VASP	5.2.2	plane waves	PAW 2007	2.0 meV/atom	K. Lejaeghere et al. [1,16]
ABINIT	7.10.2	plane waves	HGHk norm-conserving, semicore if available	2.1 meV/atom	ASE [2,16]
ATK	2015	pseudo-atomic orbitals	Morrison-Bylander-Kleinman norm-conserving (2013)	2.2 meV/atom	ATK/QuantumWise [15]
FHI-aims	081213	light numerical orbitals	all-electron (relativistic atomic_zora scalar)	2.4 meV/atom	ASE [2] (data)
CASTEP	8.0	plane waves	OTFG CASTEP 7.0	2.5 meV/atom	CASTEP [7,16]
CASTEP	8.0	plane waves	OTFG Materials Studio	3.4 meV/atom	CASTEP [7]
GPAW	0.8.0	grid-based	PAW 0.6	3.8 meV/atom	K. Lejaeghere et al. [1,16]
FPLO	14.00	default local orbitals	all-electron	3.9 meV/atom	FPLO [8,16]
Dacapo	2.7.16	plane waves	Vanderbilt ultrasoft version 2	6.2 meV/atom	ASE [2,16]
ABINIT	7.10.2	plane waves	HGHk norm-conserving	6.3 meV/atom	ASE [2]
CASTEP	8.0	plane waves	Vanderbilt ultrasoft	6.4 meV/atom	CASTEP [7,16]
ABINIT	7.6.4	plane waves	Troullier-Martins norm-conserving (FHI)	13.4 meV/atom	ASE [2,16]

Select Elements

Hovering over the Δ-value for a particular code displays the standard deviation over all elements and clicking it displays an overview of the deviations for each individual element (in meV/atom). You can also calculate Δ for a limited number of elements, using the tool "Select Elements" above. The equation of state data for each code are included in the Delta calculation package that can be downloaded at the bottom of this page ('history' archive).

For questions or to report a Δ-value, please contact Kurt.Lejaeghere@UGent.be or Stefaan.Cottenier@UGent.be.

Updates

March 25 2016:

The results of this large-scale collaboration have now appeared in Science. Thanks to all contributors!

You can now access the computational settings and results for each code, by clicking the file icon to the right of each code.

September 24 2015:

The Delta package download now also contains primCIFs.tar.gz, an archive with the primitive unit cells of the 71 investigated crystals. Thanks to Andris Gulans for verifying these files!

September 14 2015:

At the Psi-k 2015 meeting, the main results of the Δ-project were presented, and 18 of the many collaborators met on stage. See the video (left) for a 15-minute summary about the interpretation of the data set on this page. The group picture (right) shows from left to right: (front row) Nicola Marzari, Phil Hasnip, Stefaan Cottenier, Andris Gulans, Michiel van Setten, Sven Lubeck, Kay Dewhurst; (back row) Santanu Saha, François Gygi, Kurt Lejaeghere, Gian-Marco Rignanese, Peter Blaha, Kevin Garrity, Marc Torrent, Matt Probert, Gustav Bihlmayer, Keith Refson, José Flores-Livas.

May 9 2014:
Based on feedback from developers of codes en potentials, we have thoroughly modified the Δ-concept (version 3.0). You can now select any code to serve as a reference (by clicking on it in the table below), rather than only WIEN2k. The formula for Δ has also been symmetrized and is now the same for code 1 compared to code 2 as for code 2 compared to code 1. Finally, the WIEN2k data have been reassessed with rigorous accuracy settings and very small muffin-tin radii.

Future plans

We aim to extend the Δ-benchmark to multicomponent materials and more complex properties.

References

[1] K. Lejaeghere, V. Van Speybroeck, G. Van Oost and S. Cottenier, "Error estimates for solid-state density-functional theory predictions: an overview by means of the ground-state elemental crystals", Critical Reviews in Solid State and Materials Sciences 39, 1-24 (2014).
(Open Access)
[2] Marcin Dułak by means of ASE (2012-2015).
[3] F. Jollet, M. Torrent and N. Holzwarth, "Generation of Projector Augmented-Wave atomic data: A 71 element validated table in the XML format", Computer Physics Communications 185, 1246-1254 (2014).
[4] Taisuke Ozaki by means of OpenMX (2013).
[5] E. Küçükbenli et al., "Projector augmented-wave and all-electron calculations a cross the periodic table: a comparison of structural and energetic properties", arXiv:1404.3015 (2014).
[6] Torbjörn Björkman and John M. Wills by means of RSPt (2014).
[7] Chris Pickard and Keith Refson by means of CASTEP (2014-2019).
[8] Klaus Koepernik by means of FPLO (2014).
[9] Gustav Bihlmayer by means of FLEUR (2015).
[10] Andris Gulans and Sven Lubeck by means of Exciting (2015).
[11] Damien Caliste, Thierry Deutsch, Luigi Genovese and Santanu Saha by means of BigDFT (2015).
[12] Ivano Eligio Castelli by means of QuantumESPRESSO (2015).
[13] Matteo Giantomassi and Michiel J. van Setten by means of ABINIT (2015).
[14] José A. Flores Livas and John Kay Dewhurst by means of Elk (2015).
[15] Troels Markussen by means of ATK/QuantumWise (2015).
[16] K. Lejaeghere et al., "Reproducibility in density functional theory calculations of solids", Science 351 (6280), aad3000 (2016).
[17] Nicolas Tancogne-Dejean by means of Octopus (2018).
[18] Jerome Jackson, Dimitar Pashov and Mark van Schilfgaarde by means of Questaal (2019).