Numerical behavior of NVIDIA tensor cores
2021 (English)In: PeerJ Computer Science, E-ISSN 2376-5992, Vol. 7, p. 1-19, article id e330Article in journal (Refereed) Published
Abstract [en]
We explore the floating-point arithmetic implemented in the NVIDIA tensor cores, which are hardware accelerators for mixed-precision matrix multiplication available on the Volta, Turing, and Ampere microarchitectures. Using Volta V100, Turing T4, and Ampere A100 graphics cards, we determine what precision is used for the intermediate results, whether subnormal numbers are supported, what rounding mode is used, in which order the operations underlying the matrix multiplication are performed, and whether partial sums are normalized. These aspects are not documented by NVIDIA, and we gain insight by running carefully designed numerical experiments on these hardware units. Knowing the answers to these questions is important if one wishes to: (1) accurately simulate NVIDIA tensor cores on conventional hardware; (2) understand the differences between results produced by code that utilizes tensor cores and code that uses only IEEE 754-compliant arithmetic operations; and (3) build custom hardware whose behavior matches that of NVIDIA tensor cores. As part of this work we provide a test suite that can be easily adapted to test newer versions of the NVIDIA tensor cores as well as similar accelerators from other vendors, as they become available. Moreover, we identify a non-monotonicity issue affecting floating point multi-operand adders if the intermediate results are not normalized after each step.
Place, publisher, year, edition, pages
PeerJ, Inc , 2021. Vol. 7, p. 1-19, article id e330
Keywords [en]
NVIDIA V100 GPU, NVIDIA T4 GPU, Tensor core, Dot product, Matrix multiply-accumulate, Floating-point arithmetic, Half precision, Binary16, IEEE 754 arithmetic, NVIDIA A100 GPU
National Category
Computational Mathematics
Research subject
Mathematics
Identifiers
URN: urn:nbn:se:oru:diva-88210DOI: 10.7717/peerj-cs.330ISI: 000620811800001PubMedID: 33816984Scopus ID: 2-s2.0-85101610379OAI: oai:DiVA.org:oru-88210DiVA, id: diva2:1513290
Note
Funding Agencies:
UK Research & Innovation (UKRI)
Engineering & Physical Sciences Research Council (EPSRC) EP/P020720/1
UK Research & Innovation (UKRI)
Engineering & Physical Sciences Research Council (EPSRC) EP/P020720/1
Royal Society of London
European Commission
2020-12-292020-12-292021-04-13Bibliographically approved