AI precision Playground

FP4 / FP8 / BF16 / FP16 / TF32 / FP32 / FP64 / MPFR-style high-precision demos

This page lets you explore low precision used in AI, mixed-precision accumulation, scaling, and comparisons with high-precision reference values.

Function Quantization demo

AI-oriented sample functions

Start point of x

End point of x

Divisions precision formats to compare

Scaling Rounding mode

Observation points
Observe where each quantized curve departs from the reference value (thick line), and use the lower absolute-error graph (log scale) to see which format has larger error. Compare how block-wise scaling reduces error near outliers.

In this single-file version, the reference value is computed in JavaScript double precision. Use the MPFR Arbitrary-precision graph tab to generate tables using true MPFR evaluation.

Representative values table

Stable Softmax demo

inputVector input quantization format inputScaling

Observation points
When large values around 1000 are used, the naive version can overflow in exp, whereas the max-shift version remains stable. Lower the input format to FP8 and check the |error| column in the table to see how much the probabilities change.

Softmax algorithm

The input vector \(\mathbf{x}=(x_1,\ldots,x_n)\) is converted into a probability distribution.

y_i = \frac{\exp(x_i)}{\sum_{j=1}^{n}\exp(x_j)}

However, when \(x_i\) is large, \(\exp(x_i)\) overflows, so implementations subtract the maximum value \(m=\max_j x_j\).

y_i = \frac{\exp(x_i-m)}{\sum_{j=1}^{n}\exp(x_j-m)}

This transformation is mathematically equivalent, but it is much more stable in numerical computation.

Dot Product and Accumulation demo

Vector length Distribution inputformat Accumulator format

Observation points
Use the partial-sum graph to see that partial sums can collapse when the accumulator remains low precision, while an FP32 accumulator follows the reference value more closely. The 'uniform + outlier' distribution highlights the weakness of low-precision inputs.

Dot Product and Accumulation

The dot product is a fundamental operation in matrix multiplication and neural networks.

s = \sum_{i=1}^{n} x_i y_i

AI accelerators often store inputs \(x_i,y_i\) in low precision such as FP8 or FP16, while keeping the accumulated sum \(s\) in a wider format such as FP32.

s_k = \operatorname{round}_{\mathrm{acc}}\left(s_{k-1} + \operatorname{round}_{\mathrm{mul}}(x_k y_k)\right)

The aim is to reduce memory bandwidth and arithmetic cost with low-precision inputs while suppressing accumulated rounding error with a higher-precision accumulator.

format Visualization

This view shows positive values representable by small floating-point formats on a number line. It helps explain the trade-off between representable range and significant digits.

format

Observation points
Notice that the spacing between points (the difference to the next value in the table) becomes wider toward the right. As the exponent increases, representable values become sparser, so larger values have coarser resolution even with the same number of bits.

NN learning animation

Dataset Hidden units Learning rate Epochs precision used to show the boundary Displayed epoch

0Epoch 040

Playback speed

precision formats to compare

Observation points
Observe how the decision boundary moves at each epoch, how the vertical line and dot on the loss graph move together, and whether the update size becomes smaller during training. With low precision such as FP8, the corrections can become coarse, convergence may slow down, and the boundary may jitter.

Learning and correction

A small neural network with two inputs, one hidden layer, and one output is trained by full-batch gradient descent.

\begin{aligned} \mathbf{z}^{(1)} &= W^{(1)}\mathbf{x} + \mathbf{b}^{(1)},\\ \mathbf{h} &= \tanh\!\left(\mathbf{z}^{(1)}\right),\\ z^{(2)} &= W^{(2)}\mathbf{h} + b^{(2)},\\ \hat{y} &= \sigma\!\left(z^{(2)}\right) \end{aligned}

L = -\frac{1}{N}\sum_{i=1}^{N}\left\{y_i\log \hat{y}_i + (1-y_i)\log(1-\hat{y}_i)\right\},\qquad \theta \leftarrow \theta - \eta \nabla L

The update size \( \lVert\Delta\theta\rVert \) is also shown at each epoch. With lower precision, rounding of weights, activations, and gradients can make the corrections coarser.

Legend:

Comparison at the current epoch

Overview: Pursuing high-performance high-precision computing methods for the AI era

This page introduces part of the research conducted by the High Performance Computing Laboratory.

The keywords are AI, high-performance computing, and high-precision computing In AI, massive low-precision computation at roughly 4 to 16 bits is central, while scientific computing mainly relies on high-precision computation of 64 bits or more. To bridge this gap, the High Performance Computing Laboratory explores techniques for accelerating high-precision scientific computing on AI-oriented computing systems.

Low-precision computing in the AI era

formats in the 4- to 16-bit range, such as FP4, FP6, FP8, BF16, and FP16, are used at large scale.
Dedicated accelerators and Tensor-Core-like units are used to process matrix products, dot products, and convolutions quickly.
In low-precision computation, scaling is important for controlling rounding error, underflow, and overflow.

High-precision computing in scientific computing

FP64 is usually the standard, and multiple-precision or arbitrary-precision formats such as DD, TD, QD, and MPFR are used when needed.
In numerical linear algebra, nonlinear equations, eigenvalue problems, ODEs, and related problems, insufficient precision directly affects the reliability of results.
A central research goal is to make use of the high arithmetic performance of AI-oriented hardware for high-precision computing.

Basic structure of floating-point numbers

A floating-point number represents a real number by separating it into a sign, an exponent, and a mantissa. Conceptually, a normal number has the following form.

x = (-1)^S \times (1.M)_2 \times 2^{E-\mathrm{bias}}

S: Sign field — 0 means positive, and 1 means negative.
E: Exponent — This determines the range of representable values. More exponent bits make it easier to handle extremely large and small values.
M: Mantissa — This determines the number of significant digits. More mantissa bits allow closer values to be distinguished.
scale — In low-precision AI computation such as FP4/FP8, a scale factor is often assigned to the whole tensor or to each block to adjust the representable range.

In low-precision formats, the allocation of bits between exponent and mantissa is important. For example, FP8 E4M3 has more mantissa bits and favors precision, while FP8 E5M2 has more exponent bits and favors range.

Length comparison of floating-point formats

The figure below shows the floating-point formats used on this page, using MPFR 128-bit as the maximum width and arranges them with lengths proportional to their actual bit widths. Without horizontal scrolling, you can compare the length difference between FP4/FP8 and FP64/MPFR 128-bit.

Axis guide: 0, 32, 64, 96, and 128 bits from the left. FP4 through TF32 are very short, so a minimum display width is reserved to keep labels readable.

Sign field S Exponent E（Range） Mantissa M（precision） MPFR arbitrary precision scale factor

FP4 E2M1

E×2

M×1

4 bit

This is an extremely low-precision 4-bit format. Since \(S=1\), \(E=2\), and \(M=1\), it is coarse by itself, but it is useful for AI inference quantization demos when combined with scaling.

FP6 E3M2

E×3

M×2

6 bit

This is an intermediate example between FP4 and FP8. With \(E=3\) and \(M=2\), you can examine the trade-off between range and precision.

FP8 E4M3

E×4

M×3

8 bit

This is the precision-oriented FP8 format. Since \(E=4\) and \(M=3\), it has one more mantissa bit than E5M2 and is suitable for comparing quantization of weights and activations.

FP8 E5M2

E×5

M×2

8 bit

This is the range-oriented FP8 format. Since \(E=5\) and \(M=2\), the mantissa is coarse, but it is useful for explaining values or gradients whose magnitudes vary widely.

BF16

E×8

M×7

16 bit

This is \(1+8+7=16\) bits. The exponent has the same 8-bit width as FP32, so the range is wide, but the 7-bit mantissa makes precision coarse.

FP16

E×5

M×10

16 bit

This is \(1+5+10=16\)-bit half precision. In AI, it is often used for inputs and weights, while accumulation is performed in FP32 as mixed-precision computation.

TF32

E×8

M×10

Equivalent to 19 bits

Conceptually, this is equivalent to \(1+8+10\) bits. It keeps the FP32 exponent width while shortening the mantissa to accelerate matrix multiplication.

FP32

E×8

M×23

32 bit

This is \(1+8+23=32\)-bit single precision. It is a standard reference format in AI computation and is also used as an accumulator for low-precision inputs.

FP64

E×11

M×52

64 bit

This is \(1+11+52=64\)-bit double precision. It is a standard format in scientific computing and serves as a reference for comparing against low-precision AI computation.

MPFR 128

128-bit arbitrary-precision mantissa

128-bit setting

MPFR allows the mantissa precision to be specified arbitrarily. On this page, the MPFR Arbitrary-precision graph tab can generate high-precision reference values such as 128-bit results.

With scaling

Scale factor for each tensor or block

Supplementary information

For formats such as FP4/FP8, in addition to the low-precision values themselves, a scale factor \(\mathrm{scale}\) can be assigned to the whole tensor or to each block, making a practical range easier to handle with limited bits.

The bar length represents the total bit length. Inside each bar, red indicates the sign field, blue the exponent, and green the mantissa. A longer exponent gives a wider representable range, while a longer mantissa distinguishes nearby values more finely. MPFR is shown as an arbitrary-precision format with a specified mantissa precision, not as a fixed-length IEEE format.

How this demo page is organized

Material viewpoint	Tabs to try	What you can observe
4- to 16-bit low-precision AI computation	Function Quantization, Stable Softmax	rounding error, saturation, and the effect of scaling
Mixed precisioncomputation	Dot Product and Accumulation	Effect of low-precision inputs plus a higher-precision accumulator
high precisioncomputation	MPFR Arbitrary-precision graph	function tables and graphs with MPFR-specified bit precision
Understanding formats	format Visualization, material explanation	Exponent/mantissa allocation and the density of representable values

MPFR Arbitrary-precision graph

MPFR available

precision bits:

Divisions:

Function formula; the only variable is "x":

When the MPFR engine is used, mpfr_gexpr the formula is passed as-is to the evaluator. In fallback mode, the operators + - * / ^ ( ), Function sin, cos, tan, exp, log, log10, sqrt, abs, floor, ceil, tanh, pow, min, maxand constants pi, e are supported.

Start point of x:

End point of x: