attention_scaled

std.nn.attention_scaled · Level L1

Scaled dot-product attention with the standard 1/√d scale computed inside the graph (Size, then Sqrt). Calls attention.

softmax(Q·Kᵀ / √d)·V

Signature

attention_scaled(Q: f64[m, d], K: f64[t, d], V: f64[t, e]) → f64[m, e]

Structure

The function as NOVA stores it: one box per input, operation and output, and arrows that carry values. A double border marks another library function this one runs — called once, or by Scan once per element; select it to open that function.

Qf64[m, d]Kf64[t, d]Vf64[t, e]Sized1.0SqrtrdDividescaleattentionOOf64[m, e]
  • input
  • operation
  • constant
  • call
  • output

Verification

  • Signature proven by NOVA’s shape solver, for every size.
  • Agrees with the reference softmax((Q @ K.T) / np.sqrt(d), axis=-1) @ V to 80 digits (100-digit arithmetic), on all 40 test cases.
  • All 649 float64 results inside the running error bound; the closest uses 8% of it.
  • Interpreter and NumPy backend return bit-identical results.
Accuracy in detail
correctly rounded (the float64 nearest the exact value)
68%
bit-equal to the NumPy formula in float64
95%
largest error, in units in the last place
140

Large ulp counts appear where a result is tiny next to the numbers it is computed from (after cancellation, for example), so one unit in the last place is tiny too; the absolute error is still inside the bound. Results within their own error of zero are not counted.

Identity

Calls
attention
Called by
—
sha256:bcd09cb93e1757202ae0aa382c14438762cc5535b794e9f483757090d1c50a0e

The semantic hash of the graph. It changes when the program changes, and never when only its documentation does.

Control handle

Symbol
Ω:std.nn.attention_scaled · Ω:attention_scaled
Pins
sha256:1e098cde8a431c1a82212866a2143f8b9a3a246d2f33db0fc5d54cccae60a703this graph and the 1 it reaches through calls
Evidence
sha256:f07d3354fb2afe3d14617491479a5ed21d76b81a65ea7d983a903e48cd092d51the hash of its verification record
Needs
no capability: a pure function

Through NOVA’s control layer, the symbol launches this function only while the program still matches what it pins: a change to this graph, or to any graph it reaches, needs a migration first.