Errata, ambiguities and inconsistencies in arXiv 2312.17173v3

Policy: the explainer never silently fixes the paper. Each item gives the location, what the paper says (paraphrased or quoted in a few words), the problem, whether it changes any reported number, and where the explainer discusses it. Severity: [S] substantive (affects a claim or validity argument), [M] minor (notation/typo, no numeric effect), [A] ambiguity (unstated choice the reader must guess).

Items marked MY ANALYSIS are conclusions of this explainer, not statements by the authors. Items that cite the authors' public code use it only as evidence about what the paper meant.

---

A. Mathematics and derivations

E01 [M] §4.1, p. 4 — prior inequality points the wrong way. The text writes the Solomonoff prior as \(P(h) \le 2^{-K(h|A)}\) and then concludes \(\ln\frac1{P(h)} \le K(h|A)\ln 2\). The conclusion needs \(P(h) \ge 2^{-K}\). Appendix C (p. 15) and Lotfi et al. (2022) use \(P(h)=2^{-K(h)}/Z\) with \(Z\le1\), which does give \(P(h)\ge2^{-K}\). No numeric effect. → Ch. 13, 14.

E02 [A] Eqs. (1), (3), (5), (6), (8) and Alg. 1 line 20 — logarithm base never stated; bits and nats are mixed. The bound's "log" must be the natural log (it comes from \(e^{-2mt^2/\Delta^2}\) in Hoeffding's inequality). The text's \(C(h)\log 2 + 2\log C(h)\) is consistent only if log = ln (because \(2\log_2 C\cdot\ln2 = 2\ln C\)). Algorithm 1 line 20, "message_size + 2 × log(message_size)", is in bits, so its log must be \(\log_2\) (the authors' code uses np.log2). BPD uses \(\log_2\). MY ANALYSIS Using bits where nats are required only makes the bound larger, so this mix-up can never make a reported bound invalid — only looser. → Ch. 3, 11, 14 (demo ch14_budget.py).

E03 [M] Eq. (3), p. 6, and App. A.3 second display, p. 14 — missing \(1/n\). The subsampled empirical risk is written \(\hat{\hat R}(h)=\sum_{i=1}^n \hat R_{\sigma(i)}(h)\) (a sum, not an average). App. A.3's first display has the \(\frac1n\); its second display drops it again. With the sum, the bound is wrong by a factor of \(n\). Intended meaning is clearly the average. → Ch. 17.

E04 [M] App. A.3, p. 14 — Hoeffding step lacks \(\Delta\) and the argument of \(\hat R\). The intermediate statement reads \(\hat R \le \hat{\hat R}(h) + \sqrt{\ln(1/\delta_2)/(2n)}\). For a loss of range \(\Delta\) the deviation term must be multiplied by \(\Delta\); the combined bound just below does include \(\Delta\). → Ch. 17.

E05 [S] §4.4 vs App. A.3 — "with replacement" vs "with or without replacement". The main text samples \(\sigma(i)\) with replacement; the appendix allows "with or without replacement" and calls the without-replacement draws "I.I.D. random variables sampled without replacement", which is self-contradictory (draws without replacement are dependent). The bound still holds without replacement, but this needs Hoeffding (1963, Theorem 4: sampling without replacement is at least as concentrated), which is not cited or argued. The authors' code samples with replacement. → Ch. 17 (we prove Hoeffding's Theorem 4 and simulate both schemes).

E06 [M] App. A.3 — the δ-split \(s=n/(n+m)\) is optimal only for a proxy objective. The paper says \(\delta_1=s\delta\), \(\delta_2=(1-s)\delta\) solves the optimization of \(\sqrt{-\ln\delta_1/(2m)-\ln\delta_2/(2n)}\), a single square root that is not the bound actually used (which is a sum of two square roots and includes \(\ln 1/P(h)\)). Any split with \(\delta_1+\delta_2=\delta\) is valid, so this does not affect validity; the numeric loss from using the proxy is negligible. → Ch. 17 (demo computes the true optimum).

E07 [S] MY ANALYSIS §6.1 and App. C — choosing α and d on the same subsample used in the second Hoeffding term. §6.1 computes Eq. (3) "as a function of α" on the \(n=10{,}000\) subsampled documents and keeps the best α, then keeps the subspace dimension \(d\) "that produces the best bound". The first term of Eq. (3) pays for these choices through \(\ln 1/P(h)\) (a union over all hypotheses). The second term, \(\Delta\sqrt{\ln(1/((1-s)\delta))/(2n)}\), is a Hoeffding bound for one fixed hypothesis given the data; it contains no union over the alternatives that were compared on the same subsample. A rigorous version replaces \(\ln\frac1{(1-s)\delta}\) by \(\ln\frac{N_{\text{sel}}}{(1-s)\delta}\) where \(N_{\text{sel}}\) is the number of candidates compared on the subsample. With \(N_{\text{sel}}=576\) (the whole grid) and \(\Delta\approx 18\) bits this adds roughly 0.1–0.2 bits per token — small, but the published numbers are not covered by the stated theorem as written. → Ch. 17, 29.

E08 [A] §3 vs §5 — two different projector constructions. §3: \(P=(Q_1\otimes Q_2)/\sqrt D\) with \(Q_1,Q_2\sim\mathcal N(0,1)^{\sqrt D\times\sqrt d}\), "normalized so that \(P^\top P\approx I\)". §5: \(P=Q_1\otimes Q_2\) where \(Q_i\) come from QR factorizations of Gaussian matrices \(\mathcal N(0,1/\sqrt D)^{\sqrt D\times\sqrt d}\), which makes \(P^\top P = I\) exactly (and makes the scale \(1/\sqrt D\) irrelevant; "\(\mathcal N(0,1/\sqrt D)\)" is also ambiguous between variance and standard deviation). The authors' code uses the §5 (QR) version ("rdkronqr"). → Ch. 18 (demo checks both).

E09 [A] §3, §5 — shapes \(\sqrt D\times\sqrt d\) require perfect squares. \(D\) (e.g. the LoRA parameter count) and \(d\) are generally not perfect squares. The paper does not say how this is handled. The authors' code Kronecker-factorizes the largest square-rounded block and fills the remainder with a dense Gaussian block. → Ch. 18.

E10 [S] MY ANALYSIS Eq. (4), §5 — the LoRA factors need a fixed nonzero initialization that the formula omits. Read literally, \(\theta=\theta_0+\mathrm{LoRA}(Pw)\) makes both LoRA factors functions of \(Pw\) alone. At the natural starting point \(w=0\) both factors are zero, the update \(BA\) is zero, and its gradient with respect to \(w\) is also zero (each factor's gradient is multiplied by the other, zero, factor), so training cannot start. Standard LoRA avoids this by initializing one factor randomly. The authors' code does exactly this: the LoRA factors equal a fixed random LoRA initialization (A random, B zero) plus \(\mathrm{unflatten}(Pw)\). That fixed offset is part of the "random initialization" that must be reproducible from a seed. → Ch. 20 (demo shows the zero gradient).

E11 [M] Thm A.1 statement, p. 12 — "a sum over independent random variables". \(\hat R(h)\) is an average (\(\frac1m\sum\)), not a sum. → Ch. 11.

E12 [M] §4.1 — dropped constants in \(K(h|A)\ln2 \le C(h)\ln2 + 2\ln C(h)\). Prefix Kolmogorov complexity is bounded by a code length only up to an additive constant (the length of the decompressor), and an Elias-gamma length header costs \(2\lfloor\log_2 C\rfloor+1\) bits, not \(2\log_2 C\). Appendix C's \(K(h)\le K(\theta|d,r)+K(d)+K(r)\) likewise holds only up to \(O(1)\). [MY ANALYSIS, from Ch. 12] None of this matters if the prior is defined directly from the fixed compression scheme: the "length prior" \(P(h)=C(h)^{-2}2^{-C(h)}\) for \(C(h)\ge2\) sums to at most \(\pi^2/6-1\approx0.645\) and gives exactly \(\ln\frac1{P(h)}=C\ln2+2\ln C\) (this is the weight \(2^{-2\log_2 l(h)}\) of Lotfi et al. 2022, §3 footnote 2). It fails only if 1-bit messages were allowed (\(\sum_{c\ge1}c^{-2}=\pi^2/6>1\)) — irrelevant for messages of ~10⁵ bits. → Ch. 12, 13, 14.

E13 [M] Thm A.2 vs §4.3 — open vs closed interval; \(k\) vs \(L\). Eq. (7) states an open interval, §4.3 a closed one ("≤"); strictness requires \(0<p_\theta<1\). Harmless. The proof averages over \(k\) tokens while the main text uses \(L\) for document length. → Ch. 16.

E14 [M] §6.2, p. 7 — the top-1 "error" risk is written as an accuracy. The displayed per-document risk uses the indicator \(\mathbb 1[\arg\max p = x_i]\) ("=") and the next sentence says it is "the proportion of tokens accurately predicted". An error risk needs "≠" (or \(1-\) that quantity). The tables are clearly error bounds (and the code computes \(1-\)accuracy). → Ch. 6, 25.

E15 [S] App. E, Eq. (9), p. 16 — the long-document context formula does not do what the text says. With \(L'=L-(i-L)\,\%\,100\) and context \(x_{i-L':i-1}\), the window start is \(j+(j \bmod 100)\) for \(j=i-L\): it moves by two tokens per step for 99 steps and then jumps back (start 198 at \(j=99\), 100 at \(j=100\)). That is not "shifting in chunks of 100". The authors' code uses windows that start at multiples of 100 (stride 100), which corresponds exactly to \(L'=L-99+((i-L-1)\bmod 100)\) (derived in Ch. 24; an earlier version of this entry gave a form that is off at multiples of 100). As printed, a 2,048-token document would need 550 forward passes instead of 12. → Ch. 24 (demo ch24_pipeline.py tabulates both).

B. Notation collisions and undefined symbols

E16 [M] Symbol reuse. \(V\) = vocabulary size and LoRA factor \(V\in\mathbb R^{r\times b}\) (§3). \(\alpha\) = smoothing weight and "LoRA alpha value of 32" (App. E). \(L\) = document length, context length (1024), and number of quantization levels (Alg. 1 argument list; App. C "values of r, d, L and α"). \(C\) = compressed size \(C(h)\) and number of quantization levels \(C\in\{11,17\}\). \(k\) = number of tokens (§4.3), the k in top-k, and the document index (§6.2). \(A\) = architecture. The explainer renames: LoRA factors \(B,A\) (Hu et al.), \(\alpha_{\text{LoRA}}\), \(L_{\text{ctx}}\), \(L_q\), architecture \(\mathcal A\). → Conventions page.

E17 [M] Algorithm 1, p. 15. The function signature takes an argument \(L\) that is not among the listed Inputs (the inputs call the quantization levels \(C\)); line 7 passes neither δ nor α (needed for Δ); line 12 differentiates through a nearest-level assignment, which needs a straight-through estimator (explained in Lotfi et al. 2022, not here); line 20's log base (see E02). → Ch. 21, 24.

E18 [A] Alg. 1 / App. C — is the codebook and the symbol frequencies counted in the message size? Arithmetic decoding needs the codebook values \(c\) and the symbol probabilities. Lotfi et al. (2022) count \(16L_q\) bits for the codebook and \(L_q\lceil\log_2 d\rceil\) bits for the counts; this paper does not say. The authors' code passes the codebook and counts to its message-length function. Size is a few hundred bits — negligible. → Ch. 22, 24.

C. Numbers, tables and figures

E19 [S] §1, p. 1–2 — the speed-up figures disagree. "900 times faster" vs "45 minutes on a single GPU instead of 3 days on 8 GPUs": 3 × 24 × 8 = 576 GPU-hours vs 0.75 GPU-hours is 768×; wall-clock 72 h vs 0.75 h is 96×. Neither equals 900. (For reference \(m/n\approx801\).) → Ch. 17, 29.

E20 [M] §4.4, p. 6 — "dataset sizes in the 10's of millions". OpenWebText has ≈ 8.0 million documents (8,009,762 in the nanoGPT training split) and ≈ 8.8 million 1,024-token sequences. → Ch. 17.

E21 [M] Tables 1 and 4 — Top-1 random guess "99.99". \(1-1/50257 = 99.998\,\%\), which rounds to 100.00 %; the table truncates. The other random-guess entries (99.98, 99.80, 15.62) round correctly. → Ch. 6, 25.

E22 [S] App. D, p. 16 — Table 5 is described as "non-vacuous classification accuracy bounds". The entries are error bounds, and the randomly-initialized ones (QQP 71.72 %, CoLA 53.42 %) exceed the paper's own 50 % random-guess level, i.e. they are vacuous. Moreover, the 50 % baseline ignores class imbalance: a constant classifier has error 36.93 % on QQP (229,468 of 363,846 training pairs are "not duplicate") and 29.56 % on CoLA (6,023 of 8,551 training sentences are "acceptable"). The pretrained CoLA bound (38.89 %) is therefore worse than always answering "acceptable". → Ch. 26, 29.

E23 [A] Fig. 3 caption, p. 8 — "Classification error bounds consistently favor smaller models". If "smaller models" means fewer LLM parameters (the x-axis), the caption contradicts the middle panel, whose best-bound line falls from ≈0.976 to ≈0.96 as parameters grow. If it means smaller compressed size (the colour scale), it is consistent (the best top-1 points are light-coloured, the best data-compression points dark). → Ch. 26.

E24 [A] Fig. 3 vs Table 1. Fig. 3 covers the eight App. E configurations (x-positions match parameter counts including position embeddings and rank-4 LoRA parameters: 1.90M … 124.75M). Its BPD best-bound line, read exactly from the PDF's vector data (fig_extract.py), is 12.207, 12.160, 11.962, 12.205, 12.068, 12.019, 12.057, 12.007 — so at GPT-2-small size it reaches 12.007, and at 7.57M parameters 11.962, both below Table 1's "best" GPT-2-small bound of 12.12. The paper does not explain the difference (different runs or settings). → Ch. 26.

E25 [A] Fig. 1 right and Fig. 2 right — axes that cannot be read quantitatively. Fig. 1 (right) labels its log x-axis "Extent of Compression (model size KB)", but the caption does not say how the sweep was produced (which d, r, L_q). Fig. 2 (right) has a left axis "NLL (BPD)" with tick marks but no tick labels, so the empirical-risk and bound curves cannot be read in bits; only the right axis (interval size Δ, 15–25) is labelled. → Ch. 16, 23.

E26 [A] Fig. 3 right — "Compressed Dataset Size" has no unit (≈0.85·10¹¹; presumably bits). → Ch. 26.

E27 [A] Table 2 — "Validation NLL" unit not stated. The values equal BPD × ln 2 to rounding (10.53 → 7.30 vs reported 7.29; 4.35 → 3.015 vs 3.01), so they are nats per token. → Ch. 25.

E28 [A] Tables 1–3 vs Table 2 — what kind of average. The bound's risk is a per-document average of per-token BPD (every document weighs the same, regardless of length). Whether "Validation BPD" is averaged per document or per token is not stated; the two differ in general, so bound vs validation is not necessarily apples-to-apples. → Ch. 15, 25.

E29 [M] Table 3 — "58.4" has one decimal while all other entries have two. → Ch. 26.

E30 [M] Fig. 4 — truncated y-axes (they start at 9.806, 0.797 and 0.474; the lowest tick labels are 10, 0.8 and 0.5), which visually exaggerates differences of ≈0.3 BPD and ≈0.01–0.06 error. The permuted top-1 bar exceeds 1.0 (an error bound above 1 is trivially true). → Ch. 26.

D. Unstated values (not errors, but needed to reproduce numbers)

E31 [A] δ (confidence) is never stated. The authors' code uses 0.05; Lotfi et al. (2022) also report δ = 0.05. E32 [A] \(m\) is never stated. nanoGPT's OpenWebText split has 8,009,762 training documents (the code's default num_docs), and the sequence-level count is 8,823,811 (the code's default data_size). E33 [A] The compressed sizes \(C(h)\) and empirical risks behind Tables 1–4 are not reported, so the bounds cannot be recomputed from the paper alone; the explainer reverse-engineers plausible values and labels them as such.

E35 [A] Fig. 2 left, Fig. 3 — "KB" is never defined (logged by Ch. 3). Lotfi et al. (2022) use 1 KB = 8192 bits. MY ANALYSIS The 8192-bit reading is the one that reproduces Table 1: with the Fig. 2 compressed size 16.5325 KB (= 135,434 bits), Eq. (3) gives 12.1222 at 8192 bits/KB but 12.1067 at 8000 bits/KB (see "Verification" below). → Ch. 3, 25.

E36 [A] V is never printed (logged by Ch. 3). The paper writes log₂V and 1 − k/V but never gives V; GPT-2's tokenizer has V = 50,257. Table 1's rounded 15.62 alone fits any V from 50,187 to 50,535. → Ch. 3, 4, 6.

E37 [M] §3 vs §5 — symbols in the projector recipe change meaning (logged by Ch. 8). In §3, \(Q_1,Q_2\) are raw Gaussian matrices; in §5 they are QR factors, the raw matrices are \(P_1,P_2\) (reusing \(P\) — projector and prior) and the triangular factor is \(R_1\) (reusing \(R\) — risk). The explainer writes \(G_i\) (raw), \(Q_i\) (orthonormal), \(T\) (triangular). → Ch. 8, 18.

E38 [M] App. A.3 — "with probability 1 − δ₂" / "1 − δ₁" (logged by Ch. 9) should read "at least"; Hoeffding gives only a lower bound on the success probability (Theorem A.1 states it correctly). Wording only. → Ch. 9, 17.

E39 [A] Fig. 4 "Regular" bars vs Table 1. Read exactly from the PDF, the regular-context bars are 12.139 BPD, 96.36 % top-1 and 59.22 % top-100 error, whereas Table 1's best GPT-2-small bounds are 12.12, 96.41 % and 58.34 %. The figure therefore shows a different (unspecified) run or setting from the table. → Ch. 26.

E40 [A] Fig. 2 (left) vs App. E — the plotted subspace dimensions are not the stated grid. Read exactly from the PDF, Fig. 2 (left) has points at d = 5,000, 7,500, 10,000, 25,000, 50,000, 75,000, 100,000 and 200,000; App. E lists only {5,000, 10,000, 25,000, 50,000, 100,000, 200,000}. If 8 values were searched, the hyperparameter cost is log₂8 rather than log₂6 bits for d — a difference of 0.415 bits, numerically negligible, but the reported search space is incomplete. → Ch. 24, 25.

E41 [A] Fig. 2 (right) silently combines three different models. [MY ANALYSIS, found by Ch. 16, verified by the orchestrator] For one fixed model, (bound − empirical risk)/Δ(α) must be the same at every α (only Δ changes). In the plotted data (read exactly from the PDF) it takes three values: 0.0494 (α = 10⁻⁴, 10⁻³), 0.0712 (α = 0.005, 0.01) and 0.0888 (α ≥ 0.05). Solving Eq. (3) for the compressed size gives 3.883, 9.813 and 16.537 KB — exactly Fig. 2 (left)'s d = 10,000, 25,000 and 50,000 models. So each α shows the best bound over models, not one model's curve; neither the caption nor §4.3 says so. Consequence: the plotted empirical risk falls from 11.385 to 10.453 bits/token as α goes 10⁻⁴ → 0.05 only because the model changes; within the single d = 50,000 model it rises (10.453, 10.473, 10.595, 10.942), consistent with the text's "the empirical risk goes up". The plotted worst-case interval values match Theorem A.2's Δ(α) to three decimals. → Ch. 16, 25.

E. Typos

E34 [M] "probabiliiy" (A.3), "evalution" (App. E heading), "hyperaparmeters" (App. E heading), "so that optimize the bound" (A.3).

---

Notes from the authors' public code (not errors in the paper, recorded for completeness): the α choice is paid for with ceil(len(alpha_array)) = 8 extra bits rather than \(\log_2 8 = 3\), and that increment sits inside a loop so later α values pay 16, 24, … bits — conservative, so it cannot invalidate a bound.

F. Errata found while writing the chapters

Each entry was found and checked by the chapter named in it; the numbering continues from E41.

E42 [M] Table 7, p. 19 — the end-of-text token is garbled. (logged by Ch. 4) Location: Table 7 (generated text from the SubLoRA model), fourth sample, "…to they things.¡—endoftext—¿ the the…". Issue: this is GPT-2's special token <|endoftext|> typeset in a LaTeX font encoding (OT1) in which the characters <, | and > in ordinary text print as ¡, — and ¿. The paper does not mention that the sample contains an end-of-text token, i.e. that the model "finished" one document and began another mid-sample. Impact: none on any number; a reader may not recognise the token. → Ch. 4.

E43 [A] §6.4, p. 7; Table 3, p. 8; App. E, pp. 16–17; Fig. 3 — model sizes counted two (three) ways; shapes of 458M and 849M never given. (logged by Ch. 5) Location: §6.4 and Table 3 name sizes 354M (GPT-2 medium), 458M, 773M (GPT-2 large), 849M; App. E repeats them; no shape (layers, heads, embedding dimension) is given for 458M or 849M, and no head count for any larger variant. Issue: the labels follow nanoGPT's count (padded vocabulary 50,304, biases in every layer, tied head, no position embedding): 353,822,720 and 772,779,520. The authors' released code builds bias-free layers by default (sublora_pipeline.py, bias=False; only the LoRA c_attn keeps a bias), for which the 36 × 1,280 model has 772,271,360 parameters (rounds to 772M, not 773M). Figure 3's x-axis uses a third rule: all parameter tensors of the bias-free LoRA model, including the position embedding and rank-4 LoRA factors on q, v and the head (reproduced to ≤ 5.3 ppm; see E24). MY ANALYSIS 458M and 849M are most likely 20 × 1,280 (457,940,480) and 25 × 1,600 (849,009,600): the only shapes with a GPT-2 width that round to these labels, and the only ones in the family d_model = 64·n_layer (layers = heads, 64 per head) used by the paper's sweep. Head counts cannot be inferred (they do not change the count). Impact: none on any bound (bounds depend on compressed size, not on D); size labels are nominal. → Ch. 5, 26.

E44 [M] §5, p. 6 vs App. E, p. 16 — which attention matrices carry LoRA. (logged by Ch. 5) Location: §5 applies LoRA "only over the self-attention layer and the last linear layer"; App. E says the query and value weight matrices plus the linear head. Issue: "self-attention layer" could include W_K and W_O. App. E is the precise statement; the authors' code uses enable_lora=[True, False, True] (q and v only), and Figure 3's x-positions agree with q, v, head LoRA counts. Impact: none on numbers; wording only. → Ch. 5, 19, 20.

E45 [A] §6.2, p. 7; Tables 1, 3, 4 — top-k error is defined only for k = 1; ties unspecified. (logged by Ch. 6) Location: §6.2 displays only the top-1 risk; the tables report top-10 and top-100 error bounds. Issue: no formula for k > 1, and no rule for ordering tokens of equal probability (relevant to argmax and top-k sets). The natural reading is "the real token is not among the k most probable tokens". The authors' code (sublora_bounds_bound_utils.py, bound_metrics_from_logits / compute_bound_metrics) sorts the softmax probabilities in decreasing order and counts a hit when the real token's 0-based position is < k, so ties are ordered by the sort routine. Impact: none in practice (ties are rare with float logits); a rigorous definition needs the rule. → Ch. 6.

E46 [M] Table 2, p. 7 — SubLoRA "Validation NLL" 7.29 is not the rounded value of BPD × ln 2; mixed (logged by Ch. 6) rounding conventions. (Refines E27.) Issue: if BPD 10.53 and NLL 7.29 were both rounded to 2 decimals, the true BPD lies in [10.525, 10.535) and the true NLL in [7.2954, 7.3023), which rounds to 7.30. The row is consistent with truncating both numbers, not with rounding both; the other three rows are consistent with either (ch06_metrics.py, output [5]). Table 1 also mixes conventions: log2 V = 15.6170 is rounded to 15.62, while 1 − 1/V = 99.9980 % is truncated to 99.99 (E21). Impact: ≤ 0.01 nats per token; no conclusion changes. → Ch. 6, 25.

E47 [A] Table 2, p. 7 — "Validation BPD/NLL" of the smoothed model p_h or the base model p_θ? (logged by Ch. 6) Issue: the bound certifies the prediction-smoothed model p_h (§4.3); Table 2 does not say which model its validation rows score. The validation loss tracked by the authors' code (estimate_loss in sublora_sublora_pipeline.py, get_batch in sublora_utils.py) uses the base model without smoothing, on random 1,024-token windows that ignore document boundaries (token-level average; see E28). Impact: "bound 12.12 vs validation 10.53" may compare different models as well as different averages; indicative only. → Ch. 6, 25.

E48 [A] App. E, p. 16 — "no decay bias" is undefined, and most optimizer settings are unstated. (logged by Ch. 7) Location: App. E, "Pretraining" paragraph. The paper lists AdamW with weight decay 10⁻², epsilon 10⁻⁶ "and no decay bias", without explaining the phrase. In the authors' code (projectors.py, configure_optimizers), the flag no_decay_bias=True puts every parameter tensor with fewer than two dimensions into a group with weight decay 0; in subspace and SubLoRA training the only trainable tensor is the one-dimensional vector w, so under that reading the stated weight decay would have no effect. The released config_sublora_train.yaml sets no_decay_bias: False, under which w is decayed (pulling θ toward θ₀). Also unstated in the paper: β₁, β₂ (config 0.9, 0.999), learning-rate schedule (code default: 2,000-step linear warm-up then cosine decay), gradient accumulation (config 40 minibatches of 8 = 320 sequences per update), gradient clipping (code default norm 1.0), number of updates (config max 600,000), seeds (137 for θ₀ and P, 1337 for batch sampling). Impact: none on the validity of any bound (the bound holds for whatever procedure produced h); reproducibility only. → Ch. 7 (values printed by ch07_gd.py output [8]).

E49 [M] Thm A.1, p. 12 vs §4.3 p. 5 / §6.1 p. 6 — one interval width Δ for all hypotheses. (logged by Ch. 11) Location: statement of Theorem A.1 ("a bounded risk R(h, x_i) ∈ [a, a+Δ]") and Eqs. (1), (5), (6). Issue: the theorem assumes a single interval shared by every hypothesis, but the paper later treats the smoothing weight α as part of the hypothesis and uses a width Δ(α) that depends on it (Theorem A.2), selecting α by the bound. Read literally, Theorem A.1 does not cover a class with hypothesis-dependent widths. MY ANALYSIS The proof covers it unchanged: each Hoeffding step concerns one fixed h, so each h may have its own interval [a(h), a(h)+Δ(h)] fixed in advance, with margin t(h) = Δ(h)·√((ln 1/P(h) + ln 1/δ)/(2m)); the union step never uses Δ. Impact: none on any number; the statement should read Δ(h). → Ch. 11 (proof in "Edge cases" block), Ch. 16.

E50 [M] Thm A.1 proof, p. 13 — "Σ_h P(h)δ = δ" assumes the prior sums to exactly 1. (logged by Ch. 11) The theorem never states Σ_h P(h) ≤ 1; the last line of the proof writes "= δ". For a sub-probability prior (e.g. the Solomonoff prior with Z ≤ 1, or the length prior of Ch. 12) the step is "≤ δ". Impact: none (wording). → Ch. 11.

E51 [M] §4.1, p. 4 — K is called "prefix" Kolmogorov complexity but defined without the self-delimiting condition; "P" also names the programming language. (logged by Ch. 13) Location: §4.1, the two sentences after Eq. (1): "\(K\) is the prefix Kolmogorov complexity of \(h\) …" and "defined as the length of the shortest program that produces \(h\) for a fixed programming language \(P\)". Issue: the stated definition is the plain Kolmogorov complexity (no requirement that programs be self-delimiting). With plain complexity \(\sum_h 2^{-K^{\text{plain}}(h)} = \infty\) (every \(n\)-bit string has \(K^{\text{plain}} \le n + c\), so each length contributes at least \(2^{-c}\)), so the normalized prior \(2^{-K}/Z\) of App. C cannot be formed; only the prefix version gives \(Z \le 1\) (Kraft). Lotfi et al. (2022, §3) state the self-delimiting condition correctly. The same sentence also uses \(P\) for the programming language, alongside \(P(h)\) (prior) and \(P\) (projection matrix, §3/§5; cf. the Ch. 8 symbol-reuse item). Impact: none on any number; the word "prefix" shows the intended meaning. → Ch. 13 (warning block in "Precise definition"; proof of divergence in the Going-deeper block on Z).

E52 [M] App. C, Algorithm 1 line 6, p. 15 — a bit count is stored under the name "log 1/P(h)" (addendum to E02). (logged by Ch. 14) Location: Algorithm 1, line 6 ("log 1/P(h) ← GET_COMPRESSED_SIZE(ŵ)") together with line 20 (which returns message_size + 2 × log(message_size)) and line 7 (which passes the value to the Eq. (3) bound). Issue: GET_COMPRESSED_SIZE returns a size in bits (its log must be log₂), but line 6 names that number "log 1/P(h)", and Eq. (3), which receives it, needs ln 1/P(h) in nats. Read literally, the pseudo-code plugs bits in where nats belong. MY ANALYSIS This mix-up is conservative: it inflates the complexity term by at most 1/√(ln 2) = 1.2011. For the paper's best model it would give 12.3871 instead of 12.1222 bits/token; the published 12.12 matches the correct conversion, and the authors' code multiplies by ln 2. Impact: none on reported numbers; the pseudo-code should read "log 1/P(h) ← ln 2 × GET_COMPRESSED_SIZE(ŵ)". → Ch. 14 (E02 warning block, worked example (e); demo ch14_budget.py).

E53 [A] Table 1, p. 6 and Table 2, p. 7 — how the "Original Model" bound of 70.76 was computed is not stated. (logged by Ch. 14) Location: "Original Model" row, Bits per Dimension Bound column (Table 1) and Bound column (Table 2). Issue: no code, smoothing weight α, empirical risk or bound equation is given for this entry (cf. E33). Moreover, a prediction-smoothed model's per-token loss is at most log₂(V/α) ≤ 28.905 bits for every α in the App. E grid, so 70.76 exceeds the largest possible loss: it is "more than vacuous" and not on the same scale as the other entries. MY ANALYSIS Raw float32 counting (32 bits/parameter) would give 209.6–383.6 bits/token (assumed smoothed empirical risk 5), so the entry must come from some compressed code; under Eq. (3) with the compression prior and an assumed empirical risk of 4–6 bits/token, 70.76 corresponds to 13.7–50.1 MiB, i.e. 0.93–3.39 bits per parameter. Impact: the qualitative conclusion (the uncompressed model's bound is vacuous) stands; the number itself cannot be reproduced or interpreted. → Ch. 14 (worked example (f)), Ch. 25.

E54 [S] MY ANALYSIS §4.2, pp. 4–5; App. B, p. 14 — the sequence-level bound is a statement about the training corpus. (logged by Ch. 15) Location: §4.2 ("shift the interpretation … to sampling sequences … from a fixed and finite dataset"), the Note on p. 5, and App. B / Table 4 ("despite the differing interpretation, the bounds are very similar in values"). Issue: under the stated reinterpretation the population is the empirical distribution of the corpus's 1,024-token chunks. The model was trained on that corpus, so the population risk \(R(h)\) equals the full training loss \(\hat R(h)\) exactly. The \(m\)-term of Eq. (1)/(3) then has no random draw behind it (the training set is the whole population, not \(m\) i.i.d. draws from it) and is unnecessary: \(R(h)=\hat R(h)\) holds with certainty, and only the subsampling (\(n\)-) term carries probabilistic content. The Note's argument ("independence depends on how we sample") justifies the subsample, not the \(m\)-term, at either level; at document level the \(m\)-term rests on the unverifiable assumption that the corpus is an i.i.d. sample of some process. Impact: Table 4's numbers remain valid but are upper bounds on training-corpus loss, not generalization bounds; they are not comparable in meaning to Table 1 even though the values are close. → Ch. 15 (proof in "sampling from a finite set").

E55 [M] App. E, p. 16 — "LoRA alpha" and "dropout ratio" are not defined. (logged by Ch. 19) Location: App. E, "we use a LoRA alpha value of 32 and dropout ratio of 0.1". Issue: the paper never says what either hyperparameter does. Following Hu et al. (2021, §4.1), "LoRA alpha" is the numerator of the scale α_LoRA/r applied to BA, so the effective multiplier changes across the paper's rank grid (32 at r = 1, 8 at r = 4); where the dropout acts (on the input of the low-rank branch in Hu et al.'s loralib) is also unstated. The name also collides with the smoothing weight α (see E16). Impact: reproducibility only; no bound depends on it. → Ch. 19.

E56 [M] §5, Eq. (4), p. 6 — one letter D for two sizes; unflatten defined on the wrong vector. (logged by Ch. 20) Location: §3 uses θ ∈ ℝ^D (D = number of model parameters); §5 keeps "P ∈ ℝ^{D×d}" in Eq. (4) and defines unflatten(θ) = {(W_i, b_i)}, while LoRA(·) is applied to Pw. Issue: for LoRA(Pw) to make sense, Pw must be the vector of LoRA factors, so P has D_LoRA rows (351,556 for GPT-2 small at r = 4 with V = 50,257; 351,744 with nanoGPT's padded vocabulary), not D = 123,689,472. The authors' code (IDModule) indeed sizes P by the parameters left trainable after freezing everything but the LoRA factors. Impact: notation only. → Ch. 20.

E57 [M] Fig. 1 caption, p. 2 — the "complexity" axis is called "the 2nd term in Equation 1". (logged by Ch. 20) Issue: the plotted complexities are the two square-root terms of the subsampling bound Eq. (3), not Eq. (1)'s single term: risk + complexity reproduces Table 1 (12.1224 vs 12.12), and the Fig. 2 (left) gaps, which reappear among the Fig. 1 SubLoRA points (8 of 8), match Eq. (3) at α = 0.25 to ≤ 0.0006 bits/token (ch20_sublora.py output [4]). Impact: labelling only. → Ch. 20, 23.

E58 [A] Fig. 2 (left), p. 5 — the fixed hyperparameters of the plotted runs are not stated. (logged by Ch. 20) Issue: the caption gives the bound "as a function of the projection dimension d with LoRA" without the rank, smoothing α, learning rate or quantization levels. MY ANALYSIS Recomputing Eq. (3) from the plotted compressed sizes (δ = 0.05, m = 8,009,762, n = 10,000, 1 KB = 8192 bits) matches the plotted complexity at all eight d only for α = 0.25 (within 0.0006; neighbouring α change Δ by about 9 %). Impact: reading/reproducibility only. See also E40. → Ch. 20, 25.

E59 [A] §7.1, p. 8–9 (Fig. 3 right, p. 8) — "the number of bits to encode the data using the model … is the negative log-likelihood of the entire dataset": base and overhead unstated. (logged by Ch. 22) Location: §7.1, last paragraph (continues on p. 9), defining \(C(\{X\}_{i=1}^m \mid h)\). Issue: as a code length this identity holds only for the NLL measured in bits (\(\log_2\)), and only up to the arithmetic-coding overhead (at most 2 bits per coded message; at most \(2m\) bits if each document is coded separately). The paper's own Table 2 reports "Validation NLL" in nats (E27), so a reader taking "NLL" in that sense would be off by a factor \(\ln 2 = 0.693\). MY ANALYSIS The overhead is negligible: \(2m = 16{,}019{,}524\) bits (m = 8,009,762, SOURCE nanoGPT) is 0.0188 % of the plotted \(8.5028\times10^{10}\) at the largest model (fig_extract.py). Impact: none on any reported number; complements E26 (the panel has no unit). → Ch. 22 (demo ch22_arith_coder.py, blocks [6]–[7]), Ch. 26.

E60 [M] Fig. 1 caption, p. 2 — the horizontal axis is not "the 2nd term in Equation 1" (same issue as E57, found independently). (logged by Ch. 23) The plotted "Complexity" equals the sum of both extra terms of Eq. (3) (prior term over m documents + Hoeffding term over the n subsampled documents), and the vertical "BPD (train)" is the subsampled, smoothed empirical risk. For the best BPD model (16.5325 KB, α = 0.25, Δ = 17.2020): plotted 1.5278; Eq. (3) terms 1.3171 + 0.2106 = 1.5276 (unrounded terms); Eq. (1)'s second term alone 1.3170. Top-1 best point: plotted 0.0494 vs 0.0371 + 0.0122 = 0.0493. No numeric effect; the label understates what the axis contains. → Ch. 23 (demo ch23_fig1.py, output [5]).

E61 [M] Fig. 1 (left), p. 2 — the shaded "vacuous" region is truncated. (logged by Ch. 23) Its sloped edge is exactly x + y = log2 V (BPD) or 1 − 1/V (top-1), but in the top-1 panel the shading follows that line only up to complexity 0.1336 and then has a flat floor at 0.8664 (to 0.1790); in the BPD panel it stops at train 15.9036 and complexity 3.0815. The drawing therefore under-represents the vacuous set. No plotted point is misclassified (40/40 per panel agree with the arithmetic). → Ch. 23 (output [2], [3]).

E62 [A] Fig. 1 (right), p. 2 — the sweep mixes d and α. (logged by Ch. 23) Eight of the 20 sweep points coincide exactly (size and bound) with Fig. 2 (left)'s eight d values. Inverting Eq. (3) for Δ at each point's plotted size gives Δ within +0.00…+0.50 % of Δ(0.25) = 17.202 for 14 points and within +0.00…+0.75 % of Δ(0.5) = 15.617 for 6 points (0.148 KB and 168–509 KB). So "for a fixed model" means a fixed architecture; each point is a different (d, α) model, as E41 found for Fig. 2 (right). → Ch. 23 (output [6]).

E63 [M] References, pp. 10–11 — garbled author lists. (logged by Ch. 23) Dettmers et al. 2023a (QLoRA) and 2023b (SpQR) carry the same seven-author list, and GPTQ is attributed to "Frantal, Z., Gruslys, A., and Kiela, D." (also cited as "Frantal et al." in §2). QLoRA is by Dettmers, Pagnoni, Holtzman & Zettlemoyer; GPTQ by Frantar, Ashkboos, Hoefler & Alistarh (from the papers themselves; not checked against a file in sources/). No result affected. → Ch. 23.

E64 [M] App. C, Algorithm 1, lines 6 and 9–14, p. 15 — further undefined symbols (extends E17). (logged by Ch. 24) Location: Algorithm 1, function TRAIN_QUANTIZE (lines 9–15) and line 6. Issue: TRAIN_QUANTIZE returns \(\hat w\) in line 14 but never computes it (lines 10–13 update only the codebook \(c\) and \(w\); the nearest-level assignment \(\hat w_i = c_q\), \(q = \arg\min_k |w_i - c_k|\), appears only in the prose below the algorithm). The step size \(\rho\), the loss \(\mathcal L(w,c)\) and quant_epochs are never defined. Line 6 assigns a size in bits (the output of GET_COMPRESSED_SIZE) to a variable named \(\log 1/P(h)\), which Eq. (3) uses in nats; read literally (no factor ln 2) the best model's bound would be 12.3870 instead of 12.1222 bits/token (ch24_pipeline.py section [4]). Also, Algorithm 1 describes one grid point only; the loop over (d, r, lr, α, L_q) and the choice of the lowest bound are stated only in the last paragraph of App. C. Impact: no reported number changes if read as intended; a literal implementation would get line 6 wrong (conservative direction). → Ch. 24.

E65 [M] Table 1 caption, p. 6; Table 4 caption, p. 14 — "all of which are non-vacuous", and unstated capping at 100 %. (logged by Ch. 25) Location: captions of Tables 1 and 4; App. B text, p. 14. Issue: both captions call every reported bound non-vacuous, but the LoRA-only top-1 entry (100) and the whole "Original Model" column (Table 1: 100, 100, 100, 70.76; Table 4: 100, 100, 100, 65.37) are at or above the random-guess level. App. B's own text says LoRA alone is vacuous for top-1. Moreover the 100 entries are capped: Figure 1 (left) shows LoRA's best top-1 bound as 1.0058 (ch23_fig1.py [1], ch25_tables_audit.py [2]); the rule "report min(bound, 100 %)" is not stated. Impact: none on any number; the captions overstate. → Ch. 25 (warning block in "Table 1").

E66 [A] MY ANALYSIS Tables 1 and 4 — cells of one column come from different models (per-metric selection). (logged by Ch. 25) Location: Table 1 caption ("best ... bounds"), p. 6; Table 2 caption, p. 7 (makes the per-row choice explicit only for BPD). Issue: inverting Eq. (3) on Figure 1's exact points (δ = 0.05, m = 8,009,762 [SOURCE], n = 10,000, no hyperparameter bits) gives an implied compressed size of 3.883 KB for SubLoRA's best top-1 point — the d = 10,000 model of Fig. 2 left (3.878 KB) — versus 16.537 KB (d = 50,000, α = 0.25) for its best BPD point. Subspace: top-1 6.865 KB vs at most 3.948 KB for its BPD point at any α in the grid. LoRA: its BPD point at α = 0.25 (20.737 KB) coincides with its top-1 point (20.737 KB), so LoRA's two cells probably share one model. Each cell is a separate valid statement, but a column is not the profile of one model, and the four cells of a column hold jointly only with probability ≥ 1 − 4δ = 0.80 (union bound). Impact: interpretation only. → Ch. 25 (worked example (e); ch25_tables_audit.py [5], [10]).

E67 [A] Refinement of the Ch. 14 entry on the "Original Model" bound — §6.2, p. 7, gives a one-sentence method. (logged by Ch. 25) Location: §6.2 last paragraph (paraphrased: quantization and arithmetic coding alone, with neither LoRA nor a linear subspace during training, give vacuous bounds), with Tables 1, 2, 4. Issue: the Ch. 14 entry says the computation of 70.76 is not stated; this sentence partly states it (quantization + arithmetic coding of the full model). Still unstated: number of levels, α, empirical risk, whether Eq. (3) was used. MY ANALYSIS With an assumed smoothed training risk of 4.35 bits/token (the validation value), 70.76 implies 14.4–49.5 MiB = 0.98–3.36 bits per parameter over the α grid, consistent with entropy-coding 11 or 17 levels (≤ log₂ 11 = 3.459, log₂ 17 = 4.087 bits/parameter). The same size and risk at the sequence-level m = 8,823,811 would give 67.63–67.64, not Table 4's 65.37; the larger m alone can lower the bound by at most 3.343 (observed −5.39), so the Original Model's sequence-level entry must also differ in empirical risk or size. Impact: the qualitative message (vacuous) stands; the numbers remain unreproducible. → Ch. 14, Ch. 25 (worked example (g), (h)).

E68 [A] App. F, p. 17; Tables 6–7, p. 18–19 — the sampled models are not identified. (logged by Ch. 27) Location: App. F ("Text Generation") and the captions of Tables 6 and 7. Issue: App. F states only the sampling temperature (0.8). It does not say which SubLoRA configuration produced Table 7 (d, r, L_q, model size), whether samples come from the quantized model, whether prediction smoothing (α) was applied during sampling, or what prompt (if any) started each sample. For Table 6, §8 says the model was pretrained "in the standard fashion" while the caption says "pretrained 124 million parameter GPT-2-Small", which could mean OpenAI's released checkpoint or the authors' own nanoGPT run. Impact: no number changes; the reader cannot verify that Table 7 shows the same model whose bound is reported in Table 1, so §8 limitation 4 rests on an unidentified model. → Ch. 27.

E69 [A] App. D, p. 16; Table 5 — the bound setting for the fine-tuning bounds is not stated. (logged by Ch. 26) Location: App. D text and Table 5. Issue: App. D gives the fine-tuning recipe (SubLoRA rank 8, d = 30,000, 5 epochs, learning rate 2·10⁻⁵, quantized checkpoints) but not the bound recipe: which equation (Eq. 1 or the subsampled Eq. 3), which loss (0-1 error on which split), m, δ, the compressed sizes, or whether hyperparameter bits were paid. MY ANALYSIS Under Eq. (1) with m = GLUE training-split size SOURCE: HF datasets-server and δ = 0.05, the pretrained CoLA bound of 38.89 % would leave a budget of at most 3,704 bits (0.45 KB, 0.123 bits per subspace coordinate) even at zero training error (ch26_table3_audit.py [8]); QQP's 35.27 % leaves 130,559 bits. Impact: Table 5 cannot be reproduced or sanity-checked; together with E22 it weakens the "certification" of pretraining's value (the pretrained-vs-random comparison itself is not affected). → Ch. 26, 29.

E70 [A] §7.2, p. 9; Figure 4 — the permuted-context experiment is underspecified. (logged by Ch. 26) Location: §7.2 and Figure 4. Issue: the model size, compressed sizes, α and number of runs of the two bars are not given; the bound is not split into measured loss and complexity; and it is not stated explicitly whether the permuted model's bound is evaluated on permuted documents (the wording "used to train the model and evaluate the bounds" suggests so, in which case the two bars bound risks on different populations). MY ANALYSIS If the whole BPD rise (+0.3126 bits/token) were complexity, the permuted model would need +6.8 KB over a 9.28 KB model (α = 0.25, δ = 0.05, m = 8,009,762). Impact: the bars show that the bound rises, but cannot distinguish "worse generalization" from "shuffled text is harder to predict"; the claim of §7.2 is not established by the figure. → Ch. 26, 29.

E71 [A] Fig. 3 right, p. 8 — addendum to E26: what is coded is not stated. (logged by Ch. 26) Location: Figure 3, right panel; §7.1, pp. 8–9. Issue: besides the missing unit (E26), the panel does not say that the data part uses the smoothed model (one model's dataset sizes vary by 4.68 % across its 8 α values, 8.503·10¹⁰ to 8.900·10¹⁰), nor whether all 9.04·10⁹ training tokens were coded or the value was extrapolated from the 10,000-document subsample. Read as bits, the 124.75M point is 9.410 bits/token, well below the 10.5945 smoothed training BPD of the Table 1 model (ch25 output); token- vs document-weighted averaging (E28) or a different model may explain it. Impact: the panel's values cannot be reproduced exactly; its message (larger models fit training data better; the model costs ≈ 1.7·10⁻⁶ of the total) is unaffected. → Ch. 26.

G. Notes on the authors' code and addenda (not errata of the paper)

Note (Ch. 7, not an error) MY ANALYSIS §6.1, p. 7 — bits paid for the learning rate are unnecessary. The paper adds bits for the choice of learning rate (log₂ 3 = 1.585 bits for the 3-value grid). The learning rate only steers the search; a decoder needs d, r, α, the quantization levels and the quantized w to reproduce h's predictions, not the learning rate, and the bound holds simultaneously for all h. The overpayment is harmless and conservative. → Ch. 7. (logged by Ch. 7)

Code note (Ch. 15) [A] authors' code — sequence-level sampler draws from the first 8,617 chunks only. Location: sample_nonoverlapping_sequences (sublora_utils.py), called from sublora_sublora_pipeline.py with data_size = 8,823,811 (documented as the number of 1,024-token sequences). Issue: the function draws chunk indices below (data_size - 1) // block_size = 8,617, treating data_size as a token count; only the first 8,823,808 tokens (0.098 % of the chunks) can be sampled, while the bound uses \(m\) = 8,823,811. Impact: if Table 4 was produced with the released code, its subsample estimates the average over 8,617 chunks rather than the corpus, and the \(n\)-term does not certify the corpus average as stated (the values may still be close, since nanoGPT shuffles documents before splitting). Cannot be checked from the paper. Printed by ch15_iid.py output [6]. → Ch. 15. (logged by Ch. 15)

Code note [A] (not an erratum of the paper; new, logged by Ch. 18) — authors' code sublora_nn_projectors.py, RoundedDoubleKronQR: the non-square remainder block. Location: the projector used for SubLoRA ("rdkronqr"), which handles non-square \(D\), \(d\) (erratum E09). Issue: the code builds \(P=\Pi\,\mathrm{diag}(K,R)\) with \(K=Q_1\otimes Q_2\) (\(D_0\times d_0\), \(D_0=\lfloor\sqrt D\rfloor^2\), \(d_0=\lfloor\sqrt d\rfloor^2\)), a plain Gaussian remainder \(R\) of shape \((D-D_0)\times(d-d_0)\), and a row permutation \(\Pi\). Hence \(P^\top P=\mathrm{diag}(I,R^\top R)\): (a) \(R\) is not orthogonalized, so \(P^\top P\neq I\) on the remainder (eigenvalues \([0.3958,1.9407]\) for a \(50\times10\) remainder); (b) if \(D-D_0<d-d_0\), \(R^\top R\) is singular and at least \((d-d_0)-(D-D_0)\) directions of \(w\) have \(Pw=0\) (5 zero eigenvalues at \(D=10{,}005\), \(d=410\)); (c) if \(D\) is a perfect square and \(d\) is not, \(R\) has zero rows and \(d-d_0\) columns of \(P\) are identically zero. Impact: unknown for the paper's runs because the SubLoRA parameter count \(D\) is not reported; such coordinates still cost bits after quantization but cannot change the model, so the effect can only worsen reported bounds, never invalidate them (\(P\) is fixed before training). QR-orthogonalizing the remainder fixes (a) and (b) when \(D-D_0\ge d-d_0\). Also minor: Lotfi et al. (2022, §4.1, footnote 3) say the padding is at most \(\sqrt D\times\sqrt d\); it can be up to \(2\lfloor\sqrt D\rfloor\times2\lfloor\sqrt d\rfloor\). → Ch. 18 (demo ch18_subspace.py, output [4]). (logged by Ch. 18)

Note for the merger (not a paper erratum): LoRA parameter counts differ by 47·r between the true GPT-2 vocabulary (V = 50,257: 87,889 at r = 1, 351,556 at r = 4) and nanoGPT's padded vocabulary used by the authors' code (50,304: 87,936 and 351,744). Ch. 5 quotes the padded figure, Ch. 19 gives both. → Ch. 5, 19, 20. (logged by Ch. 19)

Code note (addendum to Ch. 18's note on RoundedDoubleKronQR; not a paper erratum). With D = D_LoRA, the remainder block has D − ⌊√D⌋² rows: 1,092 for D = 351,556 (no lost directions for any plotted d), but only 95 for the code's padded D = 351,744 (593² + 95). Then at d = 50,000 = 223² + 271 at least 176 coordinates of w (0.35 %) cannot affect the model; ≥ 376 at d = 75,000, ≥ 96 at d = 200,000, none at d = 10,000. Whether the authors' runs had D = 351,744 depends on their model file, which is not in the released excerpt. Impact: can only worsen reported bounds slightly, never invalidate them. → Ch. 18, 20. (logged by Ch. 20)

E18 note (Ch. 22) [A] — what the authors' code does for the coded symbols. [SOURCE: authors' code, sublora_bounds_quantize_fns.py] do_arithmetic_encoding runs an exact Decimal coder only for \(d \le 10^4\); for larger \(d\) it returns the formula \(\lceil d\,H(\hat p)\rceil + 1\) without coding. Ch. 22 proves this length is always achieved by the exact coder (codeword length \(\le \lceil\log_2(1/W)\rceil + 1\) with \(\log_2(1/W) = d\,H(\hat p)\) exactly for empirical probabilities), and the demo's 32-bit integer coder matches it (140,089 vs 140,090 bits for d = 50,000). get_message_len adds 16 bits per level (codebook) and \(\lceil\log_2 d\rceil\) bits per level (counts), confirming that the code, unlike the paper's text, counts them. Minor: a count can equal \(d\), which needs \(\lceil\log_2(d+1)\rceil\) bits — differs only when \(d\) is a power of 2, which none of the grid values is. Not an error in any reported number. (logged by Ch. 22)

E18 addendum [M] MY ANALYSIS (new; logged by Ch. 28) — width of the symbol-count header. Location: the header convention the paper inherits for GET_COMPRESSED_SIZE (Alg. 1 lines 17–18; Lotfi et al. 2022 and the authors' get_message_len: \(L_q\lceil\log_2 d\rceil\) bits for the counts). Issue: a count can equal \(d\) (one level used by every entry), and when \(d\) is a power of two (e.g. \(d=256\) or \(1024\)) \(\lceil\log_2 d\rceil\) bits cannot represent the value \(d\); \(\lceil\log_2(d+1)\rceil\) bits always suffice. Impact: at most \(L_q\) extra bits (≤ 17), i.e. a change in the fifth or sixth decimal of any reported bound; validity of the paper's numbers is unaffected in practice. Ch. 28 uses \(\lceil\log_2(d+1)\rceil\). → Ch. 22, 28.

Note (not a paper erratum) [SOURCE: authors' code config_sublora_bounds.yaml]: the public default bound config sets max_quant_iters: 0 and use_kmeans: False, i.e. no quantization-aware training and a uniform-grid codebook, whereas App. C / Alg. 1 describe k-means initialization plus quantization-aware training. Defaults may simply differ from the runs behind the tables. → Ch. 28. (logged by Ch. 28)

H. Verification: what the figures let us reproduce MY ANALYSIS

explainer/code/fig_extract.py reads every plotted point from the PDF's vector graphics. - Fig. 1 (left) points reproduce Table 1 exactly: best SubLoRA BPD = 10.5945 (train) + 1.5278 (complexity) = 12.1224; LoRA 13.0947; Subspace 14.5896; top-1: SubLoRA 0.9641, LoRA 1.0058 (> 1, reported as 100), Subspace 0.9652. - Fig. 2 pins the best BPD model: d = 50,000, α = 0.25 (empirical 10.595, bound 12.122), compressed size 16.5325 KB (plotted value; 135,434 bits). - Plugging these into Eq. (3) with δ = 0.05, m = 8,009,762, n = 10,000, natural logs and 1 KB = 8192 bits gives 12.1222 — reproducing Table 1's 12.12 and confirming the unstated δ, m, log base and KB convention.