All code demos

Every number in the explainer is printed by one of these scripts. Run any of them from the project folder, e.g. .venv/Scripts/python explainer/code/ch09_hoeffding.py, or run everything with .venv/Scripts/python explainer/code/run_all.py (each script asserts that reproduced paper numbers match the paper to its stated rounding).

Shared tools

• pacbound.py — Shared formulas and constants for the 2312.17173 explainer.
• fig_extract.py — Extract the plotted data of Figures 1-4 directly from the vector graphics in paper.pdf.
• build_depmap.py — Draw the concept dependency map of the explainer (two SVG figures).
• build_glossary.py — Merge the per-chapter glossary fragments into one glossary ordered by first use.
• build_codeindex.py — Generate explainer/docs/code.md: an index of every demo script, what it reproduces, and its outputs.
• run_all.py — Run every demo script of the explainer and report pass/fail and runtime.

Chapter demos

Chapter	Script	What it shows / reproduces	Outputs
1	ch01_probability.py	Chapter 1 demo: the probability toolkit, checked by simulation.	outputs/ch01_probability.txt ch01_dice_means.png
2	ch02_memorizer.py	Chapter 2 demo: a memorizer vs a simple rule on a toy task, and what a generalization bound is.	outputs/ch02_memorizer.txt ch02_train_vs_true.png
3	ch03_bits_nats.py	Chapter 3 - Logarithms, bits and nats.	outputs/ch03_bits_nats.txt ch03_lnx.png ch03_surprisal.png
4	ch04_bigram_lm.py	Chapter 4 demo: a character-level bigram language model, built by counting.	outputs/ch04_bigram_lm.txt ch04_bigram_heatmap.png ch04_temperature.png
5	ch05_attention.py	Chapter 5 demo -- the arithmetic inside one transformer block, on numbers small enough to check by hand.	outputs/ch05_attention.txt ch05_param_shares.png
5	ch05_param_count.py	Chapter 5 demo -- where the parameter counts of the paper's GPT-2 models come from.	outputs/ch05_param_count.txt ch05_param_shares.png
6	ch06_metrics.py	Chapter 6 demo: measuring a language model (ch06_metrics.py).	outputs/ch06_metrics.txt ch06_averaging.png ch06_example.png
7	ch07_gd.py	Chapter 7 demo: training by gradient descent, through a reparameterization, from a seed.	outputs/ch07_gd.txt ch07_gd_paths.png ch07_subspace_logreg.png
8	ch08_linalg.py	Chapter 8 demo -- linear-algebra toolkit.	outputs/ch08_linalg.txt ch08_near_orthogonal.png
9	ch09_hoeffding.py	Chapter 9 demo: Hoeffding's inequality, checked by simulation.	outputs/ch09_hoeffding.txt ch09_convexity.png ch09_hoeffding_tails.png
10	ch10_union.py	Chapter 10 demo: the winner's curse and the union-bound repair.	outputs/ch10_union.txt ch10_winners_curse.png
11	ch11_eq1_calculator.py	Chapter 11 demo: an Eq. (1) / Theorem A.1 calculator and a coverage check.	outputs/ch11_eq1_calculator.txt ch11_cheating_prior.png ch11_complexity_curve.png
12	ch12_kraft.py	Chapter 12 demo: codes, prefix-freeness and the Kraft inequality.	outputs/ch12_kraft.txt ch12_header.png ch12_kraft_tree.png
13	ch13_zlib_kolmogorov.py	Chapter 13 demo: Kolmogorov-complexity intuition with real compressors (zlib, bz2, lzma).	outputs/ch13_zlib_kolmogorov.txt ch13_compression_bars.png
14	ch14_budget.py	Chapter 14 demo: compressed size as complexity (the "compression bound").	outputs/ch14_budget.txt ch14_budget.png
15	ch15_iid.py	Chapter 15 - Making the data i.i.d.: documents, not tokens.	outputs/ch15_iid.txt ch15_iid.png
16	ch16_smoothing.py	Chapter 16 demo: prediction smoothing and Theorem A.2.	outputs/ch16_smoothing.txt ch16_alpha_tradeoff.png ch16_clamp.png
17	ch17_subsampling.py	Chapter 17 - the subsampling bound, Eq. (3) / Eq. (8) of arXiv 2312.17173.	outputs/ch17_subsampling.txt ch17_coverage.png
18	ch18_subspace.py	Chapter 18 demo -- random subspace training, theta = theta0 + P w.	outputs/ch18_subspace.txt ch18_intrinsic_dim.png ch18_ptp_error.png
19	ch19_lora.py	Chapter 19 demo -- LoRA: low-rank adaptation.	outputs/ch19_lora.txt ch19_lora.png
20	ch20_sublora.py	Chapter 20 demo -- SubLoRA: theta = theta0 + LoRA(phi0 + P w) (paper Sec. 5, Eq. 4, p. 6).	outputs/ch20_sublora.txt ch20_param_compare.png
21	ch21_quantize.py	Chapter 21 demo: quantization, k-means (Lloyd's algorithm) and quantization-aware training (QAT).	outputs/ch21_quantize.txt ch21_levels.png ch21_qat.png ch21_rate_distortion.png
22	ch22_arith_coder.py	Chapter 22 demo: entropy, symbol codes and a tiny exact arithmetic coder.	outputs/ch22_arith_coder.txt ch22_intervals.png ch22_waste.png
23	ch23_fig1.py	Chapter 23 demo: reading Figure 1 of arXiv 2312.17173 from its exact plotted data.	outputs/ch23_fig1.txt ch23_fig1_replot.png
24	ch24_pipeline.py	Chapter 24 - the pipeline: Algorithm 1 in numbers, hyperparameter bits, Eq. (9) audit.	outputs/ch24_pipeline.txt ch24_eq9_windows.png
25	ch25_tables_audit.py	Chapter 25 demo: auditing the GPT-2-small results of arXiv 2312.17173 (Tables 1, 2, 4 and Figure 2).	outputs/ch25_tables_audit.txt ch25_fig2_replot.png
26	ch26_table3_audit.py	Chapter 26 demo: scale, structure and pretraining (§6.4, §7, App. D, App. F of arXiv 2312.17173).	outputs/ch26_table3_audit.txt ch26_fig3_replot.png ch26_fig4_axes.png ch26_table3.png
27	ch27_discussion.py	Chapter 27 - the paper's own discussion (§8): putting numbers on the first and fourth limitations.	outputs/ch27_discussion.txt
28	ch28_data.py	ch28_data.py -- the KNOWN data-generating process for Chapter 28's mini replication.	ch28_alpha_tradeoff.png ch28_bound_bars.png ch28_d_tradeoff.png
28	ch28_mini_sublora.py	ch28_mini_sublora.py -- Chapter 28: Algorithm 1 (paper App. C, p. 15) end to end, in miniature, on a CPU.	outputs/ch28_mini_sublora.json outputs/ch28_mini_sublora.txt outputs/ch28_mini_sublora_quick.json outputs/ch28_mini_sublora_quick.txt ch28_alpha_tradeoff.png ch28_bound_bars.png ch28_d_tradeoff.png
28	ch28_model.py	ch28_model.py -- tiny GPT, LoRA, SubLoRA (Eq. 4), Kronecker-QR projector, quantization and an exact arithmetic coder, for Chapter 28's mini replication of Algorithm 1 (paper App. C).	ch28_alpha_tradeoff.png ch28_bound_bars.png ch28_d_tradeoff.png
29	ch29_critique_numbers.py	Chapter 29 demo: every headline number used by the critique (Ch. 29) and by the home page (docs/index.md).	outputs/ch29_critique_numbers.txt ch29_effect_sizes.png