Changelog

Append-only log of wiki ingests.

2026-04-20

• Initialized vault structure. Created raw/ and wiki/ folders, index.md, and CHANGELOG.md.
• Ingested week 1. Created: weeks/week-01.md, concepts/supervised-learning.md, concepts/logistic-regression.md, concepts/sigmoid-function.md, concepts/generalization.md, concepts/decision-boundary.md, concepts/discriminative-vs-generative-models.md (stub). Updated index.md.

2026-04-26

• Ingested week 2. Created: weeks/week-02.md, concepts/maximum-likelihood-estimation.md, concepts/cross-entropy-loss.md, concepts/gradient-descent.md, concepts/convex-function.md, concepts/taylor-polynomial.md, concepts/hessian-matrix.md, concepts/newton-raphson-method.md, concepts/iteratively-reweighted-least-squares.md. Updated index.md.
• Enhanced taylor-polynomial.md. Added “Why polynomials?” motivation, “Coefficients as Independent Derivative Controls” with the $n!$ cascade explanation, and a “Convergence and Radius of Convergence” section with three new active-recall questions.
• Removed empty gradient-descent.md stub at vault root. It was shadowing the populated wiki/concepts/gradient-descent.md in Obsidian’s wikilink resolution.
• Ingested week 3. Created: weeks/week-03.md, concepts/non-linear-transformation.md, concepts/support-vector-machine.md, concepts/margin.md, concepts/quadratic-programming.md. Updated index.md.
• Cross-referenced Notion exports (raw/p1.zip, raw/p2.zip). Enhanced generalization.md with the “Closeness vs Attainability” two-pillars framing and marble-jar metaphor; added Strengths and Limitations section to logistic-regression.md; added “MLE is the source of standard loss functions” table (Bernoulli → cross-entropy, Gaussian → MSE, etc.) to maximum-likelihood-estimation.md; added the “linear in $\varphi$-space, non-linear in original” reframing to non-linear-transformation.md.
• Ingested week 4. Created: weeks/week-04.md, concepts/lagrangian.md, concepts/kkt-conditions.md, concepts/kernel-trick.md, concepts/mercers-condition.md, concepts/gaussian-kernel.md, concepts/polynomial-kernel.md. Updated index.md. Covers SVM dualisation, KKT and complementary-slackness as the structural source of support-vector sparsity, the kernel trick, Mercer’s condition and composition rules, and the polynomial / Gaussian kernels.

2026-05-04

• Ingested week 11. Created: weeks/week-11.md, concepts/validation.md, concepts/cross-validation.md, concepts/learning-principles.md. Updated index.md. Two-part week: (1) Validation as the second cure for overfitting (the first being regularisation from week 10). Hold out ${\mathcal{D}}_{\text{val}}\subset \mathcal{D}$ of size $K$, train on the rest, evaluate on ${\mathcal{D}}_{\text{val}}$. $E_{\text{val}}(g^{-})$ is an unbiased estimate of $E_{\text{out}}(g^{-})$ with variance ${\sigma }_{\text{val}}^2={\sigma }^2(g^{-})/K\le 1/(4K)$ for binary classification (Bernoulli per-example error, $p(1-p)\le 1/4$). The K trade-off: large $K$ → low variance but $g^{-}$ trained on too few examples; small $K$ → $g^{-}\approx g$ but noisy estimate. Rule of thumb $K=N/5$. After model selection picks ${\mathcal{H}}_{m^{\ast }}$, retrain on full $N$ — $E_{\text{out}}(g_{m^{\ast }})\le E_{\text{out}}(g_{m^{\ast }}^{-})$. Validation generalisation bound: $E_{\text{out}}(g_{m^{\ast }})\le E_{\text{val}}(g_{m^{\ast }}^{-})+O(\sqrt{\log M/K})$ — only $\log M$ because $M$ is a small finite candidate list. Validation vs regularisation: both estimate the same overfit penalty equation $E_{\text{out}}=E_{\text{in}}+\text{penalty}$, regularisation estimates the penalty term, validation estimates $E_{\text{out}}$ directly. Cross-validation removes the $K$ trade-off by using all data: LOOCV ($V=N$, almost unbiased for ${\bar{E}}_{\text{out}}(N-1)$, $N$ trainings — closed form for linear regression via hat matrix), V-fold ($V=10$ practical default). (2) Three learning principles closing the module’s theory: Occam’s Razor — simpler hypothesis sets (small $\Omega (\mathcal{H})$, small VC dimension) generalise better; “better” means lower $E_{\text{out}}$, not aesthetic elegance. Sampling Bias — if $P_{\text{train}}\ne P_{\text{deploy}}$, the i.i.d. assumption fails and the VC bound is silent; no algorithm rescues a model trained on the wrong distribution. Data Snooping — any test-set influence on any decision (preprocessing, feature engineering, hyperparameter selection) contaminates it; reported $E_{\text{test}}$ is then biased downward. Discipline: lock ${\mathcal{D}}_{\text{test}}$ in the safe, evaluate once at the end. Lec 2’s recap of weeks 7–10 (probabilistic ML, VC vs bias-variance, OLS = MLE, MAP = ridge) was not duplicated — already covered by existing concept pages.

2026-05-04

• Generated flashcards for weeks 1–6. Created wiki/exam-prep/flashcards/ and added one Markdown file per week (week-01.md through week-06.md) following the Anki obsidian-to-anki callout format (> [!question]- ...). Cards are atomic, derived from week pages and concept pages, and grouped under ## subtopic headings. Week 6 (revision) is a synthesis file with cross-week comparison cards rather than fresh-content cards. Total ~70 cards across the six weeks.
• Documented flashcard-creation conventions in .obsidian/CLAUDE.md under a new ### Flashcards section in the Core Operations area: output path, frontmatter shape, atomic-card rules, voice/notation conventions, target card count per week, and append-to-CHANGELOG step.

2026-05-04

• Ingested week 10. Created: weeks/week-10.md, concepts/overfitting.md, concepts/regularization.md, concepts/lasso-regression.md. Updated index.md. Two-part week: (1) Overfitting — low $E_{\text{in}}$ but high $E_{\text{out}}$. Six common causes (model complexity, data scarcity, too many epochs, no regularisation, high-variance features, poor data processing). The “two-learners” experiment: even when both learners know the target is degree-10, ${\mathcal{H}}_2$ beats ${\mathcal{H}}_{10}$ on test error if $N$ small — the right hypothesis class isn’t enough; need data to constrain it. Stochastic noise (label randomness $ϵ$) and deterministic noise (target complexity beyond $\mathcal{H}$) are operationally equivalent — both encourage overfitting. MLE is structurally unable to prevent this because it has no preference for simpler hypotheses. (2) Regularisation as the structural cure. Built up from hard constraint ($w_3=\cdots =w_{10}=0$), to looser constraint (sparsity: $\ge 8$ of $w_q=0$), to soft constraint ${\mathbf{w}}^{\top }\mathbf{w}\le C$. Lagrangian transforms constrained → unconstrained: augmented error $E_{\text{aug}}=E_{\text{in}}+(\lambda /N){\mathbf{w}}^{\top }\mathbf{w}$, equivalent for paired $C,\lambda$. Closed-form solution ${\mathbf{w}}_{\text{REG}}=({\mathbf{Z}}^{\top }\mathbf{Z}+\lambda \mathbf{I}{)}^{-1}{\mathbf{Z}}^{\top }\mathbf{y}$ — recovers ridge regression from week 8 by a different derivation route (constrained optimisation rather than Bayesian MAP). Effective VC dimension $d_{\text{EFF}}(\mathcal{H},\mathcal{A})=d_{\text{VC}}(\mathcal{H}(C))$ smaller than nominal because the algorithm only navigates within the ball. L1 (lasso) replaces the round constraint ball with a corner-bearing diamond, producing sparse solutions; corresponds to Laplace prior in Bayesian view. Choosing $\lambda$ depends on ${\sigma }^2$, target complexity, $N$ — none observable, hence cross-validation (next week).

2026-05-04

• Ingested week 9. Created: weeks/week-09.md, concepts/dichotomy.md, concepts/growth-function.md, concepts/break-point.md, concepts/vc-dimension.md, concepts/bias-variance-decomposition.md, concepts/learning-curve.md. Updated index.md. Two-part week: (1) Replacing $M$ in the generalisation bound with a finite quantity. Dichotomies count distinct labellings on $N$ specific inputs ($\le 2^N$). The growth function $m_{\mathcal{H}}(N)$ takes the max over inputs. A break point $k$ — smallest $N$ with $m_{\mathcal{H}}(k)<2^k$ — forces $m_{\mathcal{H}}(N)$ to grow polynomially: $m_{\mathcal{H}}(N)\le N^{d_{\text{VC}}}+1$. The VC dimension $d_{\text{VC}}$ equals the largest shatterable $N$, equivalently break point minus one. For perceptron in ${\mathbb{R}}^d$: $d_{\text{VC}}=d+1$ (number of free parameters). The VC bound $\mathbb{P}[|E_{\text{in}}-E_{\text{out}}|>ϵ]\le 4m_{\mathcal{H}}(2N)e^{-{ϵ}^{2}N/8}$ contracts as $N\to \infty$ for finite $d_{\text{VC}}$. Sample complexity: theory says $N\approx 10,000d_{\text{VC}}$, practice says $N\approx 10d_{\text{VC}}$. SVMs achieve generalisation by margin-restricting to fat hyperplanes: $d_{\text{VC}}(\rho )\le \lceil R^2/{\rho }^2\rceil +1$, independent of $d$. (2) Bias–variance as the average-case complement to VC’s worst case. Decomposition ${\mathbb{E}}_{\mathcal{D}}[E_{\text{out}}]=\text{bias}+\text{var}+{\sigma }^2$ for squared loss. Bias = how far average hypothesis $\bar{g}$ is from $f$; variance = how much $g^{(\mathcal{D})}$ fluctuates around $\bar{g}$. Optimal model complexity depends on $N$ — the $\sin (\pi x)$ example shows constant beats line at $N=2$ but loses at $N=5$. Learning curves visualise the decomposition; for linear regression, exact form $E_{\text{in}}={\sigma }^2(1-(d+1)/N)$, $E_{\text{out}}={\sigma }^2(1+(d+1)/N)$, gap decays as $1/N$.

2026-04-28

• Ingested week 8. Created: weeks/week-08.md, concepts/bayesian-linear-regression.md, concepts/ridge-regression.md, concepts/hoeffding-inequality.md, concepts/generalization-bound.md. Updated index.md. Two-part week: (1) Bayesian regression — Gaussian prior + Gaussian likelihood → closed-form Gaussian posterior; MAP estimate is exactly ridge regression, so L2 regularisation is the negative log of a Gaussian prior (regularisation = prior). (2) Learning theory — Hoeffding’s inequality bounds $|E_{\text{in}}-E_{\text{out}}|$ with probability $\ge 1-2e^{-2{ϵ}^{2}N}$; union bound over $M$ hypotheses gives the generalization bound $E_{\text{out}}\le E_{\text{in}}+\sqrt{(1/2N)\log (2M/\delta )}$. The two opposing forces ($M$ small for tight bound vs $M$ large for good fit) formalise the bias-variance trade-off. Sets up VC dimension via the dichotomy/effective-$M$ idea.

2026-04-27

• Ingested week 7. Created: weeks/week-07.md, concepts/linear-regression.md, concepts/ordinary-least-squares.md, concepts/design-matrix.md, concepts/gaussian-distribution.md, concepts/bayes-law.md. Updated index.md. Covers the pivot from classification to regression, OLS via the normal equation $\mathbf{w}=({\mathbf{\Phi }}^{\top }\mathbf{\Phi }{)}^{-1}{\mathbf{\Phi }}^{\top }\mathbf{y}$, basis expansion (linear in parameters, non-linear in inputs), and the probabilistic interpretation that justifies squared error: under additive Gaussian noise, MLE for $\mathbf{w}$ is identical to OLS. Frames this as the regression analog of week 2’s logistic-regression-as-MLE result — the noise model determines the loss.
• Reorganised image assets. Created raw/images/ and renamed all 22 dated Screenshot 2026-04-26 at *.pm.png files to descriptive content-based names (e.g., svm-intro.png, kernel-trick-input-feature-space.png, soft-margin-c-effect.png). Updated all 22 references across the wiki. Obsidian’s ![[name]] syntax resolves by basename so paths didn’t need changing — just filenames.
• Added visuals to weekly synthesis pages. Embedded soft-margin motivation, slack-geometric, primal-comparison, and $C$-effect figures across week-05.md, slack-variables.md, and soft-margin-svm.md. Cross-referenced the SVM primal→dual chain, kernel-trick input→feature-space, and Gaussian bell-curve into week-04.md (same images already used in concept pages — Obsidian shares a single asset across all references).
• Added cross-week synthesis pages. Created topics/optimization-algorithms.md (compares GD, Newton-Raphson/IRLS, SMO — when each applies, cost, role of curvature/constraints) and topics/classification-approaches.md (compares LogReg, hard-margin SVM, soft-margin SVM — same hypothesis form, different fitting criteria, hinge vs cross-entropy as the structural source of SVM sparsity). Motivated by the week-6 revision lecture which presents these groupings as the module’s organising structure. Updated index.md.
• Ingested week 5. Created: weeks/week-05.md, concepts/soft-margin-svm.md, concepts/slack-variables.md, concepts/sequential-minimal-optimization.md. Updated index.md. Covers the soft-margin relaxation (slack variables, hyperparameter $C$, the box constraint $0\le a\le C$), the three-way support-vector partition from KKT (non-SV / margin-SV / bound-SV), and the SMO decomposition algorithm — including why two multipliers is the minimum updateable subset, the analytic update rule, clipping to feasibility, and pair-selection heuristics.
• Embedded slide screenshots. Added the SVM intro and non-linear-margin pictures to support-vector-machine.md; the distance-to-hyperplane and max-margin pictures to margin.md; the input/feature-space cartoon to kernel-trick.md; the bell-curve and decision-boundary plots to gaussian-kernel.md; the SVM primal→dual derivation chain to lagrangian.md; the differential-curvature ellipses, difficult-topology plot, and steepest-descent paraboloid to gradient-descent.md; the linear-classifier+sigmoid figure to logistic-regression.md; and the log-likelihood-to-cross-entropy derivation to cross-entropy-loss.md.