design-matrix

An $N\times (M+1)$ matrix $\mathbf{\Phi }$ whose row $i$ is the basis-function vector $\mathbf{\varphi }({\mathbf{x}}_i{)}^{\top }$ evaluated at training input ${\mathbf{x}}_i$. Compactly encodes “all training inputs through all basis functions” so that $\hat{\mathbf{y}}=\mathbf{\Phi }\mathbf{w}$ is a single matrix-vector product, and the OLS solution becomes $({\mathbf{\Phi }}^{\top }\mathbf{\Phi }{)}^{-1}{\mathbf{\Phi }}^{\top }\mathbf{y}$.

Construction

Given training inputs $\{{\mathbf{x}}_i{\}}_{i=1}^N$ and basis functions $\{{\varphi }_j{\}}_{j=0}^M$:

$\mathbf{\Phi }=\left(\begin{array}{cccc}{\varphi }_0({\mathbf{x}}_1)& {\varphi }_1({\mathbf{x}}_1)& \cdots & {\varphi }_M({\mathbf{x}}_1)\\ {\varphi }_0({\mathbf{x}}_2)& {\varphi }_1({\mathbf{x}}_2)& \cdots & {\varphi }_M({\mathbf{x}}_2)\\ ⋮& ⋮& \ddots & ⋮\\ {\varphi }_0({\mathbf{x}}_N)& {\varphi }_1({\mathbf{x}}_N)& \cdots & {\varphi }_M({\mathbf{x}}_N)\end{array}\right)$

By convention ${\varphi }_0(\mathbf{x})=1$, so the first column is all ones — pairing with the intercept weight $w_0$.

Dimensions. $\mathbf{\Phi }$ is $N\times (M+1)$:

• $N$ = number of training examples (rows)
• $M+1$ = number of basis functions including the intercept (columns)

Why It’s Useful

The model ${\hat{y}}_i={\mathbf{w}}^{\top }\mathbf{\varphi }({\mathbf{x}}_i)$ for all $i$ becomes a single matrix product:

$\hat{\mathbf{y}}=\mathbf{\Phi }\mathbf{w}$

The OLS objective ${\sum }_i(y_i-{\hat{y}}_i{)}^2$ becomes $\Vert \mathbf{y}-\mathbf{\Phi }\mathbf{w}{\Vert }^2$, which differentiates to the normal equation in one line.

Whatever the basis functions are — polynomial, Gaussian, sigmoidal, custom — the math is the same. The design matrix decouples what features you use from how you fit.

Examples

Polynomial basis (degree $M$, single input $x$):

$\mathbf{\Phi }=\left(\begin{array}{ccccc}1& x_1& x_1^2& \cdots & x_1^M\\ 1& x_2& x_2^2& \cdots & x_2^M\\ ⋮& & & & ⋮\\ 1& x_N& x_N^2& \cdots & x_N^M\end{array}\right)$

This is a Vandermonde matrix — a recurring object in interpolation theory.

Gaussian RBF basis ($M$ centres at ${\mu }_1,\dots ,{\mu }_M$, width $s$):

${\mathbf{\Phi }}_{i,j}=\{\begin{array}{ll}1& \text{if }j=0\\ \exp \left(-\frac{(x_i-{\mu }_j{)}^2}{2s^2}\right)& \text{otherwise}\end{array}$

Multi-input ($\mathbf{x}\in {\mathbb{R}}^d$, plain linear basis):

$\mathbf{\Phi }=\left(\begin{array}{ccccc}1& x_1^{(1)}& x_2^{(1)}& \cdots & x_d^{(1)}\\ 1& x_1^{(2)}& x_2^{(2)}& \cdots & x_d^{(2)}\\ ⋮& & & & ⋮\\ 1& x_1^{(N)}& x_2^{(N)}& \cdots & x_d^{(N)}\end{array}\right)$

Practical Notes

• The first column is always ones. This is the dummy basis function ${\varphi }_0\equiv 1$ that pairs with the intercept $w_0$. Forgetting it is a common bug — fits are forced through the origin.
• Standardise inputs first. Polynomial and Gaussian bases on raw inputs can have wildly different column magnitudes, making ${\mathbf{\Phi }}^{\top }\mathbf{\Phi }$ ill-conditioned.
• Tall vs wide. $\mathbf{\Phi }$ is “tall” when $N>M$ (more examples than parameters) — the standard setting where OLS works. “Wide” ($M>N$) makes ${\mathbf{\Phi }}^{\top }\mathbf{\Phi }$ rank-deficient and OLS degenerates.

Connections

• ordinary-least-squares — uses $\mathbf{\Phi }$ in the normal equation $\mathbf{w}=({\mathbf{\Phi }}^{\top }\mathbf{\Phi }{)}^{-1}{\mathbf{\Phi }}^{\top }\mathbf{y}$.
• linear-regression — the model whose predictions are $\hat{\mathbf{y}}=\mathbf{\Phi }\mathbf{w}$.
• non-linear-transformation — the basis-expansion idea; the design matrix is the “$\varphi$-space training set”.