[学习笔记] - 向量·矩阵的微分

涉及矩阵和向量的求导可以分为五大类别

• 标量对矩阵
• 标量对向量
• 向量对标量
• 向量对向量
• 矩阵对标量

布局

矩阵求导有两种布局

• 分子布局

$\frac{\partial \mathbf{y}}{\partial x} = \begin{bmatrix} \frac{\partial y_1}{\partial x} \\ \frac{\partial y_2}{\partial x} \\ \vdots \\ \frac{\partial y_n}{\partial x} \end{bmatrix}$

• 分母布局

$\frac{\partial \mathbf{y}}{\partial x} = \begin{bmatrix} \frac{\partial y_1}{\partial x} & \frac{\partial y_2}{\partial x} & \cdots & \frac{\partial y_n}{\partial x} \end{bmatrix}$

可以随时在两种布局间进行转换

标量/向量

• $f \in \mathbb{R}$
• $\mathbf{x} \in \mathbb{R}^n$

$\frac{\partial f}{\partial \mathbf{x}} = \begin{pmatrix} \frac{\partial f}{\partial x_1} & \frac{\partial f}{\partial x_2} &\cdots &\frac{\partial f}{\partial x_n} \end{pmatrix}^T \in \mathbb{R}^n$

$\frac{\partial^2 f}{\partial \mathbf{x}\partial \mathbf{x}^T} = \begin{pmatrix} \frac{\partial^2 f}{\partial x_1 \partial x_1} & \frac{\partial f}{\partial x_1\partial x_2} &\cdots &\frac{\partial f}{\partial x_1\partial x_n} \\ \frac{\partial^2 f}{\partial x_2 \partial x_1} & \frac{\partial f}{\partial x_2\partial x_2} &\cdots &\frac{\partial f}{\partial x_2\partial x_n} \\ \vdots & \vdots & \ddots & \vdots \\ \frac{\partial^2 f}{\partial x_n \partial x_1} & \frac{\partial f}{\partial x_n\partial x_2} &\cdots &\frac{\partial f}{\partial x_n\partial x_n} \end{pmatrix}^T \in \mathbb{R}^n$

对向量的导数是函数关于向量元素的偏导数。因此，得到的导数结果是一向量，与向量的维度一致。

标量/向量的公式

原式	结果
$\frac{\partial a}{\partial \mathbf{x}}$	$\mathbf{0}$
$\frac{\partial au(\mathbf{x})}{\partial \mathbf{x}}$	$a\frac{\partial u(\mathbf{x})}{\partial \mathbf{x}}$
$\frac{\partial(u + v)}{\partial \mathbf{x}}$	$\frac{\partial u}{\partial \mathbf{x}} + \frac{\partial v}{\partial \mathbf{x}}$
$\frac{\partial uv}{\partial \mathbf{x}}$	$v\frac{\partial u}{\partial \mathbf{x}} + u\frac{\partial v}{\partial \mathbf{x}}$
$\frac{\partial g(u(\mathbf{x}))}{\partial \mathbf{x}}$	$\frac{\partial g(u)}{\partial u}\frac{\partial u}{\partial \mathbf{x}}$

---

原式	结果
$\frac{\partial \mathbf{a}^T\mathbf{x}}{\partial \mathbf{x}}$	$\mathbf{a}$
$\frac{\partial \mathbf{x}^T\mathbf{a}}{\partial \mathbf{x}}$	$\mathbf{a}$
$\frac{\partial \mathbf{x}^T\mathbf{x}}{\partial \mathbf{x}}$	$2\mathbf{x}$
$\frac{\partial \mathbf{x}^TA\mathbf{x}}{\partial \mathbf{x}}$	$(A + A^T)\mathbf{x}$
$\frac{\partial (\mathbf{x} - \mathbf{a})^T(\mathbf{x} - \mathbf{a})}{\partial \mathbf{x}}$	$2(\mathbf{x} - \mathbf{a})$
$\frac{\partial (A\mathbf{x} - \mathbf{a})^T(A\mathbf{x} - \mathbf{a})}{\partial \mathbf{x}}$	$2A^T(A\mathbf{x} - \mathbf{a})$
$\frac{\partial (A\mathbf{x} - \mathbf{a})^TC(A\mathbf{x} - \mathbf{a})}{\partial \mathbf{x}}$	$2A^T(C + C^T)(A\mathbf{x} - \mathbf{a})$

标量/矩阵

• $f \in \mathbb{R}$
• $A \in \mathbb{R}^{n \times m}$

$\frac{\partial f}{\partial A} = \begin{pmatrix} \frac{\partial f}{\partial A_{11}} & \frac{\partial f}{\partial A_{12}} & \cdots & \frac{\partial f}{\partial A_{1m}} \\ \frac{\partial f}{\partial A_{21}} & \frac{\partial f}{\partial A_{22}} & \cdots & \frac{\partial f}{\partial A_{2m}} \\ \vdots & \vdots & \ddots & \vdots \\ \frac{\partial f}{\partial A_{n1}} & \frac{\partial f}{\partial A_{n2}} & \cdots & \frac{\partial f}{\partial A_{nm}} \end{pmatrix}$

标量/矩阵的公式

原式	结果
$\frac{\partial tr(A)}{\partial A}$	$\mathbf{I}$
$\frac{\partial tr(AB)}{\partial A}$	$B^T$
$\frac{\partial tr(BA)}{\partial A}$	$B^T$
$\frac{\partial tr(ABA^T)}{\partial A}$	$A(B+B^T)$
$\frac{\partial tr(f(U = g(A)))}{\partial A}$	$\frac{\partial tr(f(U))}{\partial U} \frac{tr(g(A))}{\partial A}$
$\frac{\partial f(A)}{\partial A^T}$	$(\frac{\partial f(A)}{\partial A})^T$
$\frac{\partial \vert A\vert}{\partial A}$	$\vert A\vert(A^{-1})^T$
$\frac{\partial \ln\vert A\vert}{\partial A}$	$(A^{-1})^T$
$\frac{\partial \parallel AX - B\parallel^2_F}{\partial X}$	$2A^T(AX - B)$
$\frac{\partial \parallel XA - B\parallel^2_F}{\partial X}$	$2(XA - B)A^T$

向量/标量

• $\mathbf{y} \in \mathbb{R}^n$
• $x \in \mathbb{R}$

$\frac{\partial \mathbf{y}}{\partial x} = \begin{pmatrix} \frac{\partial y_1}{\partial x} & \frac{\partial y_2}{\partial x} &\cdots &\frac{\partial y_n}{\partial x} \end{pmatrix}^T \in \mathbb{R}^n$

向量/标量的公式

原式	结果
$\frac{\partial \mathbf{a}}{\partial x}$	$\mathbf{0}$
$\frac{\partial a\mathbf{u}(x)}{\partial x}$	$a\frac{\partial \mathbf{u}}{\partial x}$
$\frac{\partial A\mathbf{u}}{\partial x}$	$A\frac{\partial \mathbf{u}}{\partial x}$
$\frac{\partial \mathbf{u}^T}{\partial x}$	$(\frac{\partial \mathbf{u}^T}{\partial x})^T$
$\frac{\partial (\mathbf{u} + \mathbf{v})}{\partial x}$	$\frac{\partial \mathbf{u}}{\partial x} + \frac{\partial \mathbf{v}}{\partial x}$
$\frac{\partial \mathbf{g}(\mathbf{u})}{\partial x}$	$\frac{\partial \mathbf{g}(\mathbf{u})}{\partial x} \frac{\partial \mathbf{u}}{\partial x}$

向量/向量

• $\mathbf{y} \in \mathbb{R}^m$
• $\mathbf{x} \in \mathbb{R}^n$

$\frac{\partial \mathbf{y}}{\partial \mathbf{x}} = \begin{pmatrix} \frac{\partial y_1}{\partial x_1} & \frac{\partial y_2}{\partial x_1} & \cdots & \frac{\partial y_m}{\partial x_1} \\ \frac{\partial y_1}{\partial x_2} & \frac{\partial y_2}{\partial x_2} & \cdots & \frac{\partial y_m}{\partial x_2} \\ \vdots & \vdots & \ddots & \vdots \\ \frac{\partial y_1}{\partial x_n} & \frac{\partial y_2}{\partial x_n} & \cdots & \frac{\partial y_m}{\partial x_n} \end{pmatrix} \in \mathbb{R}^{n \times m}$

向量/向量的公式

原式	结果
$\frac{\partial \mathbf{a}}{\partial \mathbf{x}}$	$\mathbf{0}$
$\frac{\partial \mathbf{x}}{\partial \mathbf{x}}$	$\mathbf{I}$
$\frac{\partial A\mathbf{x}}{\partial \mathbf{x}}$	$A^T$
$\frac{\partial f(\mathbf{u} = g(\mathbf{u}))}{\partial \mathbf{x}}$	$\frac{\partial g(\mathbf{x})}{\partial \mathbf{x}} \frac{\partial f(\mathbf{u})}{\partial \mathbf{u}}$
$\frac{\partial \mathbf{x} \dotplus \mathbf{y}}{\partial \mathbf{x}}$	$diag(y_1, y_2, \cdots, y_n)$
$\frac{\partial (f(x_1), f(x_2), \cdots, f(x_n))}{\partial \mathbf{x}}$	$\begin{pmatrix}f'(x_1) & 0 & \cdots & 0\\0 & f'(x_2) & \cdots & 0\\\vdots & \vdots & \ddots & \vdots\\0 & 0 & 0 & f'(x_n)\end{pmatrix}$

矩阵/标量

• $A \in \mathbb{R}^{m\times n}$
• $x \in \mathbb{R}$

$\frac{\partial A}{\partial x} = \begin{pmatrix} \frac{\partial A_{11}}{\partial x} & \frac{\partial A_{21}}{\partial x} & \cdots & \frac{\partial A_{m1}}{\partial x} \\ \frac{\partial A_{12}}{\partial x} & \frac{\partial A_{22}}{\partial x} & \cdots & \frac{\partial A_{m1}}{\partial x} \\ \vdots & \vdots & \ddots & \vdots \\ \frac{\partial A_{1n}}{\partial x} & \frac{\partial A_{2n}}{\partial x} & \cdots & \frac{\partial A_{mn}}{\partial x} \end{pmatrix} \in \mathbb{R}^{n \times m}$

多维随机变量

$n$个随机变量$X_1, X_2, \cdots, X_n$组成下面的$n$元列向量

$\mathbf{X} = \begin{pmatrix} X_1 \\ X_2 \\ \vdots \\ X_n \end{pmatrix} = \begin{pmatrix} X_1 & X_2 & \cdots & X_n \end{pmatrix}$

多维随机变量的期待值

$E[\mathbf{X}] = \mathbf{u} = \begin{pmatrix} u_1 \\ u_2 \\ \vdots \\ u_n \end{pmatrix} = \begin{pmatrix} E[X_1] \\ E[X_2] \\ \vdots \\ E[X_n] \end{pmatrix} = \begin{pmatrix} E[X_1]&E[X_2]&\cdots&E[X_n] \end{pmatrix}$

• $E[a\mathbf{X} + b\mathbf{Y}] = aE[\mathbf{X}] + bE[\mathbf{Y}]$
• $E[A\mathbf{X} + \mathbf{b}] = AE[\mathbf{X}] + \mathbf{b}$

多维随机变量的方差

$Var(\mathbf{a}^T\mathbf{X}) = \mathbf{a}^T Var[\mathbf{X}]\mathbf{a}$

协方差矩阵

给定$n$元的多维随机变量$\mathbf{X}$和$m$元的多维随机变量$\mathbf{Y}$，则它们的协方差矩阵为

$\begin{aligned} Cov(\mathbf{X}, \mathbf{Y}) = \Sigma &= \begin{bmatrix} Cov(x_1,y_1) & Cov(x_1,y_2) & \cdots & Cov(x_1,y_m) \\ Cov(x_2,y_1) & Cov(x_2,y_2) & \cdots & Cov(x_2,y_m) \\ \vdots & \vdots & \ddots & \vdots \\ Cov(x_n,y_1) & Cov(x_n,y_2) & \cdots & Cov(x_n,y_m) \end{bmatrix} \\ &= E(\mathbf{X}\mathbf{Y}^T) - E(\mathbf{X})E(\mathbf{Y})^T \end{aligned}$

其中$Cov(\mathbf{X}, \mathbf{X})$为$\mathbf{X}$自身的协方差矩阵可以写作

$Cov(\mathbf{X}, \mathbf{X}) = E((\mathbf{X} - \mathbf{u})(\mathbf{X} - \mathbf{u})^T) = E(\mathbf{X}\mathbf{X}^T) - \mathbf{u}\mathbf{u}^T$

• $Cov(A\mathbf{X}, B\mathbf{Y}) = A\ Cov(\mathbf{X}, \mathbf{Y})\ B^T$
• $Cov(A\mathbf{X}, A\mathbf{X}) = A\ Cov(\mathbf{X}, \mathbf{X})A^T$
• $E(\mathbf{X}^TA\mathbf{X}) = \text{tr}A\Sigma + \theta^TA\theta$

Gauss-Markov的定理

我们考虑下列线性回归模型

$y = x\beta + \varepsilon$

我们假定

• $E(\varepsilon\ |\ X) = 0$
  • 表示相互独立
• $var(\varepsilon\ |\ X) = E(\varepsilon\varepsilon'\ |\ X) = \sigma^2_\varepsilon \mathbf{I}_N$
  • 表示异方差性

则以下成立

$\hat{\beta} = (X'X)^{-1}X'y$

Gauss-Markov的定理证明

在一个模型里，一个最好的估计量是拥有最小的方差的。因为我们拥有的数据是$y$，所以我们需要的估计量是$y$的线性方程，即

$\tilde{\beta} = m + My$

• $\beta, m$：$k \times 1$
• $M$：$k \times n$
• $y$：$n \times 1$

同时，因为我们需要的是无偏估计，所以我们需要

$E(\tilde{\beta}) = \beta$

由$\tilde{\beta} = m + My$，我们可以得到

$\begin{aligned} E(\tilde{\beta}) &= E(m + My) \\ &= E(m) + E(My) \\ &= E(m) + ME(y|X) \\ &= m + ME(X\beta + \varepsilon|X) \\ &= m + MX\beta + ME(\varepsilon|X) \\ &= m + MX\beta \end{aligned}$

因此我们可以得到

$\begin{cases} m &= 0 \\ MX &= \mathbf{I}_k \end{cases}$

我们注意到最小二乘推定量$\hat{\beta} = (X^TX)^{-1}X^Ty$，则

$M = (X^TX)^{-1}X^T$

且

$MX = (X^TX)^{-1}X^TX = \mathbf{I}_k$

于是我们知道我们需要寻找以下形式的线性不偏估计量

$\tilde{\beta} = My, \quad MX = \mathbf{I}_k$

为了不失去一般性，我们重新定义$M$为以下形式

$\begin{aligned} M &= (X'X)^{-1}X' + C \\ \\ MX &= \mathbf{I}_k \\ &\Rightarrow ((X'X)^{-1}X' + C)X = \mathbf{I}_k \\ &\Rightarrow \mathbf{I}_k + CX = \mathbf{I}_k \\ &\Rightarrow CX = 0 \end{aligned}$

于是

$\begin{aligned} \tilde{\beta} &= My \\ &= M(X\beta + \varepsilon) \\ &= \beta + M\varepsilon \\ \\ \tilde{\beta} - \beta = M\varepsilon \\ E(\tilde{\beta} - \beta|X) &= 0 \end{aligned}$

于是不偏估计量的协方差矩阵为

$\begin{aligned} E((\tilde{\beta} - \beta)(\tilde{\beta} - \beta)'|X) &= E(M\varepsilon(M\varepsilon)'|X) \\ &= E(M\varepsilon\varepsilon'M'|X) \\ &= M(E\varepsilon\varepsilon'|X)M' \\ &= M\sigma^2_\varepsilon\mathbf{I}_n M' \\ &= \sigma^2_\varepsilon MM' \end{aligned}$

其中

$\begin{aligned} MM' &= ((X'X)^{-1}X' + C)((X'X)^{-1}X' + C)' \\ &= (X'X)^{-1}X'X(X'X)^{-1} + (X'X)^{-1}X'C' + CX(X'X)^{-1} + CC' \end{aligned}$

因为$CX = 0$，同理$C'X' = 0$，则

$MM" = (X'X)^{-1} + CC'$

当$CC' = 0$时，我们得到最好的估计量，可以表现为

$var(\mathbf{x}^T\tilde{\beta}) \geq var(\mathbf{x}^T\hat{\beta})$