Neural networks can be over-parameterized and still achieve good results. The Neural Tangent Kernel (NTK) helps explain how this works, especially when the number of model parameters is greater than the amount of training data. It does this by treating neural networks as a kernel that evolves during training via gradient descent. NTK provides insights into why wide neural networks can consistently converge to a global minimum when trained on an empirical loss function. --- Why it matters: Understanding NTK has implications for designing and optimizing wide neural networks, which are commonly used in deep learning applications. This knowledge can help engineers develop more efficient and effective models that take advantage of the benefits of over-parameterization. Source: https://lilianweng.github.io/posts/2022-09-08-ntk/