When I took Calc III (MAT 307 for me at Stony Brook), we used Hessian matrices in order to perform the multivariable equivalent of the second derivative test for determining whether a point was a maximum, minimum, saddle point, or point of inflection.
The way we did it was by finding the hessian matrix, which was the matrix of all the second partial derivatives. We would then determine whether the matrix was positive-definite, negative-definite, or neither by using the determinant of the matrix to inspect the eigenvalues.
I sort of just accepted it, because it kind of make sense to me; a “positive-definite” matrix corresponded to a positive second derivative in single-variable calculus, and a “negative-definite” matrix correspond to a negative second-derivative in single-variable calculus. It just made enough sense on the surface for me to not bother looking into it.
But then, as I learned more linear algebra, I became more curious about what positive and negative determinate meant. It turns out that if matrix H was positive definite, vTHv>0 for all v – basically, it meant that no matter what vector we put through H, we would get a vector pointing more or less in the same direction. The opposite held if H was negative definite: vTHv<0 for all v, meaning that no matter what vector we put through H, we would get a vector pointing more or less in the opposite direction.
This gave me the idea that perhaps vTHv represented some sort of directional second derivative, and after some searching online it turns out that yes, vTHv is equivalent to the directional derivative of the function.
I didn’t actually find a proof of this (then again, I didn’t search very hard), but I decided to prove it myself as it couldn’t be too difficult. It wasn’t; the only knowledge required was the definition of matrix multiplication, transposes, dot products, product rule for derivatives, and the multivariable chain rule. Once you’ve understood and know how to use all of the previous concepts, you might be interesting in seeing the proof I wrote up (attached as a PDF because as a subscriber to the free wordpress plan, I can’t use Mathjax to embed my proof into the page).
Once you understand that vTHv represents the directional second derivative, you also understand why we check that a hessian matrix is positive-definite or negative-definite. A positive-definite hessian matrix means that the second derivative is positive in every direction, and a negative-definite hessian matrix means that the second derivative is negative in every direction. If it’s neither, it means that there could be some directions where the second derivative is positive and some other directions where the second derivative is negative, indicating a saddle point.
Anyway, what’s left for me to understand is why inspecting the eigenvalues tells us whether a matrix is positive or negative definite, but hopefully that will be resolved in a future blog post.
All You Distrust 