Least Squares

Mathematics is a field of study many people who are not exposed to it often find as being either frustrating or for lack of a better word…boring. This may be due to a lack of understanding of the subject on their part coupled with a lack of realization of how important mathematics is in the world. One thing a college professor told me during one of my courses was that, “…math is not difficult, it has a strict set of rules, and as long as you follow those rules, you will do well in the subject…” His statement is quite true, but whether or not math is difficult is more of a personal opinion rather than a fact. It is true however, we must follow the rules and the solutions to the problems have no choice but to appear and make themselves known, which is quite a beautiful thing. But as with everything in life, even this statement is not entirely factual. When solutions to problems don’t make themselves known, mathematics has come up with a solution for that also, and it’s with linear algebra and least squares method that we can come up with a solution. During my recent course in linear algebra I found this to be quite remarkable, because not only did this allow us to come up with a solution to the problem, it allows for the prediction of future events. A statistics or regression analysis course will also study this topic, but the focus here will be in relation to linear algebra and it will be on the broadest sense.

In linear algebra, a common equation used is Ax = b, in fact this equation is one of the foundations of linear algebra, which is known as the matrix equation. So it’s when we cannot get a solution to this matrix equation that we implement the least squares method. Which gives us a good approximation for the solution we seek. Completing a few of these problems in which a professor assigned to me, I really began to see the power of this least squares method. One can determine the future profits of an airline, biomass growth, the list goes on.

Here is a summary that I wrote about the least squares problem, where x is a vector, while x is a unit vector and likewise for b& b:

…when we were faced with an over-determined system of equations Αx = b, we simply gave up and said “the system has no solution” or “the system is inconsistent”, the points are not collinear. What the least squares method seeks to do is to find an xthat minimizes the error or distance with relation to b. This gives us a solution to the problem, even though it is not an exact solution; it is the “best approximation” of a solution to the problem. The definition of this problem given by David C. Lay in his textbook entitled, Linear Algebra and its Applications is:

If Α is m x n and b is in Rm a least squares solution of Αx = b is an x in Rn : || b - Αx || ≤ || b - Αx || for all x in Rn

As Lay also points out, no matter what x is selected the vector Αxwill always be in the column space of Α, Col(Α), As we will see this justifies our use of least squares. The solution to least squares begins with using The Best Approximation Theorem to the subspace Col(Α), so let

b = proj Col(A) to b

Since there is a b in the Col(Α) and an in Rn we arrive at Αx=b because Αx=b is consistent So deriving now by orthogonal decomposition and  AT (b- Αx ) = Ο then by expanding the equation we arrive at ATb- ATx = Ο and then through algebraic manipulation we have AT Αx = ATb which represents our normal equations for Αx = b The solution to which is x.  Wrapping this all up we have AT Α which is an invertible square matrix when the columns of Α are linearly independent.

For our case x = (AT A)-1 ATthus resulting in the least squares error for approximation || b - Αx ||. This problem has some remarkable properties for mathematics and after completing a few of these problems one can see the power math has when explaining the world around us and this problem is a classic example of that power…

The reason I am writing about this is because for me, this was the most intuitive application I have found in a math course and it proves how math applies to the world in so many different ways.


Bentz, Ryan (2014). Least Squares Summary. Blackwood

Echeverria, P. (n.d.). Orthogonal Projections. Instructor Notes.

Lay, D. C. (2012). Linear Algebra and Its Applications. Upper Saddle River: Addison Wesley.