The Rambling Mind

This is a fun problem. Not just because it's entertaining to look at it geometrically, but also because it exercises a lot of fundamental linear algebra muscle, which makes it an ideal review of concepts from time to time.

Let's say you have a point $p_{0}$ that is floating above some plane $N$ and you want to project that point onto the plane. The question of projection is really a question of finding the quickest route (in lamens terms) from point $p_{0}$ to the plane, which we will identify as $p_{1}$. Another way to look at it would be to imagine dropping point $p_{0}$ straight onto the surface of the plane (imagining gravity to pull the point straight to the plane). From this analogy, the projection is then the spot (or point $p_{1}$) on the plane at which $p_{0}$ landed when it fell from its original position. Graphically it would look something like this:

But let's be a bit more precise when we talk about plane $N$ (or the surface atop which the point falls).

Two vectors are linearly independent if neither can be described in terms of the other (such as one being the multiple of the other). Another way of saying this is if two vectors are linearly independent, then neither is a linear combination of the other. However - and this is important - all linear combinations of two linearly independent vectors span (or fully make up) a plane. From this definition one can imagine a plane being a web-work of an infinite number of crisscrossing vectors, where each vector is some simple combination of two base vectors. So in linear algebra when we talk about a plane, we talk about the space spanned by the basis formed from two linearly independent vectors.

Given all of this so far we can begin to see what's really happening when we project a point onto a plane: the projection is the point on the plane - defined by a linear combination of the plane's basis vectors - that is closest to the point suspended off the plane (closest in terms of Euclidean distance). So what we're looking for is a linear combination of the plane's basis vectors that creates a point on the plane as close as possible to the point not in the plane.

What's interesting about all of this is there's no need to limit ourselves to a point and plane; we can just as easily talk about projecting a point onto a line, or a 3-dimensional subspace onto a 4-dimensional one, and so on. To see what I mean, let's look at a simple example of projecting a point onto a line.

Here we see point $p_{0}$ described by a vector $\vec{v}$. Our goal is to project $\vec{v}$ onto the line $L$ spanned by the unit vector $\vec{l}$ (see how even when we're not talking about a plane, we're still talking about a space being spanned by a linear combination of one or more vectors; in this case the line we're projecting onto is a space spanned by linear combinations of a single unit vector that lies along the line). If we continue our analogy of dropping the point $p_{0}$ (or vector $\vec{v}$) onto the subspace, then $\vec{e}$ can be seen as the path taken by $p_{0}$ as it falls onto $L$ to point $p_{1}$.

Let's pause for a moment and think about what's happening here. If we were to "drop" the point $p_{0}$ straight onto $L$, the path taken ($\vec{e}$) would be exactly perpendicular or orthogonal to $L$ (and therefore each and every linear combination of our unit vector $\vec{l}$, including $\vec{l}$ itself). This exact same situation happens if we were projecting a point onto a plane, but this time the vector which follows the projection is a linear combination of each and every vector which spans the plane instead. And again, this concept can be extended to higher level dimensions.

Continuing on, vector subtraction shows $\vec{e}$ can also be defined as
(1) $\vec{e} = \vec{v} - \vec{p_{1}}$.

And since we also know that $\vec{p_{1}}$ is a linear combination of some basis for the line $L$ (defined by some scalar $x$ applied to $\vec{l}$),  we can then define $p_{1}$ as
(2) $p_{1}=x\vec{l}$.

Combining (1) and (2) gives us
(3) $\vec{e} = \vec{v} - x\vec{l}$.

Going back to our earlier observation regarding orthogonality, taking the dot product of any vector within the line $L$ to the vector $\vec{e}$ should result in zero, then. Therefore
(4)$(\vec{v} - x\vec{l})\cdot\vec{l} = 0$

And finally, after some algebraic manipulation, we see that
(5) $x = \frac{\vec{v}\cdot\vec{l}}{\vec{v}\cdot\vec{v}}$

So the projection of $p_{0}$ onto $L$ is simply $l$ scaled by (multiplied by) $x$ (which is the dot product of $\vec{v}$ and $\vec{l}$ divided by the dot product of $\vec{v}$ with itself).

Can we extend this to projecting a point onto a plane? Of course. The only difference is that instead of projecting a point onto a space spanned by a vector, we are projecting a point onto a space spanned by two vectors; something which can easily be encoded in a matrix. So we simply make $\vec{l}$ a matrix, say $N$. Then we update our equation like this:

$\vec{x} = (N^{T}N)^{-1}(N^{T}\vec{v})$.

Therefore mutiplying $\vec{x}$ by the basis of our plane (which is $N$) will identify the point on the plane which is the projection of $p$.

I purchased two books from Amazon and am happy they arrived today. The first is one I purchased mostly for the problem set and the fact it goes alongside a particular course I'm following at the moment -

Second is this -

And finally I purchased a Nexus 7 from Staples today. Very nice device.

Thus far we have been looking at systems of linear equations in the algebraic form of -

$4x+2y=10$
$5x+3y=14$

But as with most things mathematical, brevity is desired; so this particular form is not used too often as it is both redundant and unnecessary. Instead, systems of linear equations are often expressed in matrix form. The above system can therefore be expressed as a matrix in this manner -

$\begin{bmatrix}4 & 2 & 10\\5 & 3 & 14\end{bmatrix}$

This is important because it exposes a few interesting ideas surrounding systems of linear equations. One such idea is the use of columns and rows to identify individual elements; in other words each element gets a row (i) and column (j) index (note that i and j are arbitrary and used only as an example). Another interesting thing to note is that each of the columns correspond to the same variable (column 1, for example, consists of 4 and 5, both coefficients of x).

Indeed, expanding upon this notion of columns being comprised of the coefficients for the same variable, we see ourselves becoming more comfortable grouping the columns so that we look at columns one and two distinctly (since they are made up of different "things", or variables: one is made up of "thing" x, while another is made up of "thing" y). We could go so far as to break up the matrix, then, into two separate sub-matrices being equivalent to another sub-matrix:

$x\begin{bmatrix}4\\5\end{bmatrix} + y\begin{bmatrix}2\\3\end{bmatrix} = \begin{bmatrix}10\\14\end{bmatrix}$

A more concise term for this "sub-matrix" of only one column is a vector. Therefore, we can say a matrix is made up of vectors.

We'll stop here, but spend some time glancing over things and seeing how all we're really doing is restating something (a system of linear equations), into a more concise and compact form.

In my previous post I made an observation that, in the case of the following equation -

$4x+2y=10$

we were underdetermined until we got our hands on another equation (that is, we couldn't determine a single solution to the equation). The reason was due to the fact we had one equation, but two separate unknowns. Changing one unknown made the other change too (to balance out the equality), and so we found ourselves with limitless amounts of possibilities. Once we added the following equation as another piece of evidence, we claimed we could determine a solution that satisfied both equations -

$5x+3y=14$

If you haven't caught on yet, I'm using the word determined a lot; there's a reason for that. Mainly because determining whether a system of equations (such as the one made up of the two equations here) is very important. Why? Well, mostly because we don't want to spend the next twenty years of our life trying to find a solution to a system of equations that may not have a solution. And yes, you read that right: not all systems have solutions.

Take this pair of equations as an example -

$4x + 16y = 9$
$1x + 4y = 8$

Now try as you might, you're never going to find a solution. Why? Well, that's a good question. There's a geometric answer as well as an algebraic one. In this post we'll take a look at the algebraic one. Let's try to solve for y in terms of x.

So let's take the first equation of the two equations I just presented and put $x$ on one side:

(1) $x = \frac{9 - 16y}{4}$

Now let's plug this into the last of the two equations I presented (replacing statement (1) in for $x$)

(2) $\frac{9 - 16y}{4} + 4y = 8$

Let's expand out the fraction:

(3) $\frac{9}{4} - \frac{16y}{4} + 4y = 8$

Let's subtract $\frac{9}{4}$ from both sides:

(4) $-\frac{16y}{4} + 4y = 8 - \frac{9}{4}$

Let's finish the fraction on the left-hand side:

(5) $-4y + 4y = 8 - \frac{9}{4}$

Let's take out y:

(6) $y(-4 + 4) = 8 - \frac{9}{4}$

Let's divide both sides by  $(-4 + 4)$

(7) $y = 8 - \frac{\frac{9}{4}}{(-4 + 4)}$

....and...oh crap. See what we did? We've divided a number by zero (-4 + 4 = 0). This results in infinity, and therefore we cannot recover. This means we cannot solve for y. That means we cannot solve this system of equations. It's unsolvable.

Trying to solve for x results in the same situation: a division by zero.

Let's step back for a moment here and look at what we did in the more general sense.

Take two systems:

$ax + by = r$
$cx + dy = s$

Again, let's solve for x just as we did before:

(1) $x = \frac{r - by}{a}$

..and continue by inserting this into the second equation:

(2) $c\frac{r - by}{a} + dy = s$

...expanding out the fraction:

(3) $\frac{cr}{a} - \frac{cby}{a} + dy = s$

...subtracting our first fraction on both sides:

(4) $-\frac{cby}{a} + dy = s - \frac{cr}{a}$

Take out y:

(5) $y(-\frac{cb}{a} + d) = s - \frac{r}{a}$

Put y on one side:

(6)  $y = \frac{s - \frac{r}{a}}{(-\frac{cb}{a} + d)}$

So the most we can say now is the denominator, $(-\frac{cb}{a} + d)$, cannot equal zero (because we saw from before when it did, we failed to solve the system).

So $(-\frac{cb}{a} + d)$ cannot equal zero.

A bit of algebraic fiddling and we'd see that this funky expression resolves to $ad - bc$. So, by simply subtracting $bc$ from $ad$ and seeing whether it equals zero, we can easily determine whether the system is solvable.

Interestingly, this equation ($ad - bc$) for a system of equations with two equations and two unknowns is called the determinant. In most linear algebra texts it's given as a mysterious algorithm that somehow, magically, declares whether we can solve a system. In reality, it's no more than this. In linear algebra, when say the determinant is zero, that means the system cannot be solved.

So...that's how you do it, that's how you can decide whether to attempt to solve a system (at least for a system of equations where there are two equations and two unknowns): simply determine whether $ad - bc$ equals zero before starting.

First, let me explain what I'm doing here: I'm far along in my study of linear algebra, but have yet to find a successful medium to review/teach the topics I know quite well. This is a problem as one of the best ways to learn and to reinforce what you know is to teach. So that's the goal of writing about math on this blog. Even though the topics are something I've known for a long time now, it's good to reinforce. So, my hope is this blog will always stay months behind my understanding (at least for a while).

Second, let me do it. But let me give the most basic of basics first.

We have all seen equations of the form:

$ax + b = c$

What is this? This is a linear equation; meaning it's an equation that has nothing more to it than a constant ("b" in this example), and multiples of (products of) variables to no power higher than one. Following that definition, we can say this is also a linear equation:

$ax + by + c = d$

The only difference is the inclusion of yet another term $y$.

If I were to ask you to solve the first linear equation by giving you values for a, b, and c, you more than likely could work out the answer using the most elementary understanding of algebra. Let's try it.

$4x + 2 = 10$

The solution is $x = 2$ since 4 times  2 is 8, and 2 + 8 = 10. In addition to the answer, we see there is also one and only one answer to this linear equation; meaning no other value of x can and will work. And we can see how 10 is, in fact, a combination of the factors of the equation. It is the result of four x's and the constant 2.

But what will happen to your ego if I give you the following statement:

$4x + 2y = 10$.

There is suddenly more to this problem because, in addition to the unknown $x$, we now have an unknown $y$. The issue we find when we try to solve this equation is that you really need to just pin down one variable to determine what the other would be: in other words, there is more than one solution. For example, if $x = 1$, then $y=3$. But if $x = 2$, then $y=1$. We say this is underdetermined - or we don't have enough evidence to determine a single answer.

So, like a good scientist, we need more evidence. We garner more evidence for the values of our variables by way of another expression to "pin down" the values. Let's add another, second, equation to consider along with our previous one -

$5x + 3y = 14$

What's a solution that works for both equations? That's the question now. Well, I gave it to you above as one of the possible options ($x = 1$, then $y=3$). But what if I hadn't given this to you, what would you do to find a single solution to both equations? You might figure you could sit down with a pen and paper and try many possible answers until you had something that worked. But this isn't scalable. What happens if we had five hundred variables, or a thousand, or ten thousand? In our world today we do have systems like this, and sitting down with a pencil and paper and simply trying answers won't work.

Finding a solution to this group - or system - of equations is a topic of linear algebra, which we'll continue to study.

Drosophyllum lusitanicum

I just purchased the last of the carnivorous plants I'm getting this year: a Dewy Pine.

A complicated little dude, it's soil and water and humidity requirements are vastly different than any other carnivore out there

...we'll see how I do.

....and we could have ourselves one nice little thunderstorm around here.

http://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-spring-2010/study-materials/

Damn this statement frustrates the hell out of me. Why?

Well, for one, it's totally misleading. Yes, while it's true Einstein didn't do well in school, he didn't not do well because he sucked at math (which is often the connotation carried with this statement). Instead, Einstein didn't do well in school because he didn't give a crap about it. Period. Understanding the laws of nature was more important to Einstein as a young man, not listening to some boring teacher drone on about things he, quite frankly, could figure out easily on his own. Need proof? Well, he was doing his best work in his spare time as a patent clerk, not in college. It wasn't until after his Theory of Relativity that he ever became recognized by any system of formal education. It wasn't formal education that made Einstein great, it was something else. Probably the fact he could do all of that math easily.

And second, because while I think many people believe they're using this statement as motivation (it's okay to fail, because Einstein failed too), I argue it's actually a deterrent to success and, in fact, often leads to either 1) laziness or 2) just giving up. It's much more destructive for a young mind to hear something like this than not; in fact, it might be destructive for a young mind to care at all how successful the greats (as defined by our society) were or are. Why?

You've heard of Einstein, right? You've heard of Newton, right? How about Mozart? How about Stephen Hawking? Okay. Sure. You have heard of them all.

So what about William James Sidis? ...No?

Interesting - because while Sidis' estimated IQ was around 250, Stephen Hawking's is only 160 - at about the same level as Einstein's. Richard Feynman's (a winner of the Nobel Prize) was recorded at 125 (though this is a bit misleading because the test was found to have focused on literary skills more than mathematical skills). Newton's was estimated to have sat at around 190. So by all definition Sidis should have written a symphony better than Mozart, or he should have created 24% more insight into the universe than Newton. But he didn't. Instead, his life was a rather miserable experience and he died penniless. A child prodigy who simply burned out. Aimless, directionless and depressed.

The fact of the matter is intelligence only matters up to a certain point. Beyond that point what matters more is something else: lots of fucking hard work. Dedicated, passionate work. Never-ending, totally absorbing work. And the simple truth of it is this work is tough work ... for anyone. Yes, anyone. Relativity was hard for Einstein. Understanding Gravitation was hard for Newton. Writing symphonies was hard work for Mozart. If it wasn't hard it wouldn't have consumed so much of their time. Why did it take Einstein ten more years to create the General Theory of Relativity after the Special? Because....it was hard.

You see, the reality is these people had a focused stream of energy, so much so that many of us would simply give up before exerting it, thinking (because it was so hard), "this isn't for me". And most of us do. Which is why we don't achieve things of this level. You see, my point isn't that we shouldn't encourage young minds, it's that we shouldn't encourage them by comparing them to the greatness of another person. Because it's simply not about that. Einstein wasn't trying to be Newton for he was too busy thinking about physics. We unintentionally refocus a person's whole motivational system to think about how someone else achieved success instead of just letting people do what they do naturally. Do you get my drift?

Another problem with this statement - as I've eluded to already - is it focuses too intensely on natural ability. It misses the very thing that made Einstein achieve those ends (his passionate work), and it simply leaves the listener in an odd state of comfort knowing that "hey, this great person didn't do well here either". But it takes the person nowhere beyond that point. It leaves them basking in an ill-conceived and transient feel-good moment. It burns a hole right through the point that should be made.

Why am I so confident in this? Because it got me. I remember walking to my Astronomy class when I was 14 (did I mention it was a college Astronomy course?) and realizing I was having trouble with certain elements of the math while - in my head - I imagined Newton hadn't. I remember walking to class imaging this mythical figure of Newton walking beside me, wondering how I could achieve that level of greatness. After receiving an A in the course (the highest score in the class), I never went back. "I couldn't do the math", I said. Bullshit. Had I put in the time I would have.

It just doesn't matter what Einstein could or couldn't do. Greatness is more complicated than an ability in grade school mathematics. All that matters is what drives you, what gives you passion, what wakes you up in the morning and keeps you focused throughout the day. These great people lived at the right time, and were around the right people, and had the right timing to submit their work. To busily compare yourself to them is inconsequential and distracting. Simply focus on your passion and the rest will work out.

I guess what I'm saying is this: I'm sure it was okay to Einstein that he failed math too because, honestly, he was too busy doing his math to care.