A brief survey of linear algebra

{\LARGE\bf A brief survey of linear algebra}\newline \newline \newline

# A brief survey of linear algebra

### June, 2001

Our general topics: Top

(ex): exercises.

Why linear algebra Top

• Linear models of phenomena are pervasive throughout science. The techniques of linear algebra provide tools which are applicable in a wide variety of contexts.

Beyond that, linear algebra courses are often the transition from lower division mathematics courses such as calculus, probability/statistics, and elementary differential equations, which typically focus on specific problem solving techniques, to the more theoretical axiomatic and proof oriented upper division mathematics courses.

I am going to stay with a generally abstract, axiomatic presentation of the basics of linear algebra. (But I'll also try to provide some practical advice along the way ... :-)

Vector spaces Top

• The first thing we need is a field F of coefficients for our vector space. The most frequently used fields are the real numbers R and the complex numbers C. A field F = (F, +, *, -,  -1, 0, 1) is a mathematical object consisting of a set of elements (F), together with two binary operations (+, *), two unary operations (-,  -1), and two distinguished elements 0 and 1 of F, which satisfy the fundamental properties:

1. F is closed under the four operations:
 +
 :
 Fx F® F
 *
 :
 Fx F® F
 -
 :
 F® F
 -1
 :
 F¢® F¢    F¢ = {a Î F| a ¹ 0}
Of course, we usually write a + b, a * b, -a, and a-1 instead of +(a,b), *(a, b), -(a), and  -1(a).

2. +' and *' are commutative and associative, and satisfy the distributive property. That is, for a, b, c Î F:
 a + b
 =
 b + a
 a * b
 =
 b * a
 (a + b) + c
 =
 a + (b + c)
 (a * b) * c
 =
 a * (b * c)
 a * (b + c)
 =
 a * b + a * c

3. 0 is the identity element and -' is the inverse for addition. 1 is the identity element and a-1' is the inverse for multiplication. That is, for a Î F:
 a + 0
 =
 a
 a + (-a)
 =
 0
 a * 1
 =
 a
 a * (a-1)
 =
 1     (a ¹ 0)

4. Although we won't need it for most of linear algebra, I'll mention that R and C are both complete (Cauchy sequences have limits), and R is fully ordered (a < b or b < a or a = b for all a, b Î R).

5. As needed, we will identify R as a subfield of C, and we will typically write elements of C as a + bi, where a and b are real and i2 = -1.

6. Although R and C are the most frequently used coefficient fields, there are many other fields such as Q, the rationals, and the finite fields Z/pZ for p a prime.

• If we leave out the requirement that a * b = b * a, we get what are called skew fields. An important example of a skew field is H, the quaternions (also called the hamiltonians) which contains additional square roots of -1, i2 = j2 = k2 = -1, and ij = k, jk = i, ki = j, but ij = -ji, jk = -kj, and ki = -ik. We typically write quaternions either in the form a + bi + cj + dk with a, b, c, d Î R, or
a+ bj with a, b Î C.

• We are then ready for the definition. A vector space V = (V, F, +, -, *, 0) over a field F is a set V of elements (called vectors) together with a distinguished element 0, two binary operations, and a unary operation:
 +
 :
 V x V ® V
 -
 :
 V ® V
 *
 :
 Fx V ® V

For u, v, w Î V, and a, b Î F, these operations satisfy the properties:

 v + w
 =
 w + v
 (u + v) + w
 =
 u + (v + w)
 v + 0
 =
 v
 v + (-v)
 =
 0
 1 * v
 =
 v
 a * (u + v)
 =
 (a * u) + (a * v)
 (a + b) * v
 =
 (a * v) + (b * v)
 (a * b) * v
 =
 a * (b * v)
The elements of the field F are called the scalars, or coefficients of the vector space.

• From the basic properties listed above, we can prove a variety of additional properties, such as:
 0 * v
 =
 0
 a * 0
 =
 0
 (-1) * v
 =
 -v
We can also prove that the additive identity 0 is unique, as is the additive inverse -v.

• We will usually simplify the notation, and write av instead of a * v. Furthermore, although it is important to distinguish the scalar 0 from the vector 0, we will typically write the vector in the simple form 0.

• If F = R, we call V a real vector space, and typically write the scalars as a, b, c. If F = C, we call V a complex vector space, and often write the scalars as a, b, g.

Exercises: Vector spaces Top

1. Using the basic properties listed above, prove the additional properties:

1. 0 * v = 0
2. a * 0 = 0
3. (-1) * v = -v
4. -(-v) = v
5. For a Î F and v Î V, av = 0 iff a = 0 or v = 0.

2. Prove that the additive identity 0 is unique.
3. Prove that the additive inverse -v is unique.

Examples of vector spaces Top

• The first main example of a real vector space is Rn, the Cartesian product of n copies of the real line. An element of Rn looks like (a1, a2, ¼, an). When we add two vectors, we get
 (a1, a2, ¼, an) + (b1, b2, ¼, bn)
 = (a1+b1, a2+b2, ¼, an + bn).

When we multiply by a scalar a Î R, we get

 a * (a1, a2, ¼, an) = (aa1, aa2, ¼, aan).

• Similary, we have the complex vector space Cn, with vector addition and scalar multiplication defined the same way.

• Note that we can also think of Cn as a real vector space, if we restrict the scalars to R.

• Let F¥ = {(a0, a1, ¼)}, with
(a0, a1, ¼) + (b0, b1, ¼)
= (a0 + b0, a1 + b1, ¼),
and a * (a0, a1, ¼) = (a*a0, a*a1, ¼).
This is a vector space over F.

• For a scalar field F, we can define F[x] to be the set of all polynomials with formal variable x, and coefficients in F. This is a vector space over F, the coefficient field. Each polynomial is a vector. Vector addition is just polynomial adddition, and scalar multiplication just multiplies each coefficient in the polynomial by the scalar:
 a * (a0 + a1x1 + ¼anxn)
 = aa0 + aa1x1 + ¼aanxn.

• For a scalar field F, let Fn[x] be the set of all polynomials of degree £ n with coefficients in F. We can also let F¥[x] be the vector space of (formal) power series over F. These are also vector spaces over F.

• Let C0(R) be the set of all continuous functions with domain and range the real numbers. That is:
C0(R) = {f:R® R | f is continuous}.
We can define an addition operation on this set. To specify the sum of two functions, we must specify the sum at each point in the domain. If f, g Î C0(R), we define f + g for each x by
 (f + g)(x) = f(x) + g(x).
We define scalar multiplication pointwise also: (a*f)(x) = a*(f(x)).

C0(R) thus becomes a real vector space, where each continuous function is a vector in the space.

• Let Cn(R) be the set of all continuous real functions whose first n derivatives are also continuous, and define addition and scalar multiplication pointwise. Similarly, we can define C¥(R) as functions with all derivatives continuous. Again we get real vector spaces.

• Let C0[0,1] be the set of all continuous functions with domain the closed interval [0,1], and range R. We can also define Cn[0,1] to be those functions with first n derivatives continuous, and C¥[0,1] with all derivatives continuous. We can also generalize to subsets of R other than the interval [0,1]. With pointwise addition and scalar multiplication, these are each real vector spaces.

• We get can get similar complex vector spaces if we use C instead of R in examples like those above.

Exercises: Examples of vector spaces Top

1. Show that each of the examples listed in this section is a vector space.

2. Consider the set of points in R2 of the forms (x, 0), x Î R, and (0, y), y Î R (i.e., the union of the X-axis and the Y-axis). Show that with the usual vector addition in R2, this is not a vector space over R.

3. Consider the set R2 with usual vector addition:
 (x1, y1) + (x2, y2) = (x1 + x2, y1 + y2),
but with scalar multiplication'' given by
 a * (x, y) = (ax, y)
for a Î R. Show that this is not a vector space over R.

Subspaces Top

• If V is a vector space over F, a subset U Ì V is called a vector subspace of V if U is a vector space over F using the same vector addition and scalar multiplication as in V. Often, in context, we will just call U a subspace of V.

• Most often, given a subset U Ì V, we will just let U inherit the vector addition and scalar multiplication operations from V. Hence, most of the vector space properties will come for free. What may not come for free, however, is whether the subset U is closed under the operations.

Thus, when we inherit the operations from V, we will have that

 + : U x U ® V,
and
 * : Fx U ® V,
whereas, what we need is
 + : U x U ® U,
and
 * : Fx U ® U.
In order for U to be a vector subspace, we must be sure that doing the operations will leave us in the subset U.

• Examples of subspaces:

1. {0} is a subspace of V for any vector space V.
2. If m £ n, then Fm (identified with {(a1, a2, ¼, am, 0, ¼, 0)}) is a subspace of Fn.
3. If m £ n, then Fm[x] is a subspace of Fn[x], and a subspace of F[x].
4. If m £ n, then Cn(R) and C¥(R) are subspaces of Cm(R) .
5. In R2, let U = {(x, y) | y = 3x}. Then U is a subspace.
6. More generally, in Rn, fix a1, a2, ¼, an, and let U = {(x1, x2, ¼, xn|      a1x1 + a2x2 + ¼+ anxn = 0}.
Then U is a subspace.
7. If U1 and U2 are subspaces, so is U1 ÇU2.

• Examples which are not subspaces:

1. In R2, let U = {(x, y) | x = 0 or y = 0} (i.e., U is the union of the x-axis and the y-axis). Then U is not a subspace, since the two vectors (1,0) and (0,1) are in U, but (1,0) + (0,1) = (1,1) is not in U.
2. In R2, let U = {(x, y) | x and y are both rational numbers}. Then U is not a subspace, since (1, 1) is an element of U, but Ö2*(1, 1) = (Ö2, Ö2) is not an element of U.
3. In Rn, fix a1, a2, ¼, an, and let U = {(x1, x2, ¼, xn| a1x1 + a2x2 + ¼+ anxn = 2}. Then U is not a subspace, since, for example, 0 is not an element of U.
4. In general, if U1 and U2 are subspaces, U1 ÈU2 is not a subspace (except in special cases, such as when U1 Ì U2).

• If U1 and U2 are both subspaces of a vector space V, we can define the subspace which is the sum of the two subspaces by:
 U1 + U2 = {u1 + u2 | u1 Î U1 and u2 Î U2}.

• If in addition U1 ÇU2 = {0}, then each element u Î U1 + U2 can be written in a unique way as u = u1 + u2. In this case, we call the sum of U1 and U2 a direct sum, and write it U1 ÅU2.

• These ideas generalize in a straightforward way to U1+¼+ Un and U1Å¼ÅUn for a finite number of subspaces, and U1+U2+¼  and U1ÅU2Å¼ for countably many subspaces.

Exercises: Subspaces Top

1. Show that each of the examples identified in this section as a subspace actually is a vector subspace.

2. Which of the following are vector subspaces of R3?

1. {(x1, x2, x3) Î R3 | 3x1 + 2x2 - x3 = 0}
2. {(x1, x2, x3) Î R3 | 3x1 + 2x2 - x3 = 4}
3. {(x1, x2, x3) Î R3 | x1x2x3 = 0}

3. Suppose that U is a vector subspace of V, and V is a vector subspace of W. Show that U is a vector subspace of W.

4. Show that the intersection of any collection of vector subspaces of V is a vector subspace of V.

5. Let
 l2 = {(ai) Î R¥ | ¥ å i = 0 |ai|2 < ¥}.
Show that l2 is a vector subspace of R¥.

6. Let
 L2 = {f Î C0(R) | óõ ¥ -¥ |f(x)|2 dx < ¥}.
Show that L2 is a vector subspace of C0(R).

Linear dependence and independence Top

From here on, we'll assume that U and V are vector spaces over a field F.

• Suppose that v1, v2, ¼, vn are vectors in V, and that for some a2, a3, ¼, an Î F,
 v1 = a2v2 + a3v3 + ¼+ anvn.
Then we say that v1 is linearly dependent on v2, ¼, vn. Note that if we move v1 to the other side, and let a1 = -1, we have a1v1 + a2v2 + ¼+ anvn = 0, and not all of the ai are zero (in particular, a1 ¹ 0).

This motivates a general definition: a set S of vectors in a vector space V is called linearly dependent if, for some n > 0, and distinct v1, v2, ¼vn Î S, there exist a1, a2, ¼, an Î F, not all 0, with

 a1v1 + a2v2 + ¼+ anvn = 0.

• The flip side of this definition is the following: a set S of vectors in a vector space V is called linearly independent if, given distinct v1, v2, ¼vn Î S, the only way for a1, a2, ¼, an Î F to give
 a1v1 + a2v2 + ¼+ anvn = 0
is if every one of the ai are zero.

A useful way to think about this is that a set S of vectors is linearly independent if no individual one of the vectors is linearly dependent on a finite number of the rest of them.

• Some examples:

1. In R2, the sets of vectors {(1,0),(0,1)}, {(1,2),(1,3)}, {(1,0)} and {(2,2),(3,2)} are each linearly independent sets.
2. In R2, none of the sets of vectors {(1,0),(2,0)}, {(1,0),(0,1), (1,1)}, {(0,0),(0,1)}, {(0,0)}, {(1,2),(2,3),(3,4),(4,1)}, nor {(1,2),(3,4),(2,1)} is a linearly independent set.
3. In any vector space, if a set of vectors contains the 0 vector, the set is not linearly independent.
4. In C0(R), if f1(1) = f2(2) = ¼ = fn(n) = 1, but fi(j) = 0 for i ¹ j, then the set of functions {f1, f2, ¼, fn} is linearly independent.
5. In C¥(R), the set of functions {cos(nx) | n = 0, 1, 2, ¼} is a linearly independent set.
6. The empty set, {}, is a linearly independent set.

• Mathematics is an extremely precise language. These two sentences do not mean the same thing:

1. The set {v1, v2, ¼, vn} of vectors is linearly independent.
2. {v1, v2, ¼, vn} is a set of linearly independent vectors.

One of the hardest parts of doing mathematics is developing your mathematical intuition. It is tempting to imagine that intuition is what you have before you know anything, but that is nonsense. Intuition is just the automatic part of your knowledge, derived from your past experience. Becoming better at mathematics involves learning new mathematics, and then integrating that new knowledge into your intuition. Doing that takes care, precision, and lots of practice!

Exercises: Linear dependence and independence Top

1. Verify the statements in each of the six examples in this section.

2. Suppose that P0, P1, ¼, Pn are polynomials in Fn[x], and Pi(1) = 0 for i = 0, 1, ¼,n. Show that {P0 , P1, ¼, Pn} is not linearly independent in Fn[x].

3. In C¥(R), show that each of these sets of functions is linearly independent:

1. {xn | n = 0, 1, 2, ¼}.

2. {sin(nx) | n = 0, 1, 2, ¼}.

3. {enx | n = 0, 1, 2, ¼}.

Span of a set of vectors Top

• If S is a set of vectors in the vector space V over F, we define the span of the set S by:
 span(S) = {a1v1 + a2v2 + ¼+ anvn |
 ai Î F, vi Î S, n > 0}.
This says that span(S) is the set of all vectors that can be written as finite linear combinations of the vectors in S.

• Examples:

1. In R1, span({(1)}) is all of R1.

2. In R3, span({(1,0,0), (0,1,0)}) is the x-y plane. Similarly, span({(1,0,0), (0,0,1)}) is the x-z plane, span({(0,1,0), (0,0,1)}) is the y-z plane, and span({(1,0,0), (0,1,0), (0,0,1)}) is all of R3.

3. Still in R3, span({(1,2,0), (2,1,0)}) is the x-y plane, span({(1,0,0)}) is the x axis, span({(1,0,0), (1,1,0), (1,1,1)}) is all of R3, span({(1,2,3), (1,1,0), (1,1,1), (2,1,4)}) is all of R3, and span({(1,2,0), (1,1,0), (3,1,0), (2,1,0)}) is the x-y plane.

4. In Fn[x], span({1, x1, x2, ¼, xn}) = Fn[x].

5. In F[x], span({1, x1, x2, ¼}) = F[x].

6. In C0(R), span({f1(x) = 1, f2(x) = x}) is the set of all linear functions, y = mx + b.

7. For any set S, span(S) is a subspace.

8. In particular, span(S) is sometimes defined to be the smallest subspace of V containing S, or the intersection of all subspaces of V that contain S.

Exercises: Span of a set of vectors Top

1. Verify the statements in each of the eight examples in this section.

2. Show that if span({v0, v1, ¼, vn}) = V, then span({v0 - v1, v1 - v2, ¼,vn-1 - vn, vn}) = V.

Basis for a vector space Top

• Suppose S is an ordered set (or list) of vectors in a vector space V. Suppose further that:

1. S is linearly independent, and
2. span(S) = V.

Then we call S a basis for the vector space V.

• Given a basis S for the vector space V, every vector v Î V, v ¹ 0, can be written in a unique way as:
 v = a1v1 + a2v2 + ¼+ anvn
with a1, a2, ¼, an Î F, ai ¹ 0, and v1, v2, ¼, vn Î S. For uniqueness, we take the vi to be distinct, and to be in the order they appear in S. We are writing each v as a finite linear combination of the basis vectors.

• Examples:

1. In F3, the ordered set S = ((1,0,0),(0,1,0),(0,0,1)) is a basis (often called the standard basis). This generalizes in the obvious way to Fn.

2. In F3, the ordered set S = ((1,0,0),(1,1,0),(1,1,1)) is a basis. So is ((1,2,3),(2,1,3),(1,2,2)).

3. In general, in Fn any linearly independent ordered set S = (v1, v2, ¼, vn) of size n is a basis.

4. (1, x, x2, x3, ¼) is a basis for F[x].

5. If S is a linearly independent ordered set, then S is a basis for span(S).

• If a vector space V has a finite basis S = (v1, v2, ¼, vn), we say that V is finite dimensional, and we define the dimension of V by:
 dim(V) = n.

We define dim({0}) = 0.

Note that if a vector space V has a finite basis of size n, then every basis for V contains n vectors, and thus the definition makes sense.

For example, dim(Fn) = n for any field F and n > 0.

We also have that dim(Fn[x]) = n + 1.

• If a vector space V has a finite or countably infinite basis S, then we can uniquely represent each vector v with respect to S by a list of the form:
v = (b1, b2, ¼, bn) in the finite case, or
v = (b1, b2, ¼) in the infinite case.

Each bi is either the non-zero coefficient corresponding with the ith element of S from the unique representation described above, or 0 if the basis element does not appear there. In the infinite case, only finitely many of the bi are non-zero.

We represent 0 by (0, 0, ¼, 0) in the finite case, and by (0, 0, ¼) in the infinite case.

• In these cases, we also have, for vi Î S, that
 V = span(v1)Å¼Åspan(vn)
in the finite case, and
 V = span(v1)Åspan(v2)Å¼
in the countably infinite case.

Exercises: Basis for a vector space Top

1. Verify the statements in each of the five examples in this section.

2. Let U be the subspace of R6 given by U = {(x1, x2, x3, x4, x5, x6) Î R6 |  2x1 + 3x3 = 0 and x2 = 4x5}.

Find a basis for U.

3. Show that if U1 and U2 are subspaces of a finite dimensional vector space, then dim(U1 + U2) = dim(U1) + dim(U2) - dim(U1 ÇU2).

Linear transformations Top

• If U and V are vector spaces over F, then a function T : U ® V is called a linear transformation or linear mapping if
 T(a1u1 + a2u2) = a1T(u1) + a2T(u2)
for all a1, a2 Î F and u1, u2 Î U.

An equivalent pair of conditions is that T(u1 + u2) = T(u1) + T(u2), and T(au) = aT(u).

• For a linear transformation T : U ® V, we call U the domain and V the codomain (or sometimes range) of T. We also define the kernel (or null space) of T by:
 ker(T) = {u Î U | T(u) = 0}.
Further, we define the image (or sometimes range - be careful here!) of T by:
 im(T) = {v Î V | v = T(u) for some u Î U}.

• For a linear transformation T : U ® V we have the nice properties:

1. ker(T) is a subspace of U, and im(T) is a subspace of V. (ex)
2. If U is finite dimensional, then
 dim(U) = dim(ker(T)) + dim(im(T)).
(ex)
3. If U is finite dimensional with basis (u1, u2, ¼, un), and V is finite dimensional with basis (v1, v2, ¼, vm), then T is determined by its effect on the basis elements ui. There exist aij, 1 £ i £ m, 1 £ j £ n with:
 T(uj) = a1jv1 + a2jv2 + ¼+ amjvm,
and in general, if u = åjbjuj, then
 T(u) = m å i = 1 n å j = 1 aijbjvi.

4. Thus, given particular bases for U and V, we can represent the linear transformation T by the matrix
[T] = é
ê
ê
ê
ê
ë
 a11
 a12
 ¼
 a1n
 a21
 a22
 ¼
 a2n
 :
 :
 :
 :
 am1
 am2
 ¼
 amn
ù
ú
ú
ú
ú
û
or [T]ij = aij.
5. If we represent the vector u Î U by the column matrix [b1, b2, ¼, bn]t, then we have
 [T(u)]
 =
 [T][u]
 =
é
ê
ê
ê
ê
ë
 a11
 a12
 ¼
 a1n
 a21
 a22
 ¼
 a2n
 :
 :
 :
 :
 am1
 am2
 ¼
 amn
ù
ú
ú
ú
ú
û
é
ê
ê
ê
ê
ë
 b1
 b2
 :
 bn
ù
ú
ú
ú
ú
û
 =
é
ê
ê
ê
ê
ë
 a11b1 + a12b2 + ¼+ a1nbn
 a21b1 + a22b2 + ¼+ a2nbn
 :
 am1b1 + am2b2 + ¼+ amnbn
ù
ú
ú
ú
ú
û

6. Sometimes it is more convenient to use the Einstein summation conventions, where we would write:
 T(u)i = Tijuj
with implied summation over the repeated upper and lower indices.

7. If T1 : U ® V and T2 : U ® V are two linear transformations, we can define the sum of the two transformations as
 (T1 + T2)(u) = T1(u) + T2(u),
and we can define scalar multiplication of a linear transformation T by a as
 (a * T)(u) = a * (T(u)).
We can thus define L(U, V) to be the space of all linear transformations from U to V. We define the zero transformation 0 : U ® V by 0(u) = 0. With these definitions, L(U, V) is a vector space. (ex)

8. Given particular bases for U and V, the matrix representation of T1 + T2 is given by: [T1 + T2] = [T1] + [T2].

9. If U and V are finite dimensional, then L(U, V) is also finite dimensional, and
 dim(L(U, V)) = dim(U)*dim(V).
(ex)

10. If T : U ® V and S : V ® W are linear transformations, then the composition S °T : U ® W is also a linear transformation. (ex)

Recall that (S °T)(u) = S(T(u)).

Given particular bases for U, V, and W, the matrix representation of S °T is the matrix product [S °T] = [S][T]. We usually abbreviate S °T as ST.

It is worth noting that unless W Í U, it doesn't even make sense to talk about T °S.

11. The matrix of the composition of two linear transformations is given by the product of the two matrices, given by:
 [ST] = [S][T] =
é
ê
ê
ê
ê
ë
 a11
 a12
 ¼
 a1n
 a21
 a22
 ¼
 a2n
 :
 :
 :
 :
 am1
 am2
 ¼
 amn
ù
ú
ú
ú
ú
û
é
ê
ê
ê
ê
ë
 b11
 b12
 ¼
 b1p
 b21
 b22
 ¼
 b2p
 :
 :
 :
 :
 bn1
 bn2
 ¼
 bnp
ù
ú
ú
ú
ú
û
 =
é
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ë
 n å i = 1 a1ibi1
 n å i = 1 a1ibi2
 ¼
 n å i = 1 a1ibip
 n å i = 1 a2ibi1
 n å i = 1 a2ibi2
 ¼
 n å i = 1 a2ibip
 :
 :
 :
 :
 n å i = 1 amibi1
 n å i = 1 amibi2
 ¼
 n å i = 1 amibip
ù
ú
ú
ú
ú
ú
ú
ú
ú
ú
ú
û

12. We also have the distributive property:
 S(T1 + T2) = (ST1) + (ST2).
The matrix representation for this is:
 [S][T1 + T2] = [S][T1] + [S][T2].

• Some examples of linear transformations:

1. The function T : R2 ® R given by T((x, y)) = x + y is linear.

2. The function T : R2 ® R3 given by T((x, y)) = (x + 2y, x - 2y, x - y) is linear.

3. Given a1, a2, ¼, an Î F, and V a finite dimensional vector space over F with basis (v1, v2, ¼, vn), the function T : V ® F given by T(b1v1 + ¼+ bnvn) = a1b1 + ¼+ anbn is linear.

In general, for a vector space V, a linear transformation T : V ® F is called a linear functional. The study of these transformations is called functional analysis.

4. The function D : Fn[x] ® Fn-1[x] given by
D(anxn + ¼+ a1x + a0)
= nanxn-1 + ¼+ 2a2x + a1
is linear. This linear transformation is called the derivative ...

5. Similarly, we have the derivative transformation D: F¥[x] ® F¥[x] given by
 D( ¥ å n = 0 anxn) = ¥ å n = 1 n*anxn-1.

6. The shift map S : F¥ ® F¥ is given by: S((a0, a1, ¼)) = (a1, a2, ¼).

7. The difference map D: F¥ ® F¥ is given by:
 D((a0, a1, ¼)) = (a1 - a0, a2 - a1, ¼),
or, abbreviating (a0, a1, ¼) by (an),
 D((an)) = (an+1 - an).

Exercises: Linear transformations Top

1. Verify each of the statements marked (ex) in this section.

2. Verify that each of the 7 examples actually are linear transformations.

3. Show that the usual calculus derivative [d / dx] : C¥(R) ® C¥(R) given by:
 d f(x) dx = lim h® 0 f(x+h) - f(x) h
is a linear transformation.

4. Is the usual calculus indefinite integral
 óõ f(x)dx
a linear transformation? Why or why not? What about the definite integral?

Morphisms - mono, epi, and iso Top

• In algebra, we call a function that preserves structure a morphism. In our current context, a linear transformation preserves the linear structure of a vector space in the sense that T(u1 + u2) = T(u1) + T(u2) and T(au) = aT(u). Thus, linear transformations are morphisms in the category of vector spaces.

• We will be particularly interested in morphisms that have additional properties. Specifically, we are likely to look for morphisms T : U ® V which have one or more of the properties:

1. One-to-one. Such transformations are also called injective, or monomorphisms. These transformations have the property that if u1 ¹ u2, then T(u1) ¹ T(u2). This says that different things get sent different places - that is, no two things get sent to the same place.

Monomorphisms are nice because the subspace im(T) Ì V looks just like U.

2. Onto. Such transformations are also called surjective, or epimorphisms. These transformations have the property that for every v Î V, there exists some u Î U with T(u) = v. This says that every element of V is hit by some element of U. Another way to say this is that im(T) = V.

Epimorphisms are nice, because the algebraic properties of V will be reflected back in U.

3. Both one-to-one and onto. Such transformations are also called bijective. A bijective function f : X ® Y has an inverse f-1 : Y ® X with the properties that for all x Î X and all y Î Y, f-1(f(x)) = x and f(f-1(y)) = y.

A bijective morphism whose inverse also preserves algebraic structure is called an isomorphism. In linear algebra, we have the nice property that if a linear transformation is bijective, then its inverse is also linear, and thus it is an isomorphism.

4. If there is an isomorphism T : U ® V, we say that the spaces U and V are isomorphic. If two spaces are isomorphic, then they share all relevant properties. Thus, two isomorphic vector spaces are indistinguishable as vector spaces except for a renaming of the elements.

• We have the nice fact that a linear transformation T : U ® V is a monomorphism (is one-to-one) if and only if ker(T) = {0}.

Pf.: Suppose T is a monomorphism. We know that for every linear transformation, T(0) = 0. Then, since T is a monomorphism, we know that if T(u) = 0 = T(0), it must be that u = 0. Thus ker(T) = {0}.

On the other hand, suppose that ker(T) = {0}. Then, if T(u1) = T(u2), we will have 0 = T(u1) - T(u2) = T(u1 - u2). But this means that u1 - u2 Î ker(T) and hence, since we are assuming that ker(T) = {0}, we must have u1 - u2 = 0, or u1 = u2. By the contrapositive, this means that if u1 ¹ u2, then T(u1) ¹ T(u2). Q.E.D.

(I had to do at least one proof,
didn't I? :-)

• An important example of an isomorphism is the identity transformation IU : U ® U given by IU(u) = u. (ex)

• If T : U ® V is a monomorphism, then there is a linear transformation S1 : im(T) ® U with S1(T(u)) = (S1T)(u) = u for all u Î U. This says that S1T = IU. S1 is called a left (partial) inverse of T. (ex)

• If T : U ® V is an epimorphism, then there is a linear transformation S2 : V ® U with T(S2(v)) = (TS2)(v) = v for all v Î V. This says that TS2 = IV. S2 is called a right (partial) inverse of T. (ex)

• If T : U ® V is an isomorphism, then since T is an epimorphism, both S1 and S2 (as above) exist. Also, im(T) = V. We therefore have that for all v Î V,
 S1(v) = S1((TS2)(v)) = (S1T)(S2(v)) = S2(v).
Thus S1 = S2. In this case, the (two-sided) inverse of T exists, and we have T-1 = S1 = S2.

• If U and V are finite dimensional vector spaces over F with dim(U) = dim(V), then U and V are isomorphic. (ex)

(Big) hint for proof: Let (u1, u2, ¼, un) and (v1, v2, ¼, vn) be bases for U and V respectively. Define T : U ® V by T(ui) = vi for 1 £ i £ n, and extend by linearity. Make sense of the phrase extend by linearity,'' and then show that T is an isomorphism.

• We also have the nice fact that if dim(U) = dim(V) = n, U, V over F, and T : U ® V is linear, then the following are equivalent: (ex)

1. T is a monomorphism.
2. T is an epimorphism.
3. T is an isomorphism.

This means, for example, that such a T is onto if and only ker(T) = 0.

• If T : U ® V and S : V ® W, then (ex)

1. If both S and T are monomorphisms, then so is ST.
2. If ST is a monomorphism, then so is T.
3. If both S and T are epimorphisms, then so is ST.
4. If ST is an epimorphism, then so is S.

• If T : V ® V and S : V ® V, and ST = TS, then (ex)

1. Both S and T are monomorphisms if and only if ST is a monomorphism.
2. Both S and T are epimorphisms if and only if ST is an epimorphism.

• Let matrix(F, n, m) be the space of all n by m matrices with entries from F. We use ordinary entry by entry matrix addition, where (A + B)ij = Aij + Bij, scalar multiplication, where (aA)ij = aAij, and let (0)ij = 0. Then matrix(F, n, m) is an n*m-dimensional vector space over F. If dim(U) = m and dim(V) = n, then we can define
 mat : L(U, V) ® matrix(F, n, m)
by mat(T) = [T].

This transformation is an isomorphism. (ex)

Exercises: Morphisms - mono, epi, and iso Top

1. Verify each of the statements marked (ex) in this section.

2. Show that if a function has an inverse, then the inverse is unique.

3. Consider the function T : Fn[x] ® Fn+1 given by
 T(a0 + a1x + ¼+ anxn) = (a0, ¼, an).
Show that this function is an isomorphism.

4. Consider the function T : C® matrix(R, 2, 2) given by
 T(a + bi)
 =
é
ê
ë
 a
 -b
 b
 a
ù
ú
û

1. Show that if we consider C as a real vector space, this function is a monomorphism.
2. Show that this function also respects complex multiplication, that is,
 T((a + bi)(c + di)) = T(a + bi) T(c + di).

5. More generally, consider the function T : matrix(C, n, n) ® matrix(R, 2n, 2n) given by
T æ
ç
ç
ç
è
é
ê
ê
ê
ë
 a11 + b11i
 ¼
 a1n + b1ni
 :
 :
 an1 + bn1i
 ¼
 ann + bnni
ù
ú
ú
ú
û
ö
÷
÷
÷
ø

=   é
ê
ê
ê
ê
ê
ê
ê
ê
ë
 a11
 -b11
 ¼
 a1n
 -b1n
 b11
 a11
 ¼
 b1n
 a1n
 :
 :
 :
 :
 an1
 -bn1
 ¼
 ann
 -bnn
 bn1
 an1
 ¼
 bnn
 ann
ù
ú
ú
ú
ú
ú
ú
ú
ú
û

1. Show that if we consider matrix(C, n, n) as a real vector space, this function is a monomorphism.
2. Show that this function also respects matrix multiplication, that is,
 T(A B) = T(A) T(B).

6. What the heck. Let H be the quaternions (described above). Consider the function T : matrix(H, n, n) ® matrix(C, 2n, 2n) given by
T æ
ç
ç
ç
è
é
ê
ê
ê
ë
 a11 + b11j
 ¼
 a1n + b1nj
 :
 :
 an1 + bn1j
 ¼
 ann + bnnj
ù
ú
ú
ú
û
ö
÷
÷
÷
ø

=   é
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ë
 a11
 - b11
 ¼
 a1n
 - b1n
 b11
 a11
 ¼
 b1n
 a1n
 :
 :
 :
 :
 an1
 - bn1
 ¼
 ann
 - bnn
 bn1
 an1
 ¼
 bnn
 ann
ù
ú
ú
ú
ú
ú
ú
ú
ú
ú
ú
ú
ú
û

1. Show that if we consider matrix(H, n, n) as a complex vector space, this function is a monomorphism.
2. Show that this function also respects matrix multiplication, that is,
 T(A B) = T(A) T(B).

Thus, if we denote by O(n), U(n), and Sp(n) the distance preserving linear operators on Rn, Cn, and Hn respectively (called the orthogonal, unitary, and symplectic groups), then we have the monomorphisms:

 ¼® O(n) ® Sp(n) ® U(2n) ® O(4n)
 ® Sp(4n) ® U(8n) ® O(16n) ® ¼

Linear operators Top

• If T : V ® V is a linear transformation, we call T a linear operator on V. Note that if S and T are operators on V, then so is ST. We can abbreviate L(V, V) as L(V).

L(V) has the algebraic structure of a ring with identity. A ring is similar to a field (as defined above), except without the requirements that multiplication be commutative and that there be multiplicative inverses for non-zero elements. The identity element is IV. L(V) is a non-commutative ring, since in general ST ¹ TS. This is reflected in the fact that matrix multiplication is non-commutative. Only in very special cases is it true that [S][T] = [T][S] (for example, if both [S] and [T] are diagonal matrices, with [S]ij = 0 for i ¹ j, and [T]ij = 0 for i ¹ j).

• Note that for operators, it makes sense to talk about T °T, and we can thus define Tn = T °T °¼°T (n times). We also define T0 = I, the identity operator.

• Thus, if we have a polynomial P(x) = anxn + an-1xn-1 + ¼ + a1x + a0 from F[x], we can talk about the polynomial in T:
 P(T) = anTn + an-1Tn-1 + ¼+ a1T + a0.
(The a0 term stands for a0I.) This is an operator on V which acts on vectors as:
 P(T)(v) = anTn (v) + ¼+ a1T(v) + a0v.

• In fact, we can generalize this to power series. If
 p(x) = ¥ å n = 0 anxn,
then we can define
 p(T) = ¥ å n = 0 anTn.

• This means, for example, that we can define the exponential of an operator:
 exp(T) = ¥ å n = 0 1 n! Tn.

• We can even define the cosine or sine of an operator, etc.:
 cos(T) = ¥ å n = 0 (-1)n (2n)! T2n,
 sin(T) = ¥ å n = 0 (-1)n (2n+1)! T2n+1.

• We can talk about square roots of an operator, saying that S is a square root of T if S2 = T. It is unlikely that S is unique.

• We can also talk about logarithms of an operator, saying that S is a logarithm of T if exp(S) = T. Again, it is unlikely that S is unique.

• Examples of linear operators:

1. A first important example begins like this: suppose we have a system of m linear equations in n variables
 a11x1 + a12x2 + ¼+ a1nxn
 =
 b1
 a21x1 + a22x2 + ¼+ a2nxn
 =
 21
 :
 :
 am1x1 + am2x2 + ¼+ amnxn
 =
 bm.

If we let A : Fn ® Fm be the linear transformation with [A]ij = aij, x be the vector (x1, x2, ¼, xn)t, and b the vector (b1, b2, ¼, bm)t, then we can rewrite the equation in the form:

 Ax = b.
We then have several possibilities.

1. The first case is when m = n. There are then two possibilities:

1. If A is an isomorphism, then A-1 exists, and we can solve the equation for x:
 x = A-1Ax = A-1b.

2. If A is not an isomorphism, then in particular A is not a an epimorphism, and thus it is possible that b Ï im(A). In this case, there are no solutions to the equation.

On the other hand, if b Î im(A) there is at least one solution x0 with Ax0 = b.

Note, though, that we also know A is not a monomorphism, and hence dim(ker(A)) ³ 1. Then, if z Î ker(A), we have A(x0 + z) = Ax0 + Az = Ax0 + 0 = Ax0 = b, and so x0 + z is another solution. Furthermore, if y is another solution with Ay = b, then A(x0 - y) = Ax0 -Ay = b - b = 0. This means x0 - y Î ker(A), and so y = x0 + z for some z Î ker(A).

Thus, if x0 is a particular solution, then every solution is of the form x0 + z for some z Î ker(A). The space of solutions is then a translation of the kernel of A, of the form x0 + ker(A). We then only need to find one particular solution.

In this case, we have broken the problem down into two parts: first, we solve Ax = 0 (called the homogeneous equation), then we find a single solution Ax0 = b. For F = R or C, there will be infinitely many solutions.

2. The second case is when m < n. In this case, A cannot be a monomorphism, and things are then similar to the second part of the previous case. If A is an epimorphism, there is sure to be at least one solution. Again, we solve the homogeneous case, and then find one particular solution. If A is not an epimorphism, there may not be any solutions. Otherwise, as above, in general we will have infinitely many solutions.

3. If m > n, then A cannot be an epimorphism, and hence there may not be any solutions. If A is a monomorphism, if there is a solution, there will be just one. If A is not a monomorphism, if there is one solution, there will be infinitely many, as above.

4. In all three cases, we can start by trying to find the inverse (or left partial inverse) of A. If A is a monomorphism, we can expect to be successful, and to be able to find the unique solution if it exists. If A is not a monomorphism, we solve the homogeneous equation, and then look for a single particular solution.

5. Question: How can we tell if A is a monomorphism? How can we find A-1, or at least S1, the left partial inverse of A?

2. A second important example is the derivative operator
 D: C¥(R) ® C¥(R).
Note that this is a linear operator, since D(f + g) = D(f) + D(g), and D(af) = aD(f).

Note that ker(D) = span({1}), the one-dimensional space consisting of all constant functions. If we collapse ker(D) down to nothing (in technical terms, form the quotient space ...) then we can think of D as an isomorphism (on the quotient space). D has an inverse,

 óõ : C¥(R) ® C¥(R).
We then have (a variant of) the fundamental theorem of calculus:
 D óõ (f) = óõ D(f) = f
(on the reduced space - that is, up to a constant).

3. We can then consider the general linear ordinary differential operators with constant coefficients. These are operators in R[D], that is operators of the form:
 P(D) = anDn + ¼+ a1D+ a0.
These operators act on a function f Î C¥(R) as
 P(D)(f) = anDn(f) + ¼+ a1D(f) + a0f.
We then have the general nth order linear ordinary differential equation with constant coefficients:
 P(D)(f) = g.
We can work on solving these equations with an approach similar to the method for systems of linear equations.

We first note that if the function f0 is a solution to this equation, and z = z(x) Î ker(P(D)), then f0 + z is also a solution, and if f1 is another solution, then P(D)(f0 - f1) = P(D)(f0) - P(D)(f1) = g - g = 0. Thus, all solutions are of the form f0 + z where f0 is some particular solution, and z Î ker(P(D)).

We thus separate the problem into two parts. First we solve the associated homogeneous equation:

 P(D)(f) = 0
to find ker(P(D)), and then we look for a single particular solution to the original equation.

In general, we have that dim(ker(P(D))) = n, the degree of P. This is not an entirely obvious fact, but it is not counterintuitive ...

Hence what we need to do is find n functions which form a basis for ker(P(D)). What we need, then, are n linearly independent functions each of which is a solution to the homogeneous equation.

In theory (:-) this is not too hard.

We note first that for the first-order case, we have the solution:

 (D- r)(erx)
 =
 D(erx) - rerx
 =
 rerx - rerx
 =
 0,
and, in the kth order extension of this,
 (D- r)k(xk-1erx) = 0.

We also have that the set of functions A = {xjerix | 0 £ j £ k, 1 £ i £ m}
is linearly independent. From this we see how to solve equations of the form:

 Õ i (D- ri)ki(f) = 0.

Now, consider the operator

 D2 - 2sD+ s2 + t2
for s, t Î R.

We have that

 (D2 -2sD+ s2 + t2)k(xk-1esxcos(tx)) = 0,
 (D2 -2sD+ s2 + t2)k(xk-1esxsin(tx)) = 0.
We can think of this as a = s + ti, and that we are working with
 D2 - 2sD+ s2 + t2 = (D- a)(D- a ).

We have that the set of functions

 B = {xjeskxcos(tkx), xmesnxsin(tnx)}
is also linearly independent. From this we see how to solve equations of the form:
 Õ i (D2 -2siD+ si2 + ti2)ki(f) = 0.

We can now put together all the pieces to solve the homogeneous equation P(D) = 0. We use the fact that any polynomial over R can be completely factored as

 P(D) = Õ i (D- ri)ki Õ j (D-aj)kj (D- a j )kj
with ri Î R and aj Î C.

To solve the inhomogeneous equation, we need only to find one particular solution of P(D)(f) = g.

This is just the bare beginnings of techniques for solving differential equations, but it gives the flavor of some relatively powerful methods, and the role that linear algebra plays. I haven't even mentioned the issues of initial values/boundary conditions. For much more on these topics, look at a book such as Elementary Differential Equations with Linear Algebra, by Finney and Ostberg.

4. These approaches generalize to linear ordinary differential operators with non-constant coefficients:
 an(x)Dn + an-1(x)Dn-1 + ¼+ a0(x),
to systems of linear ordinary differential operators, and on to linear partial differential operators and systems of linear partial differential operators. But I think for now someone else will have to write that up ... :-)

5. We can develop a similar approach to solving linear discrete difference equations. For example, the difference equation (an+2 - an+1 - an) = (0) has as a solution the Fibonnacci sequence (an) = (1, 1, 2, 3, 5, 8, ¼). We could work with the discrete version D of the differential operator D, where Df(x) = (f(x + 1) - f(x))/1 = f(x + 1) - f(x). In the discrete case, we can't let h go to zero.

We would then have D((an)) = (an+1 - an). Our example difference equation would be

 (D2 + D-1)((an)) = (0).
Often, however, it is more convenient to work with the discrete increment operator E, with E(f(x)) = f(x + 1), or E((an)) = (an+1). Both D and E are linear operators. Our example equation is then E2((an)) = E((an)) + (an), or
 (E2 - E - 1)((an)) = (0).
In a general form, we have a polynomial P(x) for an equation of the form P(E)((an)) = (0). We can then find a general solution by using the facts that
 (E - a)k((njan)) = (0)
for 0 £ j < k, and
 {(njan) | 0 £ j < k}
is a linearly independent set, etc.

6. Of course, all of these are the easy cases. The hard ones are the nonlinear equations ...

7. At times, this reminds me of a comment made by my Ph.D. thesis advisor, after I had finished the proof of my main result for all odd primes. He said there were three ways to think about it:

1. I had handled infinitely many cases, and omitted only one, the even prime 2.
2. I had done half the cases. I had handled all the odd primes, but none of the even ones.
3. I had dealt with all the uninteresting cases, but not the single interesting case :-)

Exercises: Linear operators Top

1. Find a linear operator S such that
 S2
 =
é
ê
ë
 -1
 0
 0
 -1
ù
ú
û

2. If S is the operator
 S
 =
é
ê
ê
ê
ê
ê
ê
ê
ê
ë
 0
 - p 4
 p 4
 0
ù
ú
ú
ú
ú
ú
ú
ú
ú
û
,
what is exp(S)?

3. Solve the system of equations:
 3x1
 +
 x2
 -
 2x3
 =
 3
 x1
 -
 2x2
 -
 x3
 =
 1
 2x1
 -
 4x2
 -
 x3
 =
 3
4. Solve the differential equation:
 d5f dx5 - 7 d4f dx4 + 23 d3f dx3 - 45 d2f dx2 + 48 df dx -20f
 = 26cos(x) - 18sin(x)
Hint:
 x5 - 7x4 + 23x3 - 45x2 + 48x - 20
 = (x - 1)(x - 2)2(x2 - 2x + 5)

5. Find the general solution to the difference equation
 (E2 - E - 1)((an)) = (0)

6. Find the general solution to the difference equation
 (E4 + 2E2 + 1)((an)) = (0)

7. Verify that

1. D, the derivative, is a linear operator.
2. (D- r)k(xk-1erx) = 0
 (D2 -2sD+ s2 + t2)k(xk-1esxcos(tx))
 =
 0
 (D2 -2sD+ s2 + t2)k(xk-1esxsin(tx))
 =
 0
3. E, the discrete increment, is a linear operator.
4. (E - a)k((njan)) = (0) for 0 £ j < k.

8. What happens in nonlinear cases? Sometimes they are manageable, sometimes not.

1. Solve the differential equation
 Df = b * f * (1 - f).

2. What can be said about the difference equation
 E((an)) = (b * an * (1 - an))?
Note: this is often called the logistics equation.

Normed linear spaces Top

• A norm on a real or complex vector space V is a function
 || · || : V ® R
with the properties, for v, v1, v2 Î V,

1. ||v|| ³ 0
2. ||v|| = 0 if and only if u = 0
3. ||av|| = |a| ||v||
4. ||v1 + v2|| £ ||v1|| + ||v2||.

A space with such an associated function is called a normed linear space.

• Using the norm, we can define a topology on the space, and can then talk about continuity of functions or operators, limits of sequences, etc.

I won't go into this much here beyond a few examples, but good places to look are books on Hilbert Spaces and/or functional analysis. There are a few books indicated in the references.

• Using the norm, we can define a metric on V by: d(v1, v2) = ||v1 - v2||. Metrics have the properties:

1. d(v1, v2) ³ 0
2. d(v1, v2) = 0 iff v1 = v2
3. d(v1, v2) = d(v2, v1)
4. d(v1, v2) £ d(v1, v3) + d(v3, v2)
(the triangle inequality).

• Some examples:

1. On Rn or Cn, we have the norm
 ||(a1, a2, ¼, an)||2 = (|a1|2 + ¼+ |an|2)1/2.
This is usually called the Euclidean norm.

We can also think of this in terms of the inner product given by
< (a1, a2, ¼, an), (b1, b2, ¼, bn) >
= a1[b]1 + a2[b]2 + ... + an[b]n.

We then have ||v||2 = < v, v > 1/2.

2. We can generalize, for p > 0, to
 ||(a1, a2, ¼, an)||p = (|a1|p + ¼+ |an|p)1/p,
and
 ||(a1, a2, ¼, an)||¥ = max (|a1|, ¼, |an|).
These are called the p-norms and the ¥-norm (or max-norm).

A fun little exercise is to draw the circle of radius 1 in R2 for each of these norms:

 circlep(1) = {(x, y) Î R2 | ||(x, y)||p = 1}
for p > 0, or for p = ¥.

One of these constitutes a `proof'' that a square is a circle :-)

3. We can generalize this to R¥ or C¥:
 ||(a1, a2, ¼)||p
 =
 æè ¥ å n = 1 |an|p öø 1/p ,
 ||(a1, a2, ¼)||¥
 =
 sup n (|an|).
For this to make sense, we will have to limit ourselves to the subspaces where the sum converges:
 lp
 =
 {(a1, a2, ¼) | æè ¥ å n = 1 |an|p öø 1/p < ¥},
 l¥
 =
 {(a1, a2, ¼) | sup n (|an|) < ¥}.

4. On C0(R) or C0(C), we can define
 ||f||p = æè óõ |f|p öø 1/p
 ||f||¥ = sup x (|f(x)|).
Again, we limit ourselves to the subspaces where these are < ¥, and call the corresponding spaces Lp or L¥.

5. We can think each of these Euclidean norms, || · ||2, as coming from an inner product:
 ||(a1, a2, ¼)||2
 =
 < (a1, ¼), (a1, ¼) > 1/2
 =
 æè ¥ å n = 1 an _a n öø 1/2
 ||f||2
 =
 < f, f > 1/2
 =
 æè óõ f _f öø 1/2 .
When a complex Euclidean normed linear space is complete (that is, every Cauchy sequence converges), it is called a Hilbert space.

Exercises: Normed linear spaces Top

1. Verify that each of the examples actually are norms.

2. In R2, draw the unit circles

1. circle1(1) = {(x, y) Î R2 | ||(x, y)||1 = 1}
2. circle3/2(1) = {(x, y) Î R2 |
||(x, y)||3/2 = 1}
3. circle2(1) = {(x, y) Î R2 | ||(x, y)||2 = 1}
4. circle3(1) = {(x, y) Î R2 | ||(x, y)||3 = 1}
5. circle¥(1) = {(x, y) Î R2 |
||(x, y)||¥ = 1}

3. Suppose that V is a real or complex vector space. An inner product on V is a conjugate-bilinear function on V:
 < ·, · > : V x V ® F
where, for all v1, v2, v Î V, and a Î F,
 < v , v >
 ³
 0
 < v , v >
 =
 0 iff v = 0
 < v1 + v2 , v >
 =
 < v1 , v > + < v2 , v >
 < v1 , v2 >
 =
 < v2 , v1 >
 < av1 , v2 >
 =
 a < v1 , v2 > .
Show that for an inner product,
 < v , v1 + v2 >
 =
 < v , v1 > + < v , v2 >
 < v1 , av2 >
 =
 a < v1 , v2 > .
Show that for a finite dimensional real or complex vector space V with basis (v1, v2, ¼, vn), the function f : V x V ® F given by
 f æè å i ai vi, å i bi vi öø = å i ai bi
is an inner product.
4. Recall that a linear functional on a vector space V is a linear map f : V ® F. For a finite dimensional real or complex inner product space V, define the dual space of V to be the space
 V* = { f : V ® F |  f is a linear functional}
Show that V* is a vector space over F, and that V and V* are isomorphic to each other.

Hint: Show that every f Î V* corresponds with a function of the form

 < vf , · > : V ® F
for some vf Î V.

Eigenvectors and eigenvalues Top

• Suppose T is a linear operator on V. Then l Î F is called an eigenvalue or characteristic value of T if there exists v ¹ 0 in V with T(v) = lv. In this case we call v an eigenvector or characteristic vector of T.

• An equivalent definition is that l is an eigenvalue of T if and only if (T - lI)v = 0 for some v ¹ 0. This follows from the fact that 0 = (T - lI)v = T(v) - lv if and only if T(v) = lv. Note that this also means that ker(T - lI) ¹ {0}. Thus, l is an eigenvalue if and only if (T - lI) is not a monomorphism. In the finite dimensional case, this is equivalent to (T - lI) not being invertible.

• Now suppose that l1, ..., lk are distinct eigenvalues of T (i.e., li ¹ lj for i ¹ j), with associated eigenvectors v1, ¼, vk. Then {v1, ¼, vk} is a linearly independent set.

• This means in particular that if T Î L(V) with dim(V) = n, and T has n distinct eigenvalues li with associated eigenvectors vi, then the set S = (v1, ¼, vn) is a basis for V. Furthermore, the matrix representation of T with respect to the basis S is
[T] = é
ê
ê
ê
ê
ë
 l1
 0
 ¼
 0
 0
 l2
 ¼
 0
 :
 :
 :
 :
 0
 0
 ¼
 ln
ù
ú
ú
ú
ú
û
or [T]ii = li, [T]ij = 0 for i ¹ j. This is also sometimes written as [T] = diag(l1, ¼, ln).

Eigenvalues and eigenvectors can thus give us a very simple representation for T.

• Not all linear operators have eigenvalues (or eigenvectors). For example, the linear operator Tq : R2 ® R2 given by
[Tq] = é
ê
ë
 cos(q)
 -sin(q)
 sin(q)
 cos(q)
ù
ú
û
is a (counter-clockwise) rotation around the origin through the angle q. If q is not an integral multiple of p, then Tq has no eigenvalues.

• More generally, consider a non-zero linear operator T : R2 ® R2. If 0 ¹ u Î R2, then there must be a linear combination of {u, T(u), T2(u)} with
 0 = T2(u) + aT(u) + b(u) = Pu(T)(u).
If Pu(T) factors into linear factors Pu(T) = (T - l1I)(T - l2I) with l1, l2 Î R, then l1 and l2 are eigenvalues of T. On the other hand, if a2 - 4b < 0, then T has no eigenvalues (it is a general rotation in R2).

• On the other hand, every non-zero linear operator T : Cn ® Cn has an eigenvalue. We can see this from the fact that, for any u ¹ 0 in Cn, the set of vectors {u, T(u), T2(u), ¼, Tn(u)} must be a linearly dependent set, and hence there are a0, a1, ¼, an not all zero with
 0 = n å i = 0 aiTi(u) = Pu(T)(u).

We know that we can factor the polynomial Pu(T) over C into a product of linear factors

 Pu(T) = Õ j (T - zjI)kj(T).
This linear operator is not a monomorphism (since u Î ker(Pu(T))). It is a product of commutative factors, and hence at least one of the factors (T - zjI) is not a monomorphism. This says then that at least one of the zj is an eigenvalue of T. We can proceed to find n eigenvalues and eigenvectors.

• For for every non-zero linear operator T : Cn ® Cn, there is a polynomial PT(T) = åiaiTi of degree n, called the characteristic polynomial of T, with:

1. PT(T) = 0. That is, PT(T)(u) = 0 for all u Î Cn.
2. The linear factors of PT(T) are each of the form (T - ljI) for some eigenvalue lj. Also, an = 1. That is,
 PT(T) = Õ j (T - ljI)kj(T).
3. For each eigenvalue lj, there is an eigenvector uj. If kj > 1, there is a linearly independent set of kj eigenvectors for lj. The set of n eigenvectors {ui} is a linearly independent set, and thus is a basis for Cn. As above, the matrix representation of T with respect to this basis is [T] = diag(l1, ¼, ln) (with repetitions as necessary).

• A non-zero linear operator T : Rn ® Rn also has a characteristic polynomial of degree n. The big difference for the real case is that the polynomial will factor into a combination of linear and quadratic factors. There is then a basis consisting partially of eigenvectors and partially of pairs of basis vectors for two-dimensional subspaces on which T is a rotation. The matrix of T with respect to this basis then looks like [T] = diag(l1, ¼, lk, Ak+1,¼,A(n-k)/2) where each Ai is a two-by-two matrix without eigenvalues.

• The set of eigenvalues of an operator is sometimes called the spectrum of the operator. In cases like Cn where the eigenvectors form a basis for the space, using such as basis is sometimes known as the spectral decomposition of the operator.

• Just briefly, let's glance at differential operators. D: C¥(R) ® C¥(R) has uncountably many eigenvalues. Each real number a Î R is an eigenvalue since
 D(eax) = aeax.
The corresponding eigenvectors are of course fa(x) = eax.

The differential operator D2 also has uncountably many eigenvalues and eigenvectors since for a > 0,

 D2(cos(Öax))
 =
 -acos(Öax),
 D2(sin(Öax))
 =
 -asin(Öax), and
 D2(eÖax)
 =
 aeÖax.

• In this context, here's an example which puts many of these pieces together. In quantum mechanics, a typical expression of Schrödinger's equation looks like
 éê ë - (h/2p)2 2me Del2+V(x,y,z) - i(h/2p) ¶ ¶t ùú û Y = 0.
This example is for an electron (with mass me) in a potential field V(x,y,z).

The general solution of this operator equation is

 Y(x,y,z,t) = ¥ å n = 0 cnYn(x,y,z)exp æç è -iEnt (h/2p) ö÷ ø
where Yn(x,y,z) is an eigenfunction solution of the associated time independent Schrödinger equation, with En the corresponding eigenvalue. The inner product, giving a time dependent probability, looks like
 P(t) = óõ Y Y dv.

Exercises: Eigenvectors and eigenvalues Top

1. Show that if v is an eigenvector of T corresponding with the eigenvalue l, and a Î F (a ¹ 0), then av is also an eigenvector of T corresponding with l. In particular, eigenvectors are not unique.

2. Define T Î L(F2) by T((x, y)) = (y, x). Find all the eigenvalues and eigenvectors of T.

3. Define T Î L(F3) by T((x, y, z)) = (-y, 0, 2z). Find all the eigenvalues and eigenvectors of T.

4. Define T Î L(F¥) by T((a1, a2, ¼)) = (a2, a3, ¼) (i.e., T is the left shift operator). Find all the eigenvalues and eigenvectors of T.

5. Suppose T Î L(V) is invertible. Show that l ¹ 0 is an eigenvalue of T if and only if l-1 is an eigenvalue of T-1.

6. Suppose S, T Î L(V). Show that ST and TS have the same eigenvalues.

7. Give an example of an operator whose matrix has all zeros on the diagonal with respect to some basis, but which is invertible. Give an example of an operator whose matrix has all diagonal elements non-zero with respect to some basis, but which is singular (i.e., has no inverse).

8. Show that if S, T Î L(V), S is invertible, and P(x) Î F[x], then P(S-1TS) = S-1P(T)S.

9. Suppose that V is a finite dimensional normed complex vector space, and T is an isometry of V (i.e., ||T(v)|| = ||v|| for all v Î V). Show that every eigenvalue l of T has |l| = 1. Hint: show that there is a basis for V consisting of eigenvectors ei, with ||ei|| = 1 for 1 £ i £ n.

10. Let V be a complex vector space, and T Î L(V). Let P(x) Î C[x]. Show that a Î C is an eigenvalue of P(T) if and only if a = P(l) for some eigenvalue l of T.

11. Is the preceding true for real vector spaces? Why not? (Note: this is another example of why C is so much nicer a place to work than R ...)

Is the preceding true if we replace C[x] with C¥[x], power series? If so, show it. If not, give a counterexample.

Change of basis Top

• Recall that the matrix associated with a linear transformation depends on the particular bases we use. In the case of a linear operator T : V ® V, it is typical to use the same base for V as domain and as codomain. However, as we have seen, a basis consisting of eigenvectors is particularly convenient, and hence it is useful to know how to change bases.

If V is finite dimensional, with two bases S1 = (u1, ¼, un) and S2 = (v1, ¼, vn), we can consider the matrices of T with respect to the mixed bases. We can indicate this by the various symbols [T]S1, [T]S2, [T]S1S2, and [T]S2S1, where, for example

 T(ui)
 =
 å j [T]jiS1uj, and
 T(ui)
 =
 å j [T]jiS1S2vj.

• We can look in particular at the matrix representations for the identity operator I. For ease of notation, let [B] = [I]S1S2, the matrix representation for I using S1 for the domain and S2 for the codomain. In particular, we then have
 ui = å j [B]jivj = Bijvj.
We also have that the matrix inverse [B]-1 = [I]S2S1 which takes us in the opposite direction. Putting the pieces together, we then have
 [B][T]S1 = [T]S2[B].
That is, doing the operator T in the basis S1 and then converting to the basis S2 is the same as converting bases first, and then doing T in the second base.

Another way to say this is:

 [T]S1 = [B]-1[T]S2[B]
or
 [T]S2 = [B][T]S1[B]-1.

Exercises: Change of basis Top

1. Show that if A and B are operators on a finite dimensional vector space with AB = I, the identity operator, then BA = I, and so B = A-1.

2. Suppose that T is an operator that has the same matrix with respect to every basis. Show that T must be some multiple of the identity operator I.

Trace and determinant Top

• Let V be a finite dimensional vector space over R or C, and T : V ® V an operator on V. Let PT(T) = åi = 0n aiTi be the characteristic polynomial of T. We can then define the trace of T by trace(T) = an-1, the coefficient of Tn-1.

We can also define the trace of an n x n real or complex matrix A by tracem(A) = åi = 1nAii, the sum of the diagonal elements of A.

These definitions are consistent, in the sense that trace(T) = tracem([T]), where [T] is the matrix of T with respect to some basis for V. An exercise will be to show that it doesn't matter what basis we use (they all come out the same). Since these definitions are consistent, we will ordinarily write them both the same way, as trace().

• For S,T : V ® V, and a, b Î F, the trace has the nice properties:

1. trace(aS + bT) = a trace(T) + b trace(T). This says that the trace is a linear functional on L(V).
2. In the complex case, we have
 trace(T) = å i li
where the li are the eigenvalues of T.
3. trace(ST) = trace(TS).

• From these we can derive nice facts like:

If we define the commutator of S and T by [S, T] = ST - TS, then we always have [S, T] ¹ I, the identity operator.

• We can define the determinant of an operator T by det(T) = (-1)n a0, where a0 is the constant term in the characteristic polynomial.

• The determinant has the nice properties:

1. In the complex case, we have
 det (T) = Õ i li
where the li are the eigenvalues of T.
2. det(T) ¹ 0 iff ker(T) = {0} iff T is invertible.
3. det(ST) = det(S)det(T) = det(TS).
4. PT(z) = det(zI - T). That is, the characteristic polynomial is the determinant of the generalized operator (zI -T), where we think of z as a complex variable.
5. We can define the determinant of a square matrix M = [aij] by
 detm(M) = å p Î perm(n) sign(p) Õ i ap(i)i
where perm(n) is the set of all permutations of the numbers (1, 2, ¼, n), sign(p) is the sign of the permutation, and p(i) is the ith number in the permutation p.

We then have:

 detm([T]) = det (T).
As for the trace, since the two definitions are consistent, we will typically denote both determinants by the same symbol det().

Showing that these two definitions are consistent is a fair amount of work (just looking at the formula for detm() should give you some idea). You can look it up if you are interested.

6. There is another definition of determinant. Suppose V is a real or complex n-dimensional vector space. Then there is an alternating multi-linear functional
 detv: Vn ® F
with the properties, for all v1, v2, ¼, vn, vi1, vi2 Î V and a, b Î F:

1. detv(v1, ¼, vi, ¼, vj, ¼, vn) = - detv(v1, ¼, vj, ¼, vi, ¼, vn) (alternating)

2. detv(v1, ¼, avi1 + bvi2, ¼, vn) = adetv(v1, ¼, vi1, ¼, vn) + bdetv(v1, ¼, vi2, ¼, vn) (multi-linear)

3. detv(e1, ¼, en) = 1 for a particular basis {e1, ¼, en} for V. (uniqueness)

We then have detm(A) = detv(A1, ¼, An), where Ak is the kth column of the matrix A, considered as a vector.

We have that det(T) = detm([T]) = detv([T]1, ¼, [T]n). The fact that there are three different versions of the same thing suggests that many people have worked on this topic, and that this topic occurs in a variety of contexts ...

7. Recall the discussion of norms on vector spaces. The Euclidean norms (|| v ||2), which can be thought of as coming from an inner product ( < v1, v2 > ), have a nice relationship with the determinant:
 ||T(v)||2 = || v ||2 for all v Î U
 iff | det (T)| = 1.

8. An operator which preserves the Euclidean norm (||T(v)||2 = ||v||2) is called an isometry. When the scalar field is R, these are called orthogonal operators. In the case of C, they are called unitary operators. In the case of the quaternions, they are called symplectic operators.

9. The isometries on a normed linear space have the properties:

1. If T is an isometry, then det(T) ¹ 0, and hence T has an inverse, T-1, which is also an isometry. Of course, I, the identity operator, is an isometry.
2. If T1 and T2 are isometries, then T1T2 is also an isometry.
3. This means that the set of isometries forms a group. In the three cases described above, these are called the orthogonal group, the unitary group, and the symplectic group.

10. Another way to characterize the isometries (in the finite dimensional case, but also with appropriate generalizations in infinite dimensional cases) is that an operator is an isometry if 1.) its inverse exists, and 2.) the matrix of its inverse is given by the conjugate transpose. That is, if
 [T-1]ij = [T]ji .

11. The determinant also plays an important role in change of variables formulas (in Rn, for example).

Suppose W Ì Rn and s: W ® Rn. We will think of s as a (local) coordinate system on W, or as a change of variables.

The derivative of s at x is the unique operator T (if it exists) satisfying:

 lim y® 0 ||s(x + y) - s(x) - Ty|| ||y|| = 0.
If this operator T exists, s is called differentiable, and T is denoted by s¢(x). Note that s¢(x) Î L(Rn). We can write s(x) = (s1(x), ¼, sn(x)), and we can denote the partial derivative of sj with respect to the kth coordinate by Dksj(x).

If s is differentiable at x, then the matrix of s¢(x) is given by:

[s¢(x)] = é
ê
ê
ê
ë
 D1s1(x)
 ¼
 Dns1(x)
 :
 :
 D1sn(x)
 ¼
 Dnsn(x)
ù
ú
ú
ú
û
If we assume that W is a reasonable set (e.g., open, or measurable) and f : W® R is also reasonable (e.g., continuous, or measurable), then we have the change of variables formula:
 óõ s(W) f(y)dy = óõ W f(s(x))| det (s¢(x))|dx.

Exercises: Trace and determinant Top

1. Given two real or complex n x n matrices A and B, show that tracem(AB) = tracem(BA).

2. Given an operator T on a real or complex finite dimensional vector space V, show that tracem([T]) and detm([T]) are independent of the basis used for the matrix representation [T]. Hint: use the change of basis formula [T]S1 = [B]-1[T]S2[B], the previous problem, and detm(AB) = detm(BA).

3. Show that trace is linear, that is, trace(aS + bT) = atrace(S) + btrace(T).

4. Show that trace(T) = åi = 1n[T]ii.

5. For two operators S and T, show that [S, T] ¹ I. Recall that [S,T] = ST - TS.

6. Show by example that in general, trace(ST) ¹ trace(S)trace(T). In particular, find an operator T on a real vector space V with trace(T2) < 0.

7. Suppose A is an n x n real matrix, S Î L(Rn) has matrix representation A with respect to some basis, and T Î L(Cn) has matrix representation A with respect to some basis. Show that trace(S) = trace(T) and det(S) = det(T).

8. Show that if T is an isometry on a finite dimensional normed vector space, then |det(T)| = 1.

9. Show that if T is an operator on a complex vector space of dimension n, and a Î C, then det(aT) = andet(T).

Top

## References

[1]
Axler, Sheldon, Linear Algebra Done Right, second edition, Springer-verlag, New York, 1997.

[2]
Finney, Ross L., and Ostberg, Donald R., Elementary Differential Equations with Linear Algebra, Addison-Wesley, 1976.

[3]
Hoffman, Kenneth, and Kunze, Ray, Linear Algebra, second edition, Engineering/Science/Mathematics, 1971.

[4]
Naimark, M. A., Normed Algebras, Wolters-Noordhoff, Groningen, the Netherlands, 1974.

[5]
Noble, Ben, and Daniel, James W., Applied Linear Algebra, third edition, Engineering/Science/Mathematics, 1988.

[6]
Prugovecki, Eduard, Quantum Mechanics in Hilbert Space, second edition, Academic Press, New York, 1981.

[7]
Rudin, Walter, Functional Analysis, McGraw-Hill, New York, 1973.

[8]
Strang, Gilbert, Linear Algebra and its Applications, Academic Press, New York, 1976.