Chapter 1. Vector Spaces(vector space)
Recommended post: 【Linear Algebra】 Table of Contents for Linear Algebra
1. Matrix Operations
2. Vector Spaces
1. Matrix Operations
⑴ Definition of a Matrix
① Define an m × n matrix A, its i-th row vector, and its j-th column vector as follows
② Zero matrix or null matrix : a matrix whose entries are all 0
③ Square matrix
⑵ Matrix Addition
⑶ Matrix Multiplication
① Let X[i][j] denote the entry in the i-th row and j-th column of matrix X. If A ∈ ℝl×m, B ∈ ℝm×n, C ∈ ℝl×n, and C = A × B, then the following holds
② Property 1. In general, AB ≠ BA
③ Property 2. AB = O does not necessarily imply that A = O or B = O
④ Property 3. The following relationships hold between matrix multiplication and addition
○ A(B+C) = AB + AC, (A+B)C = AC + BC
○ A(BC) = (AB)C
○ c(A + B) = cA + cB
⑤ Theorem 1. Cayley–Hamilton theorem
○ By applying this theorem, An (where n ≥ 2) can be calculated in the form An = (A2 - (a + d)A + (ad - bc)E) Q(A) + kA + sE
⑥ Programming Code
○ Using Python numpy
○ Using Python sympy
⑷ Transposed Matrix
① Definition
② Symmetric matrix, sym : a square matrix A that is equal to its transpose
③ Given X ∈ ℝN×d (N pieces of d-dimensional data),
○ XtX ∈ ℝd×d is the covariance matrix when μ = 0
○ XXt ∈ ℝN×N is the dot matrix (similarity matrix)
④ Property 1. (At)t = A
⑤ Property 2. (A + B)t = At + Bt
⑥ Property 3. (rA)t = rAt
⑦ Property 4. The transpose of AB is (AB)t = BtAt
⑧ Property 5. If A and B are symmetric and AB = BA, then AB is symmetric
⑨ Property 6. The inverse of a symmetric matrix is also symmetric
⑸ Trace : denoted by trace or tr
① Property 1. tr(E) = n, tr(O) = 0
② Property 2. tr(A + B) = tr(A) + tr(B)
③ Property 3. tr(cA) = ctr(A), c ∈ ℝ
④ Property 4. tr(AT) = tr(A)
⑤ Property 5. tr(AB) = tr(BA)
⑥ Property 6. In general, tr(AB) ≠ tr(A) × tr(B)
⑦ Property 7. tr(ATB) = vec(A)·vec(B)
⑧ Property 8. tr(ABC) = tr(BCA) = tr(CAB) (∵ associativity)
⑨ Property 9. If A and B are positive semi-definite matrices, then tr(AB) ≤ tr(A)·tr(B)
⑩ Property 10. tr(An) = tr(P-1DnP) = tr(P-1PDn) = tr(Dn) = ∑i λin
○ However, the equation above appears to hold independently of diagonalizability
⑪ Property 11. If X ~ 𝒩(X̄, ∑) and S is a symmetric matrix, then 𝔼[XSX] = X̄SX̄ + tr(S∑)
⑫ Application 1. Hilbert-Schmidt test
○ Calculates trace of 2 kernels to determine whether they are dependent non-parameteric, non-linear pattern detection.
○ For two matrices M(1) and M(2) to which kernels have been applied, the cosine similarity is
T := (1/n) ∑i,k M(1)ik M(2)ik = (1/n) ∑i,k M(1)ik M(2)ki = (1/n) tr(M(1)×M(2))
⑹ Gaussian Elimination
① System of Linear Equations
② Augmented matrix : an m × (n + ℓ) matrix denoted by (A B)
③ Gauss-Jordan elimination : operations are performed by row
○ Operation 1. Swap the positions of two row vectors : changes the order of the equations
○ Operation 2. Multiply a specific row vector by c : multiplies both sides of one equation by c
○ Operation 3. Multiply one row vector by c and add it to another row vector : multiplies one equation by c and adds it to another equation
○ Final step : a reduced row echelon form (RREF) must be obtained
④ Advantage : it can be implemented through programming
2. Vector Spaces(vector space)
⑴ Conditions for a Vector Space : one type of algebraic structure, such as ℝn
① Closed under addition : if x, y ∈ V, then x + y ∈ V
② Closed under scalar multiplication : if ∀a ∈ ℝ and ∀x ∈ V, then ax ∈ V
③ Additive identity : for every element x of V, there exists 0 in V such that x + 0 = x
④ Additive inverse : if x ∈ V, then there exists -x in V such that x + (-x) = 0
⑤ Commutativity of addition : if x , y ∈ V, then x + y = y + x
⑥ Associativity of addition : if x , y , z ∈ V, then (x + y) + z = x + (y + z)
⑦ Distributive law : c(x + y) = cx + cy
⑧ Distributive law : (c1 + c2)x = c1x + c2x
⑨ Multiplicative identity : 1x = x
⑩ Associativity of scalar multiplication : c1(c2x) = (c1c2)x
⑵ Linear Transformations
① Linear transformation : a mapping T : U → V that satisfies the following conditions for x , y ∈ U and c ∈ F
② Linear combination
③ Linearly dependent
④ Linearly independent
⑤ Spanning set : a set S such that every vector in the vector space V can be expressed as a linear combination of elements of S
⑥ Basis : a linearly independent spanning set
⑶ If the inverse map T-1 : V → U exists, then T-1 is also a linear transformation
⑷ Inner Product Space
① Conditions for an inner product (dot product)
② Standard inner product : the standard inner product in V = ℂn is defined as follows
③ Properties of an inner product
④ Inner product space : a vector space on which an inner product is defined
○ Real inner product space : a case in which the scalar field is the set of real numbers
○ Complex inner product space : a case in which the scalar field is the set of complex numbers
⑤ Norm : a vector space equipped with a norm is called a norm vector space or normed linear space
⑥ Theorem 1. Cauchy-Schwartz inequality
⑦ Theorem 2. Triangle inequality
⑷ Metric Space
① Distance function, metric
② Relationship between a norm and distance
○ If a norm is defined, a distance d can be defined
○ Even if a distance is defined, a corresponding norm does not always exist
③ Kernel
○ A distance d may be defined differently using a nonlinear map φ(x) : X → H
○ Definition of a kernel : k(x i, y i) ≡ φ(x i)Tφ(x j)
○ Characteristics of kernels
○ Kernel Examples
○ φ(x, y) = (x, y, x2 + y2)T
○ k(v , u) = u Tv (in this case, φ(·) is the identity function)
○ Gaussian kernel : k(v , u) = exp(- v - u 2 / 2σ2)
○ Polynomial kernel : k(v , u) = (u Tv + 1)d (where d is the polynomial degree)
○ Sigmoid kernel : k(v , u) = tanh(αu Tv + β)
○ RBF kernel (radial basis function kernel) : k(v , u) = exp(-γ v - u 2)
○ Lorentz kernel
○ Heat kernel
○ Graph Matérn kernel
⑸ Subspace : a spanning set spans a subspace
Posted: 2020.04.07 21:36
Modified: 2024.10.10 08:12