Korean, Edit

Chapter 1. Vector Spaces

Recommended post: 【Linear Algebra】 Table of Contents for Linear Algebra

1. [Matrix Operations]

2. [Vector Spaces]

1. Matrix Operations

⑴ Definition of a Matrix

① Define an m × n matrix A, its i-th row vector, and its j-th column vector as follows

② Zero matrix or null matrix: a matrix whose entries are all 0

③ Square matrix

⑵ Matrix Addition

⑶ Matrix Multiplication

① Let X[i][j] denote the entry in the i-th row and j-th column of matrix X. If A ∈ ℝ^l×m, B ∈ ℝ^m×n, C ∈ ℝ^l×n, and C = A × B, then the following holds

② Property 1. In general, AB ≠ BA

③ Property 2. AB = O does not necessarily imply that A = O or B = O

④ Property 3. The following relationships hold between matrix multiplication and addition

○ A(B+C) = AB + AC, (A+B)C = AC + BC

○ A(BC) = (AB)C

○ c(A + B) = cA + cB

⑤ Property 4. Submultiplicative property: ‖𝐴^𝑘𝑥₀‖_∞ ≤ ‖𝐴‖_∞^𝑘‖𝑥₀‖_∞, where ‖·‖_∞ is a maximum element value of a matrix or vector)

⑥ Theorem 1. Cayley–Hamilton theorem

○ By applying this theorem, Aⁿ (where n ≥ 2) can be calculated in the form Aⁿ = (A² - (a + d)A + (ad - bc)E) Q(A) + kA + sE

⑦ Programming Code

○ Implementing Matrix Multiplication in C

○ Using Python numpy

import numpy as np

# Define two matrices
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])

# Print matrices
print("Matrix A:")
print(A)
print("\nMatrix B:")
print(B)

# Matrix multiplication
result = np.dot(A, B)
## Alternatively, result = A@B

print("\nResult of Numpy matrix multiplication:")
print(result)

○ Using Python sympy

from sympy import Matrix, init_printing

# Initialization of LaTeX style print
init_printing()

# Define two matrices
A = Matrix([[1, 2], [3, 4]])
B = Matrix([[5, 6], [7, 8]])

# Print matrices 
print("Matrix A:")
display(A)

print("Matrix B:")
display(B)

# Matrix multiplication
result = A * B

print("Result of Sympy matrix multiplication:")
display(result)

⑷ Transposed Matrix

① Definition

② Symmetric matrix, sym: a square matrix A that is equal to its transpose

③ Given X ∈ ℝ^N×d (N pieces of d-dimensional data),

○ X^tX ∈ ℝ^d×d is the covariance matrix when μ = 0

○ XX^t ∈ ℝ^N×N is the dot matrix (similarity matrix)

④ Property 1. (A^t)^t = A

⑤ Property 2. (A + B)^t = A^t + B^t

⑥ Property 3. (cA)^t = cA^t

⑦ Property 4. The transpose of AB is (AB)^t = B^tA^t

⑧ Property 5. If A and B are symmetric and AB = BA, then AB is symmetric

⑨ Property 6. The inverse of a symmetric matrix is also symmetric

⑸ Trace: denoted by trace or tr

① Property 1. tr(E) = n, tr(O) = 0

② Property 2. tr(A + B) = tr(A) + tr(B)

③ Property 3. tr(cA) = ctr(A), c ∈ ℝ

④ Property 4. tr(A^T) = tr(A)

⑤ Property 5. tr(AB) = tr(BA)

⑥ Property 6. In general, tr(AB) ≠ tr(A) × tr(B)

⑦ Property 7. tr(A^TB) = vec(A)·vec(B)

⑧ Property 8. tr(ABC) = tr(BCA) = tr(CAB) (∵ associativity)

⑨ Property 9. If A and B are positive semi-definite matrices, then tr(AB) ≤ tr(A)·tr(B)

⑩ Property 10. tr(Aⁿ) = tr(P^-1DⁿP) = tr(P^-1PDⁿ) = tr(Dⁿ) = ∑_i λ_iⁿ

○ However, the equation above appears to hold independently of diagonalizability

⑪ Property 11. If X ~ 𝒩(X̄, ∑) and S is a symmetric matrix, then 𝔼[XSX] = X̄SX̄ + tr(S∑)

⑫ Application 1. Hilbert-Schmidt test

○ Calculates trace of 2 kernels to determine whether they are dependent non-parameteric, non-linear pattern detection.

○ For two matrices M⁽¹⁾ and M⁽²⁾ to which kernels have been applied, the cosine similarity is

⑹ Gaussian Elimination

① System of Linear Equations

② Augmented matrix: an m × (n + ℓ) matrix denoted by $(A \mid B)$

③ Gauss-Jordan elimination: operations are performed by row

○ Operation 1. Swap the positions of two row vectors: changes the order of the equations

○ Operation 2. Multiply a specific row vector by c: multiplies both sides of one equation by c

○ Operation 3. Multiply one row vector by c and add it to another row vector: multiplies one equation by c and adds it to another equation

○ Final step: a reduced row echelon form (RREF) must be obtained

④ Advantage: it can be implemented through programming

2. Vector Spaces

⑴ Conditions for a Vector Space: one type of algebraic structure, such as ℝⁿ

① Closed under addition: if x, y ∈ V, then x + y ∈ V

② Closed under scalar multiplication: if ∀a ∈ ℝ and ∀x ∈ V, then ax ∈ V

③ Additive identity: for every element x of V, there exists 0 in V such that x + 0 = x

④ Additive inverse: if x ∈ V, then there exists -x in V such that x + (-x) = 0

⑤ Commutativity of addition: if x , y ∈ V, then x + y = y + x

⑥ Associativity of addition: if x , y , z ∈ V, then (x + y) + z = x + (y + z)

⑦ Distributive law: c(x + y) = cx + cy

⑧ Distributive law: (c₁ + c₂)x = c₁x + c₂x

⑨ Multiplicative identity: 1x = x

⑩ Associativity of scalar multiplication: c₁(c₂x) = (c₁c₂)x

⑵ Linear Transformations

① Linear transformation: a mapping T: U → V that satisfies the following conditions for x , y ∈ U and c ∈ F

② Linear combination

③ Linearly dependent

④ Linearly independent

⑤ Spanning set: a set S such that every vector in the vector space V can be expressed as a linear combination of elements of S

⑥ Basis: a linearly independent spanning set

⑶ If the inverse map T^-1: V → U exists, then T^-1 is also a linear transformation

⑷ Inner Product Space

① Conditions for an inner product (dot product)

② Standard inner product: the standard inner product in V = ℂⁿ is defined as follows

③ Properties of an inner product

④ Inner product space: a vector space on which an inner product is defined

○ Real inner product space: a case in which the scalar field is the set of real numbers

○ Complex inner product space: a case in which the scalar field is the set of complex numbers

⑤ Norm: a vector space equipped with a norm is called a norm vector space or normed linear space

⑥ Theorem 1. Cauchy-Schwartz inequality

⑦ Theorem 2. Triangle inequality

⑷ Metric Space

① Distance function, metric

○ Types of Distance Functions

② Relationship between a norm and distance

○ If a norm is defined, a distance d can be defined

○ Even if a distance is defined, a corresponding norm does not always exist

③ Kernel

○ A distance d may be defined differently using a nonlinear map φ(x): X → H

○ Definition of a kernel: k(x_i, y_i) ≡ φ(x_i)^Tφ(x_j)

○ Characteristics of kernels

○ Kernel Examples

○ φ(x, y) = (x, y, x² + y²)^T

○ k(v , u) = u^Tv (in this case, φ(·) is the identity function)

○ Gaussian kernel: k(v , u) = exp(-‖v - u‖² / 2σ²)

○ Polynomial kernel: k(v, u) = (u^Tv + 1)^d (where d is the polynomial degree)

○ Sigmoid kernel: k(v, u) = tanh(αu^Tv + β)

○ RBF kernel (radial basis function kernel): k(v , u) = exp(-γ ‖v - u‖²)

○ Lorentz kernel

○ Heat kernel

○ Graph Matérn kernel

⑸ Subspace: a spanning set spans a subspace

Posted: 2020.04.07 21:36

Modified: 2024.10.10 08:12