II.2 Unsupervised
learning
1
LE THI HOAI AN
University of Lorraine
2
Outline
 Clustering
Hard clustering, Fuzzy clustering,
Hierarchical clustering,
Weighted clustering,
Clustering by maximizing Modularity,
Self Organizing Map (SOM)
 Principal Component Analysis (PCA)
 Singular Value Decomposition (SVD)
Clustering
4
Clustering
A technique to find similar
groups in data clusters
General Objective:
Given: A dataset of 𝑚 points in 𝑛-dimensional real space
Problem: Extract hidden distinct properties by clustering the
dataset into 𝑘 clusters
⇒ Choose 𝑘 members as centroid (or “median”) and assign
each member to its closest centroid.
5
Clustering
Clustering is a fundamental problem in unsupervised
learning which has many applications in various domains.
In recent years, there has been significant interest in
developing clustering algorithms to the massive data sets.
Two main approaches have been studied for clustering:
 the statistical and machine learning based on learning
mixture models
 the mathematical programming approach that considers
clustering as an optimization problem.
6
Clustering
The general term “clustering” covers many different
types of problems.
All consist of subdividing a data set into groups of
similar elements,
But there are many measures of similarity, many ways
of measuring, and various concepts of subdivision.
7
Clustering
8
Clustering
 Clustering rows:
grouping similar objects
 Clustering columns:
grouping similar variables across
samples
 Bi-clusteing/Two-way clustering:
grouping aobjects that are similar
across a subset of variables
9
Clustering
Two essential components of cluster analysis:
 Distance measure:
A notion of distance or similarity of two objects:
When are two objects close to each other?
 Cluster algorithm:
A procedure to minimize distance of objects
within groups and/or maximize distance between
groups
10
Examples of distance measure
 Euclidean distance measure
average
difference
across
coordinates
 Manhattan distance measures
average
difference
across
coordinates in a robust way
Manhattan distance: the red, yellow, and blue paths all
have the shortest length of 12
Euclidean distance: the green line has length 6 2 =
8.49, and is the unique shortest path.
11
Clustering algorithms
 Popular algorithms for clustering
 Hard clustering
 Hierarchical clustering
 K-mean
 SOMs (Self-Organizing Maps)
 Autoclass, mixture models
 Hierarchical clustering allows the choice of the
dissimilarity matrix
 K-mean and SOMs take orginal data directly as input.
Attributes are assumed to live in Eulcidean space.
Hard Clustering
12
Minimum Sum-of-Squares Clustering
 An instance of the partitional clustering problem consists
of a data set 𝐴 ≔ {𝑎1 , … , 𝑎𝑚 } of 𝑚 points in ℝ𝑛 , a
measured distance, and an integer 𝑘.
 Goal: Choose 𝑘 members 𝑥 ℓ (ℓ = 1, … 𝑘) (in 𝐴 and/or
ℝ𝑛 ) as ”centroid” (or ”median”) and assign each member
of A to its closest centroid.
 The assignment distance of a point 𝑎 ∈ 𝐴 is the distance
from 𝑎 to the centroid to which it is assigned,
 The objective function, which is to be minimized, is the
sum of assignment distances.
13
Hard Clustering
 If the centroids are not necessarily in 𝐴, then the
problem can be formulated as a unconstrained
optimization problem.
 In the contrary case, we are faced with a discrete
optimization problem.
 In both cases, different objective functions
corresponding to the distance metric being considered
are possible.
 Two models widely studied in the literature are the
cases where the points come from a real space ℝ𝑛 and
the assignment distance of a point is defined as the
squared Euclidean distance (2-norm) and/or the 1norm
15
𝒌-mean algorithm
Given a random initial set of k means, the algorithm
proceeds by alternating between two steps:
Assignment step: Assign each observation to the
cluster whose mean yields the least within-cluster
sum of squares (i.e. the "nearest" mean in
Euclidean distance)
Update step: Calculate the new means to be the
centroids of the observations in the new clusters.
19
Clustering: 𝒌-means model
If the squared Euclidean distance is used and the
centroids are not necessarily in 𝑨,
then the corresponding optimization problem can be
expressed as (|| ⋅ || denotes the Euclidean norm)
Bilevel model
𝑚
min ෍ min
𝑖=1
ℓ=1,…,𝑘
2
ℓ
𝑖
𝑥 −𝑎
: 𝑥 ℓ ∈ ℝ𝑛 , ℓ = 1, … , 𝑘
20
Kernel MSSC
Consider the Gaussian kernel function 𝜅: ℝ𝑑 × ℝ𝑑 → ℝ
and the corresponding mapping 𝜙: ℝ𝑑 → 𝐻 (a Hilbert
space) such that
𝑥−𝑦 2
𝜙 𝑥 , 𝜙(𝑦) = 𝜅 𝑥, 𝑦 = exp −
2𝜎 2
𝜎>0
We have the expression
𝜙 𝑥
ℓ
𝑖
− 𝜙(𝑎 )
2
ℓ
𝑥 −𝑎
= 2 − 2exp −
2𝜎 2
𝑖 2
The optimization model of the Gaussian kernel MSSC
𝑚
𝑥 ℓ − 𝑎𝑖
min ෍ min −2exp −
ℓ=1,…,𝑘
2𝜎 2
𝑖=1
2
: 𝑥 ℓ ∈ ℝ𝑛 , ℓ = 1, … , 𝑘
21
Clustering:
𝐢𝐧𝐭𝐞𝐠𝐞𝐫 𝐩𝐫𝐨𝐠𝐫𝐚𝐦𝐦𝐢𝐧𝐠 𝐦𝐨𝐝𝐞𝐥
Let 𝑈 = 𝑢ℓ𝑖 ∈ ℝ𝑘×𝑛 with ℓ = 1, … , 𝑘 and 𝑖 = 1, … , 𝑚 be
the matrix defined by
1 if 𝑎𝑖 belongs to cluster ℓ𝑡ℎ,
𝑢ℓ𝑖 ≔ ቊ
0 otherwise.
Then a straightforward mixed integer formulation of MSSC
𝑚
𝑘
ℓ
෍ ෍ 𝑢ℓ𝑖 𝑥 − 𝑎
𝑖 2
:
𝑖=1 ℓ=1
min
𝑢ℓ𝑖 ∈ 0,1 , 𝑖 = 1, … , 𝑚, ℓ = 1, … , 𝑘
𝑘
෍ 𝑢ℓ𝑖 = 1, 𝑖 = 1, … , 𝑚
ℓ=1
22
𝒌-median problem: bilevel model
In the 𝑘-median problem, the centers must be among the
given data set, i.e. we are to choose 𝑘 members 𝒙ℓ ∈ 𝑨 as
centers and assign each member of 𝐴 to its closest center.
Formulation as a bilevel model
𝑚
෍ min
min
𝑖=1
𝑘
ℓ=1,…,𝑘
෍ min
ℓ=1
𝑖=1,…,𝑚
2
ℓ
𝑖
𝑥 −𝑎
:
𝑥 ℓ − 𝑎𝑖 = 0
23
𝒌-median problem
In the 𝑘-median problem, the centers must be among the
given data set, i.e. we are to choose 𝑘 members 𝒙ℓ ∈ 𝑨 as
centers and assign each member of 𝐴 to its closest center.
Formulation as a linear 0-1 programming:
For 𝑥 ∈ 𝐴, define 0-1 variable 𝑚(𝑥) by
1 if 𝑥 is a center,
𝑚 𝑥 = ቊ
0 otherwise.
For each pair 𝑥 and 𝑦 of 𝐴, define the variable 𝑎 𝑥, 𝑦 :
1 if 𝑥 is a center and 𝑦 is assigned to 𝑥,
𝑎 𝑥, 𝑦 = ቊ
0 otherwise.
24
𝒌-median problem
The 𝑘-median (𝑘 − 𝑀𝑃) takes the form:
min ෍ 𝑎 𝑥, 𝑦 𝑑(𝑥, 𝑦)
𝑥≠𝑦∈𝐴
s. t.
෍ 𝑎 𝑥, 𝑦 ≥ 1 , ∀𝑦 ∈ 𝐴
𝑥∈𝐴
𝑎 𝑥, 𝑦 ≤ 𝑚 𝑥 ∀𝑥, 𝑦 ∈ 𝐴
(𝑘-MP) is
NP-hard
෍𝑚 𝑥 ≤ 𝑘,
𝑥∈𝐴
𝑚 𝑥 ∈ 0,1 , ∀𝑥 ∈ 𝐴
𝑎 𝑥, 𝑦 ∈ 0,1 , ∀𝑥, 𝑦 ∈ 𝐴
First constraint : all points 𝑦 of 𝐴 are assigned to a center.
Second one: if 𝑦 is assigned to 𝑥 then 𝑥 must be a center.
𝑑 𝑥, 𝑦 : the assigned cost (“distance” between 𝑥 and 𝑦).
25
Fuzzy clustering
26
Fuzzy Clustering
 In fuzzy clustering, objects are not classified as belonging
to one and only one cluster, but instead, they all possess a
degree of membership with each of the clusters.
 In real applications, there is very often no sharp boundary
between clusters so that fuzzy clustering is often better
suited for the data.
 This work in included in the mathematical programming
approach and concerns with the fuzzy clustering. We
consider the Fuzzy C-Means (FCM) clustering model that is
undoubtedly a most widely used fuzzy clustering
technique. It was originally introduced by Bezdek in 1981
as a fuzzification of 𝑘-Means model of hard clustering.
27
Fuzzy Clustering
 Let 𝑋 ∶= {𝑥1 , 𝑥2 , … , 𝑥𝑛 } denote 𝑛 objects to be
partitioned into 𝑐 (2 ≤ 𝑐 ≤ 𝑛) homogeneous clusters
𝐶1 , 𝐶2 , … , 𝐶𝑐 where 𝑥𝑘 ∈ ℝ𝑝 (𝑘 = 1, … , 𝑛) represents
multispectral (features) data.
 Consider the matrix 𝑈 = 𝑢𝑖,𝑘 𝑐×𝑛 called the fuzzy
partition matrix in which each element 𝑢𝑖,𝑘 indicates
the membership degree of each object 𝑥𝑘 in the
cluster 𝐶𝑖 (the probability that 𝑥𝑘 belongs to the
cluster 𝐶𝑖 ).
28
Fuzzy Clustering
 The FCM technique is based on optimizing the
objective function:
where 𝑉 is the 𝑐 × 𝑝 -matrix whose 𝑖th row is 𝑣𝑖 ∈
ℝ𝑝 the conter of cluster 𝐶𝑖 . 𝑚 ≥ 1: the fuzziness
index of membership of each datum.
29
Fuzzy Clustering
The mathematical model of FCM is
30
Image
segmentation by fuzzy clustering
Medical image with the Gaussian noise and the results of segmentation (c=3)
31
Hierarchical clustering
32
Hierarchical clustering
 Multilevel hierarchical clustering consists of grouping data
objects into a hierarchy of clusters.
 It has a long history and has many important applications
in various domains, since many kinds of data, including
observational data collected in the human and biological
sciences, have a hierarchical, nested, or clustered
structure.
 Hierarchical clustering algorithms are useful to determine
hierarchical multicast trees for Gird computing using in eScience, e-Medicine or e-Commerce, Multimedia
conferencing, Large-scale dissemination of timely
information, etc.
33
Hierarchical clustering
 A hierarchical clustering of a set of objects can be
described as a tree, in which the leaves are precisely the
objects to be clustered.
 A hierarchical clustering scheme produces a sequence of
clusterings in which each clustering is nested into the next
clustering in the sequence.
 Standard existing methods for Multilevel hierarchical
clustering are often based upon nonhierarchical clustering
algorithms coupled with several iterative control strategies
to repeatedly modify an initial clustering (reordering, and
reclustering) in search of a better one.
34
Hierarchical clustering
 While mathematical programming is widely used for
nonhierarchical clustering problems there exist a few
optimization models and techniques for multilevel hierarchical
clustering ones.
 Problem statement
 Given a set 𝑋 of 𝑛 objects 𝑋 ∶ = {𝑥𝑖 ∈ ℝ𝑑 : 𝑖 = 1, … , 𝑛}, a
measured distance, and an integer 𝑘.
 We are to choose 𝑘 + 1 members in 𝐴, one as the total center
(the root of the tree) and others as centers of 𝑘 disjoint clusters,
and assign other members of 𝐴 to their closest center.
 The total center is defined as the closest object to all centers (in
the sense that the sum of distances between it and all centers is
the smallest).
35
Hierarchical clustering
 A practical application widely studied in the networking
community that is the network topology identification based on
end-to-end measurements.
 Consider a communication network, in which a sender node
transmits information packets to a set of receiver nodes. The
receivers are, in this case, the usual ”objects” to be clustered.
 Assume that the routes from the sender to the receivers are
fixed.
 The physical network topology is essentially a graph, where each
node corresponds to a physical device (e.g., router, switch,
terminal, etc...) and the links correspond to the connections
between them.
 Knowledge of the network topology is essential for tasks like
monitoring and provisioning a network.
36
Hierarchical clustering
The first optimization model:
37
38
Hierarchical clustering
We consider a new mathematical program in which all the
centers are variables. Denote 𝒗𝒍 𝑙 = 1, … , 𝑘 the center of
clusters in the second level and 𝒗𝒌+𝟏 the total center. The
third optimization model:
39
Construction of a multicast
communication network
 The identification problem of the topology of a network:
using clustering algorithms.
 Construction of a multicast communication network
• Objective: minimize the total cost of communication
• Groupware systems, Video conference, video streaming
of international events, P2P applications to share data or
processing, etc.
40
Construction of a multicast
communication network
 A hierarchical multicast tree contains a source (the total
center) with several levels of hierarchy
 A node is connected to a higher-level node (except the
source) and its lower-level nodes (if any)
44
Weighted Clustering
45
Weighted clustering
In many applications such as e-commerce applications,
computational biology, text classification, image analysis,
etc. datasets are large volume and contain a large
number of features.
Three categories of Features: relevant, redundant and
irrelevant features.
 Relevant features are essential for classification
process,
 Redundant features add no new information to the
classifier (i.e., information already carried by other
features) while
 Irrelevant features do not carry any useful information.
46
Weighted clustering
 Feature selection deals with irrelevant or redundant
features.
Goal: select a subset of features that minimize
redundancy while preserving or improving the
classification rate of algorithm.
 Feature weighting: an extension of feature selection.
In feature weighting, each feature is assigned a
continuous value, named a weight, in the interval [0,1].
Relevant features correspond to a high weight value,
whereas a weight value close to zero represent
irrelevant features.
Objective: improve the quality of classification
algorithm, but not to reduce the number of features.
47
Weighted clustering
Problem statement
Given 𝑋 ∶ = {𝑥𝑖 ∈ ℝ𝑑 : 𝑖 = 1, … , 𝑛} and an integer 𝑘.
The dissimilarity measure 𝛿𝑊𝐹 between 𝑣𝑙 and 𝑥𝑖 is now
defined by 𝑑 weighted features:
48
Weighted clustering
Bilevel formulation of MSSC using weighted features
49
50
51
52
Clustering by Maximizing
Modularity
53
Clustering by Maximizing
Modularity
 Community structure: one important network features.
The whole network: composed of densely connected subnetworks, with only sparser connections between them. Such
sub-networks are called communities or modules.
 Detection of communities is of significant practical
importance as it allows to analyze networks at a megascopic
scale:
• identification of related web pages in the WWW,
• uncovering of communities in social networks,
• decomposition of metabolic networks in functional
modules.
54
Clustering by Maximizing
Modularity
 Detecting communities ⇒ searching for the structure
that maximizes the number of intra-community edges
while minimizing the number of inter-community edges.
 The modularity (Q measure): characterize the existence
of community structure in a network.
 The modularity Q of a particular partition is defined as
the number of edges inside clusters, minus the expected
number of such edges if the graph were random
conditioned on its degree distribution.
55
56
57
58
59
Self Organizing Map (SOM)
60
Self Organizing Map (SOM)
 The Self-Organizing Map (SOM), introduced by
Kohonen in 1982, and its variants are a popular
artificial neural network approaches in unsupervised
learning.
 The principal goal of an SOM is to transform an
incoming signal pattern of arbitrary dimension into a
low (usually one or two)-dimensional discrete map,
and to perform this transformation adaptively in a
topologically ordered fashion.
61
Self Organizing Map (SOM)
 The SOM is an elegant way to interpret complex data and
an excellent tool for data visualization and exploratory
cluster analysis. SOM is primarily a data-driven
dimensionality reduction and data compression method.
 In data mining and machine learning, SOM is a widely
used tool in the exploratory phase for various technique
such as data clustering, data classification and graph
mining.
62
63
64
Principal component analysis
(PCA)
65
Principal component analysis (PCA)
66
Principal component analysis (PCA)
 The principal components of a collection of points are a
sequence of direction vectors, where the ith vector is the
direction of a line that best fits the data while being
orthogonal to the first i-1 vectors.
 PCA is the process of computing the principal components
and using them to perform a change of basis on the data,
sometimes using only the first few principal components
and ignoring the rest.
 Use an orthogonal transformation: convert a set of
observations of possibly correlated variables ⇒ a set of
values of linearly uncorrelated variables.
 The number of principal components ≤ that of original
variables.
67
Objectives of PCA
 Reduce dimensionality (pre-processing for other methods)
 Choose the most useful (informative) variables
 Compress the data
 Visualize multidimensional data
• To identity groups of objects
• To identity outliers
68
Example: visualization
Given 53 blood and urine samples (features) from 65
people. How can we visualize the measurements?
550
500
450
400
350
300
250
200
150
100
50
0 50
Tri-variate (3 features)
4
M-EPI
C-LDH
Bi-variate (2 features)
3
2
1
0
600
400
150 250 350 450
C-Triglycerides
C-LDH 200 0 0
500
400
300
200
100
C-Triglycerides
How can we visualize the other variables???
… difficult to see in 4 or higher dimensional spaces...
69
 Is there a representation better than the coordinate axes?
 Is it really necessary to show all the 53 dimensions?
 How could we find the smallest subspace of the 53-D
space that keeps the most information about the original
data?
 A solution: Principal Component Analysis
70
PCA Algorithm
Input: 𝑛 × 𝑚 data matrix 𝑋, with one row vector 𝑥𝑖 per
data point
Step 1: 𝑋 ← subtract mean 𝑥 from each row vector 𝑥𝑖 in 𝑋
Step 2: Σ ← covariance matrix of 𝑋
Step 3: Find eigenvectors and eigenvalues of Σ
Step 4: PC’s ← the eigenvectors with largest eigenvalues
Step 5: Derive the new data
PCA Example –STEP 1
DATA:
x
y
2.5
2.4
0.5
0.7
2.2
2.9
1.9
2.2
3.1
3.0
2.3
2.7
2
1.6
1
1.1
1.5
1.6
1.1
0.9
Original data
71
PCA Example –STEP 1
• Subtract the mean from each of the data
dimensions.
All the 𝑥 values have 𝑥 subtracted and 𝑦 values have
𝑦 subtracted from them. This produces a data set
whose mean is zero.
Subtracting the mean makes variance and covariance
calculation easier by simplifying their equations.
The variance and co-variance values are not affected
by the mean value.
72
PCA Example –STEP 1
DATA:
𝑥
𝑦
2.5
2.4
0.5
0.7
2.2
2.9
1.9
2.2
3.1
3.0
2.3
2.7
2
1.6
1
1.1
1.5
1.6
1.1
0.9
𝑥𝑖 – mean(𝑥)
𝑦𝑖 – mean(𝑦)
ZERO MEAN DATA:
𝑥
𝑦
.69
.49
-1.31
-1.21
.39
.99
.09
.29
1.29
1.09
.49
.79
.19
-.31
-.81
-.81
-.31
-.31
-.71
-1.01
73
PCA Example –STEP 2
• Calculate the covariance matrix of zero mean data
cov =
cov(𝑥, 𝑥)
cov(𝑦, 𝑥)
.616555556
=
.615444444
where
cov(𝑥, 𝑦)
cov(𝑦, 𝑦)
.615444444
.716555556
𝑛
1
cov 𝑥, 𝑦 =
෍(𝑥𝑖 − mean(𝑥))(𝑦𝑖 − mean(𝑦))
𝑛−1
𝑖=1
74
PCA Example –STEP 3
• Calculate the eigenvectors and eigenvalues of the
covariance matrix
 Find eigenvalues 𝜆 = 𝜆1 , 𝜆2 𝑇 of matrix cov:
det(cov-𝜆𝑖 I) = 0
.0490833989
𝜆 =
1.28402771
 Find eigenvectors 𝑣 = (𝑣1 , 𝑣2 ) of cov:
cov*𝑣𝑖 =𝜆𝑖 ∗ 𝑣𝑖
−.735178656
𝑣=
.677873399
−.677873399
−.716555556
75
PCA Example –STEP 4
• order eigenvectors found from eigenvectors by
eigenvalue, highest to lowest.
-.677873399 -.735178656
-.735178656 .677873399
• choose the principle components in order of
significance.
Here, the first principle component :
- .677873399
- .735178656
76
PCA Example –STEP 5
• Deriving the new data
FinalData = RowFeatureVector x RowZeroMeanData
RowFeatureVector is the matrix with the eigenvectors
in the columns transposed so that the eigenvectors
are now in the rows, with the most significant
eigenvector at the top
RowZeroMeanData is the mean-adjusted data
transposed, ie. the data items are in each column,
with each row holding a separate dimension.
77
PCA Example –STEP 5
x
-.827970186
1.77758033
-.992197494
-.274210416
-1.67580142
-.912949103
.0991094375
1.14457216
.438046137
1.22382056
The data after transforming using only the most significant eigenvector
78
79
PCA and Optimization
 𝑋 : 𝑛 × 𝑚 data matrix with rows 𝑥𝑖 ;
 𝑞 : dimension of a subspace into which points are
projected (𝑞 ≤ 𝑚);
 𝑍 : 𝑛 × 𝑞 matrix of scores;
 𝑉 : 𝑚 × 𝑞 matrix comprised of the first 𝑞 columns 𝑣𝑖 ,
𝑖 = 1, … , 𝑞 of the rotation matrix
 We need to find 𝒁, 𝑽 such that
𝑿𝑻 = 𝑽𝒁𝑻
under the constraint 𝑽𝑻 𝑽 = 𝑰 where 𝑰 is the identity
matrix.
80
PCA and Optimization
 Given a data matrix 𝑛 × 𝑚 𝑿 and 𝑞 ≤ 𝑚
 Goal is to minimize the sum of squared distances of points to
their projections in a 𝑞-dimensional subspace.
𝑛
min ෍ 𝑥𝑖 − 𝑉𝑧𝑖 22
𝑉,𝑍
s. t. 𝑉 𝑇 𝑉 = 𝐼
𝑖=1
where 𝐼 is the identity matrix.
• Score 𝑧𝑖 : projection of 𝑥𝑖 in the 𝑞-dimensional subspace,
• Quantity 𝑦𝑖 = 𝑉𝑧𝑖 : projected point in terms of the original
coordinates ⇒ 𝒀 : 𝑛 × 𝑚 matrix of such projected points;
•Columns of 𝑉 define a basis for the subspace.
• An optimal solution is to set 𝑉 to be the loadings for the
first 𝑞 PCs and 𝑍 = 𝑋𝑉
81
Alternative optimization problem
𝑛
min ෍ 𝑥𝑖 − 𝑦𝑖 22
𝑌
𝑖=1
s. t. 𝑟𝑎𝑛𝑘 𝑌 ≤ 𝑞
• In this formulation, we do not decompose the projected
points to find the fitted subspace explicitly.
• Rather, the sum of squared distances of points 𝑥𝑖 to their
projections in terms of the original coordinates 𝑦𝑖 is
minimized while restricting the dimension of the subspace
containing the projections to a value at most q.
• Setting 𝑌 = 𝑋𝑉𝑉 𝑇 where the columns of 𝑉 are the
loadings of first 𝑞 PCs provides an optimal solution to this
problem.
Singular Value Decomposition
(SVD)
83
Singular Value Decomposition (SVD)
 Singular value decomposition (SVD) is a
factorization of a real or complex matrix into
simpler meaningful pieces, used to detect groupings
in data
 It is the generalization of the eigen decomposition
of a positive semidefinite normal matrix (for
example, a symmetric matrix with positive
eigenvalues) to any 𝑚 × 𝑛 matrix via an extension
of polar decomposition.
 It has many useful applications in signal processing
and statistics.
84
Singular Value Decomposition (SVD)
For an n m matrix A of rank r there exists a factorization
(Singular Value Decomposition = SVD) as follows:
A  ULV
𝑛𝑟
𝑟𝑟
T
𝑟𝑚
The columns of U are orthogonal eigenvectors of AAT.
The columns of V are orthogonal eigenvectors of ATA.
Eigenvalues 𝜆1 … 𝜆𝑟 of AAT are the eigenvalues of ATA.
𝜎𝑖 =
𝜆𝑖
𝐿 = diag 𝜎1 . . . 𝜎𝑟
Singular value
85
Let
SVD: Example
 2 2
A


1
1


1 0   2 2
0 1  

  0
Thus 𝑛 = 𝑚 = 2. Its SVD is
0   1 2 1 2


2   1 2 1 2 
Typically, the singular values arranged in
decreasing order.
86
SVD and Optimization
 Given a data matrix 𝑛 × 𝑚 𝐴
 GOAL: find 𝑽, 𝑳, 𝑼 such that
𝑨 = 𝑼𝑳𝑽𝑻
 In PCA, under the constraint 𝑽𝑻 𝑽 = 𝑰 where 𝑰 is the
identity matrix, we need to find 𝒁, 𝑽 such that
𝑿𝑻 = 𝑽𝒁𝑻
• Note that based on SVD we can rewrite the objective
function 𝑥𝑖 − 𝑉𝑧𝑖 22 in PCA as 𝑎𝑖 − 𝑉𝐿𝑢𝑖 22 .
• Replace ℓ2 -norm by 𝑄, 𝑅 norm: 𝐴 𝑄,𝑅 =
tr 𝑄𝐴𝑅𝐴𝑇 .
• Adjusting 𝑄 and 𝑅: control sparsity and smoothness.
87
SVD and Optimization
The Generalized least squares Matrix Decomposition
optimization problem that generalizes SVD:
𝑛
min ෍ 𝑎𝑖 − 𝑉𝐿𝑢𝑖 2𝑄,𝑅
𝑖=1
s. t.
𝑉 𝑇 𝑅𝑉 = 𝐼,
𝑈 𝑇 𝑄𝑈 = 𝐼,
diag 𝐿 ≥ 0
where 𝐼 is the appropriately-sized identity matrix,
𝐴 𝑄,𝑅 =
tr 𝑄𝐴𝑅𝐴𝑇 .