warning: uncondensed distance matrix in python

Question

I try to make a Dendrogram Associated for the Agglomerative Hierarchical Clustering and I need the Distance Matrix. I started with:

import numpy as np 
import pandas as pd
from scipy import ndimage 
from scipy.cluster import hierarchy 
from scipy.spatial import distance_matrix 
from matplotlib import pyplot as plt 
from sklearn import manifold, datasets 
from sklearn.cluster import AgglomerativeClustering 
from sklearn.datasets.samples_generator import make_blobs 
%matplotlib inline
X1, y1 = make_blobs(n_samples=50, centers=[[4,4], [-2, -1], [1, 1], [10,4]], cluster_std=0.9)
plt.scatter(X1[:, 0], X1[:, 1], marker='o') 
agglom = AgglomerativeClustering(n_clusters = 4, linkage = 'average')
agglom.fit(X1,y1)
# Create a figure of size 6 inches by 4 inches.
plt.figure(figsize=(6,4))

# These two lines of code are used to scale the data points down,
# Or else the data points will be scattered very far apart.

# Create a minimum and maximum range of X1.
x_min, x_max = np.min(X1, axis=0), np.max(X1, axis=0)

# Get the average distance for X1.
X1 = (X1 - x_min) / (x_max - x_min)

# This loop displays all of the datapoints.
for i in range(X1.shape[0]):
    # Replace the data points with their respective cluster value 
    # (ex. 0) and is color coded with a colormap (plt.cm.spectral)
    plt.text(X1[i, 0], X1[i, 1], str(y1[i]),
             color=plt.cm.nipy_spectral(agglom.labels_[i] / 10.),
             fontdict={'weight': 'bold', 'size': 9})

# Remove the x ticks, y ticks, x and y axis
plt.xticks([])
plt.yticks([])
#plt.axis('off')



# Display the plot of the original data before clustering
plt.scatter(X1[:, 0], X1[:, 1], marker='.')
# Display the plot
plt.show()
dist_matrix = distance_matrix(X1,X1) 
print(dist_matrix)

and I get an error when I write this:

Z = hierarchy.linkage(dist_matrix, 'complete')

/home/jupyterlab/conda/envs/python/lib/python3.6/site-packages/ipykernel_launcher.py:1: ClusterWarning: scipy.cluster: The symmetric non-negative hollow observation matrix looks suspiciously like an uncondensed distance matrix """Entry point for launching an IPython kernel.

First of all, what does that mean and how can I solve it? Thanks

yeah, still, what does it mean? could not find a proper answer yet — AKP, Mar 23 '20 at 16:37

score 5 · Answer 1 · answered Jul 28 '21 at 13:46

scipy.cluster.heirarchy.linkage expects a condensed distance matrix, not a squareform/uncondensed distance matrix. You've calculated a squareform distance matrix, and need to convert it to a condensed form. I suggest using scipy.spatial.distance.squareform. The following snipped reproduces your functionality (I've removed the plotting for brevity) without a warning.

from sklearn.cluster import AgglomerativeClustering 
from sklearn.datasets import make_blobs
from scipy.spatial import distance_matrix
from scipy.cluster import hierarchy
from scipy.spatial.distance import squareform

X1, y1 = make_blobs(n_samples=50, centers=[[4,4],
                                           [-2, -1],
                                           [1, 1],
                                           [10,4]], cluster_std=0.9)

agglom = AgglomerativeClustering(n_clusters = 4, linkage = 'average')
agglom.fit(X1,y1)

dist_matrix = distance_matrix(X1,X1)
print(dist_matrix.shape)
condensed_dist_matrix = squareform(dist_matrix)
print(condensed_dist_matrix.shape)
Z = hierarchy.linkage(condensed_dist_matrix, 'complete')

Can you explain what is the difference between a condensed and an uncondensed distance matrix? — Soerendip, Jul 01 '22 at 13:31
Since a distance matrix is usually computed between a set of points and itself (ie all pairs of elements in the set of points), it is symmetric and the diagonal is all zero. Squareform/condensed saves memory by only representing the upper triangular points, whereas uncondensed is the full matrix. Does this help? — ahagen, Jul 02 '22 at 14:13
Ok, the condensed form is only the upper (or lower) triangle without the diagonal of the distance matrix. You could add two images to make that more clear. Thanks — Soerendip, Jul 06 '22 at 19:43

score 3 · Answer 2 · answered Jun 04 '20 at 08:54

3

It means X1 is too close to X1.T in

agglom.fit(X1,y1)

You can add below code in the header to ignore it!

from scipy.cluster.hierarchy import ClusterWarning
from warnings import simplefilter
simplefilter("ignore", ClusterWarning)

answered Jun 04 '20 at 08:54

blue

109
6

warning: uncondensed distance matrix in python

2 Answers2