# Introduction As part of our experimentation with methods for entity resolution tasks (see our publication ["Entity Resolution: Learned Representations of Tabular Data with Classic Neural Networks"](https://app.readytensor.ai/publications/entity-resolution-learned-representations-of-tabular-data-with-classic-neural-networks-MtUrsAPP6Mdt )) this experiment attempts to solve an entity resolution problem based on blocking rules and a KMeans clustering algorithm. The project is an experimental prototype for research and does not address full error handling or production standards, and it's rather a setup for quick prototyping. We build a hierarchical directed graph where each level corresponds to nodes generated from a blocking rule/s. This approach reduces all dataset record comparisons which is an O(n$$^2$$) to O(n) thus reducing the computational requirements for downstream processing since we would only be comparing records within components; the approach helps us present a reasonable scalable solution. Each node in a graph level groups co-occurring records based on a rule. Hierarchical rules applied to these nodes yield children nodes at deeper graph levels. Records in nodes can be grouped together in components subject to a minimum number of co-ocurrences within the graph. The resulting records in a component are broken down into clusters applying KMeans, referred herein and existing [literature](https://arxiv.org/pdf/1810.05497) as KLSH. * **KLSH:** We use a K-Means algorithm to cluster records where each record belongs to the same entity. K-means is helpful in that it can place records that are similar in the same bucket or cluster but apart from those that are different; the approach treats each cluster as a bucket mimicking LSH though it doesn't use hashing. We apply weights and standardize features which are inputs to Kmeans to calculate the square ecludian distances to centroids and cluster records. We conclude the results are encouraging and we identify multiple further research opportunities to improve our results. The code for this repo is available in the [Github repo](https://github.com/sergiosolorzano/entity_resolution/block_klsh). :::youtube[Title]{#xp78pBUlAHU}--DIVIDER--# Dataset The dataset has been synthetically generated by looking at different brands of pianos and selecting features that require normalization with different methods, including numeric, dates and ordinal and boolean categorical. Typos for the features of the dataset have been added to challenge the approach. Taking a supervised approach, we compare the resulting entity components to the expected results we had in mind when creating the dataset. Since the proposed solution does not rely on training a model, the dataset is small and holds twenty records, sufficient for an initial indication of the potential of this approach. Sample Dataset: | Name | Tension_Adj | Tension | Resonance | Longevity | Quality | Amt_Sold | |-------------------|-------------|------------|------------|-------------|---------|----------| | Apollo | 1 | 3.051514 | 110.993063 | 2028-04-13 | 0 | 5000 | | Apolo | 1 | 3.500000 | 108.500000 | 2028-03-13 | 1 | 5000 | | Apo | 1 | 3.051514 | 109.250000 | 2028-04-13 | 0 | 5000 | | Apol | 1 | 3.050000 | 11.090000 | 2029-04-13 | 0 | 4000 | | Apollo | 1 | 3.500000 | 109.750000 | 2028-04-03 | 0 | 4500 | | Apollorium | 0 | 6.250000 | 92.000000 | 2033-08-24 | 1 | 2500 | | Apollo | 0 | 2.627367 | 96.071012 | 2025-09-24 | 1 | 3000 | | Apollo | 0 | 2.627367 | 99.000000 | 2025-08-02 | 1 | 3000 | | Apollo | 0 | 2.327367 | 95.000000 | 2025-08-24 | 1 | 3000 | | Apollo | 0 | 2.600000 | 95.500000 | 2025-08-24 | 1 | 3000 | | August FÖrster | 0 | 11.670953 | 119.348834 | 2030-12-09 | 3 | 5000 | | August Forster | 1 | 11.600000 | 117.000000 | 2030-12-08 | 3 | 5000 | | August F | 1 | 116.000000 | 118.000000 | 2030-11-09 | 3 | 5000 | | August FÖrster | 0 | 1.600000 | 119.000000 | 2030-11-08 | 2 | 5000 | | August Forster | 0 | 6.000000 | 105.750000 | 2025-11-03 | 3 | 5000 | | Agust FÖrster | 0 | 12.670953 | 121.000000 | 2029-12-09 | 3 | 3000 |--DIVIDER--# Methodology ## Expected results: Our labelled dataset has X true entities: - Component 0 (see dataset row related to name Apollo): We have created the dataset where two sub-groups should be created for these records containing multiple records, and a third - record 5 - a singleton. We deem record 3 part of a component but contains multiple typos. Most records data include to assess the robustness of the approach. ```python ground_truth_pairs_component_0 = [(0, 1), (0, 2), (0, 3), (0, 4), (1, 2), (1, 3), (1, 4), (2, 3), (2, 4), (3, 4), (6, 7), (6, 8), (6, 9), (7, 8), (7, 9), (8, 9)] singleton = [5] ``` - Component 1 (see dataset row related to name August): We have created the dataset where all records except one - record 14 - belong to the same group. Record contain multiple typos. ```python ground_truth_pairs_component_1 = true_pairs_component_1 = [(10, 11), (10, 12), (10, 13), (10, 15), (11, 12), (11, 13), (11, 15), (12, 13), (12, 15), (13, 15)] singleton = [14] ``` - Component 2 (see dataset row related to name Baldwin): We have created the dataset where all in this component should be grouped together. Record contain multiple typos. ```python ground_truth_pairs_component_2 = true_pairs_component_1 = [(16, 17), (16, 18), (16, 19), (16, 20), (17, 18), (17, 19), (17, 20), (18, 19), (18, 20), (19, 20)] singleton = [ ] ```--DIVIDER--##Methodology ### Step 1 - Meta-Blocking #### Block Generation We create blocks with a relaxed phonetic rule applied to the dataset's piano model names. Records are clustered in the nodes where they match a rule - the first char, first two char, the first three char, last three char or has all consonants. Rules are easily extendable and can be arranged to generate blocks in a hierarchical or flat order. For example, we can block at the first graph hierarchical level for the rule phonetic_combination which generates flat nodes for multiple sub-rules ("first_char", "first_two_char", "first_three_char", "last_three_char", "consonants"), and generate the second level graph nodes from the first for the adaptive_quantile_binning rule. Blocking rules are easily extendable to experiment, adopt new business rules or edge cases: ```json { "rules": { "phonetic_combination": { "type": "text", "params": { "algorithms": [ "first_char", "first_two_char", "first_three_char", "last_three_char", "consonants" ] }, "default": false, "description": "multi-method name blocking" }, "adaptive_quantile_binning": { "type": "numeric", "params": { "n_bins": 10, "KBinsDiscretizer_encode": "ordinal", "KBinsDiscretizer_binning_method": "quantile" }, "default": true, "description": "RobustScaler Quantile-KBinsDiscretizer on data for bins with ~equal num samples in each bin" } } } ``` ```python def _phonetic_combination(self, feature_series: pd.Series, method_params:dict, feature_name:str) -> pd.Series: def encode_combo(x): key_list = [] if "first_char" in method_params["algorithms"]: key_list.append(str(x).lower()[0:1]) if "first_two_char" in method_params["algorithms"]: key_list.append(str(x).lower()[0:2]) if "first_three_char" in method_params["algorithms"]: key_list.append(str(x).lower()[0:3]) if "last_three_char" in method_params["algorithms"]: key_list.append(str(x).lower()[-3:]) if "consonants" in method_params["algorithms"]: key_list.append(''.join([c for c in str(x).lower() if c.isalpha() and c not in 'aeiou'])) return key_list return feature_series.apply(encode_combo) ``` This is an example of a created block that generates a block for the first three letters: ``` Block Key initial_block-name_phonetic_combination:ap Block Hierarchy Level: 1 Block Feature: name Block Parent Key: initial_block Block Key: initial_block-name_phonetic_combination:ap Leaf Key: phonetic_combination:ap Block Rule: phonetic_combination Block Record Indices: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] ``` This tree graph shows where each record lands: ![graph_tree.png](graph_tree.png) We will not use this graph in this experiment, but as an example, this is how a two level graph with a second binning rule may look like: ![sample_twolevel_graph_tree.png](sample_twolevel_graph_tree.png)--DIVIDER--#### Graph Components Each time records co-ocurr for a rule they'll be together in a node (e.g. as per a blocking rule, they both have the first three letters in the name). If they match on multiple rules they'll co-ocur on multiple nodes. Therefore records can have multiple connections to other records as they co-ocur in multiple nodes. Each co-ocurrence between records represents an edge. We track block provenance so that each time two records co-occur in the same block, their edge weight increases by 1. Records whose edge weight is above a certain threshold form a graph component. Similarly, very different names will not co-ocur in any nodes with other records (Apollo and Baldwin), and thus these create their own graph component. The numbers on the edge shown below tells us how many times records co-ocur in the graph. ![preprun_components_graph.png](preprun_components_graph.png) #### Pruned Components We can assume that two records co-ocurring once in all nodes is not sufficient to help us assume two records are the same and to be in the same component. We set the number of required edges to be >=2 for records to be a component. ![Pruned_components_graph.png](Pruned_components_graph.png) This leaves us with the two Apollo entities clustered into a first component which needs to be resolved. August Forster and Baldwin are correctly separated into two components. This reduces the comparison space or number of record pairs to compare while still maintaining high recall of potential matches. Additional rules can be added to block further, but the discrete approach to blockinig may overfit. Here, there may be **scope for further research** that may yield interesting results by selecting blocking rules, their hierarchical order and alter rule values (e.g. sampling such as np.linspace, np.logspace, etc) with bayesian optimization combined with a transitive approach.--DIVIDER--### Step 2 - Kmeans Locality Sensitive Hashing (KLSH) Recap: We initially broke out records into components by taking a blocking approach that operates in discrete space and by generating sparse blocks. We do not block further at the risk of overfitting. Instead, for each component, we generate dense clusters using the [k-means algorithm](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html) which operates on standardized numerical features in continuous space, and to map similar items to the same bucket based on minimizing distance to cluster centroids. This method helps us separate the Apollo records into the two separate entities. #### Feature Standardization: - Numeric: StandardScaler - Ordinal and boolean categorical, dates: quarter circle encoding. This helps us retain a distance measure for each record feature's value and it's more adaptable to scale it by feature weight. #### K-means clustering: Standardized Numeric Data **Let's analyze Component Zero** We standardize numeric data to be used by Kmeans for the first component with record indices {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}. We run K-means for multiple K clusters: ```python == Final Clusters: == k=1: {0: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]} k=2: {0: [3], 1: [0, 1, 2, 4, 5, 6, 7, 8, 9]} k=3: {0: [3], 1: [5, 6, 7, 8, 9], 2: [0, 1, 2, 4]} k=4: {0: [3], 1: [6, 7, 8, 9], 2: [0, 1, 2, 4], 3: [5]} k=5: {0: [3], 1: [6, 7, 8, 9], 2: [0, 1, 2], 3: [5], 4: [4]} k=6: {0: [3], 1: [6, 7, 8, 9], 2: [0, 2], 3: [5], 4: [4], 5: [1]} k=7: {0: [3], 1: [6, 7, 9], 2: [0, 2], 3: [5], 4: [4], 5: [1], 6: [8]} k=8: {0: [3], 1: [7], 2: [0, 2], 3: [5], 4: [4], 5: [1], 6: [8], 7: [6, 9]} k=9: {0: [3], 1: [7], 2: [0], 3: [5], 4: [4], 5: [1], 6: [8], 7: [6, 9], 8: [2]} k=10: {0: [3], 1: [7], 2: [0], 3: [5], 4: [4], 5: [1], 6: [8], 7: [9], 8: [2], 9: [6]} ``` A visual interpretation of the Elbow method suggests K=4; the [KneeLocator](https://kneed.readthedocs.io/en/stable/parameters.html) library suggests K=1 or 4 (depending on omitting curve and direction) ![klsh_elbow_component0.png](klsh_elbow_component0.png) Sklearn's [Silhouette](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.silhouette_score.html) method suggests K=4. Best F-1 suggests K=4. | K | F1 | Precision | Recall | |----|----------|-----------|--------| | 1 | 0.524590 | 0.355556 | 1.0000 | | 2 | 0.461538 | 0.333333 | 0.7500 | | 3 | 0.750000 | 0.750000 | 0.7500 | | 4 | 0.857143 | 1.000000 | 0.7500 | | 5 | 0.720000 | 1.000000 | 0.5625 | | 6 | 0.608696 | 1.000000 | 0.4375 | | 7 | 0.400000 | 1.000000 | 0.2500 | | 8 | 0.222222 | 1.000000 | 0.1250 | | 9 | 0.117647 | 1.000000 | 0.0625 | | 10 | 0.000000 | 0.000000 | 0.0000 | Thus, we choose the highest F-1 KMeans best result, k=4; the algorithm incorrectly classifies record 3 as a singleton: ```python k=4: {0: [3], 1: [6, 7, 8, 9], 2: [0, 1, 2, 4], 3: [5]} ``` ![klsh_entities_graph_k_4_comp_0.png](klsh_entities_graph_k_4_comp_0.png)--DIVIDER--We run the equivalent predictions for Component 1 and 2 with the following results: **Component 1:** ```python == Final Clusters: == k=1: {0: [10, 11, 12, 13, 14, 15]} k=2: {0: [11, 12, 14], 1: [10, 13, 15]} k=3: {0: [12], 1: [15], 2: [10, 11, 13, 14]} k=4: {0: [12], 1: [15], 2: [10, 11, 13], 3: [14]} k=5: {0: [12], 1: [15], 2: [10, 13], 3: [14], 4: [11]} k=6: {0: [12], 1: [15], 2: [10], 3: [14], 4: [11], 5: [13]} Singletons: [] ``` | K | F1 | Precision | Recall | |----|--------:|----------:|-------:| | 1 | 0.800000 | 0.666667 | 1.0000 | | 2 | 0.500000 | 0.666667 | 0.4000 | | 3 | 0.375000 | 0.500000 | 0.3000 | | 4 | 0.461538 | 1.000000 | 0.3000 | | 5 | 0.181818 | 1.000000 | 0.1000 | | 6 | 0.000000 | 0.000000 | 0.0000 | Elbow K is hard to tell, suggested [KneeLocator](https://kneed.readthedocs.io/en/stable/parameters.html) K=1 or 4. ![klsh_elbow_component1.png](klsh_elbow_component1.png) Sklearn's [Silhouette](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.silhouette_score.html) method suggests K=4. Best F-1 suggestion K=1. We choose the highest F-1 at K=1 as a possible best solution, which incorrectly classifies record 14. ```python k=1: {0: [10, 11, 12, 13, 14, 15]} ```--DIVIDER--**Component 3:** ```python == Final Clusters: == k=1: {0: [16, 17, 18, 19, 20]} k=2: {0: [17], 1: [16, 18, 19, 20]} k=3: {0: [17], 1: [20], 2: [16, 18, 19]} k=4: {0: [17], 1: [20], 2: [16, 18], 3: [19]} k=5: {0: [17], 1: [20], 2: [16], 3: [19], 4: [18]} Singletons: [] ``` | K | F1 | Precision | Recall | |----|---------:|----------:|-------:| | 1 | 1.000000 | 1.000000 | 1.0000 | | 2 | 0.750000 | 1.000000 | 0.6000 | | 3 | 0.461538 | 1.000000 | 0.3000 | | 4 | 0.181818 | 1.000000 | 0.1000 | | 5 | 0.000000 | 0.000000 | 0.0000 | Elbow K suggested by [KneeLocator](https://kneed.readthedocs.io/en/stable/parameters.html) K=1 or 4; visually k=4. ![klsh_elbow_component2.png](klsh_elbow_component2.png) Sklearn's [Silhouette](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.silhouette_score.html) method suggests K=4. Best F-1 suggestion K=1. We choose the highest F-1 at K=1 as a possible best solution, which correctly classifies all records. We conclude - Component 0: Incorrectly classifies record 3 as a singleton. - Component 1: Incorrectly groups record 14. - Component 2: Correctly groups all records when choosing best F-1--DIVIDER--### Step 3 - K-means clustering: Weighting Standardized Numeric Data We will now seek to improve classification. So far we have applied a uniform weight to all numeric features. Next, we explore custom weights for each feature using supervised Bayesian Optimization across all K over 100 calls, with objective function targeting the average of all components' results F1=1. **Further research** is warranted to explore the design of an objective function targeting desired outcome such as F1=Precision=Recall=1 and biasing the optimization towards the preferred domain requirements. In the scenario we have run, Bayesian plateaus and provides multiple - 56 - best weights for the highest best results. Thus we average these. We also note that a weight observed multiple times at zero can be a hint that a particular feature is not adding much to model training and perhaps can be dropped. In this publication, we do not remove a feature. The best Baysian optimization results has the following feature weights and metrics: Average F1 0.9523 Average Precision 1 Average Recall 0.9166 ```json Average Bayesian Feature Weights = [0.32922948, 0.36130566, 0.20008195, 0.82066852, 0.44855293, 0.62657605, 0.36378109, 0.4405338, 0.2413675] ``` We re-run KLSH with these weights and for each Component we select the cluster results for the earliest highest F-1 in the dataframe. **Component 0: Incorrect at K=4** Best silhouette K = 2; Elbow Knee K = 1 or 3, Visual Elbow k=3 or 4; F1=4 ![klsh_elbow_component0_weight.png](klsh_elbow_component0_weight.png) ```json == Final Clusters: == k=1: {0: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]} k=2: {0: [3], 1: [0, 1, 2, 4, 5, 6, 7, 8, 9]} k=3: {0: [3], 1: [5, 6, 7, 8, 9], 2: [0, 1, 2, 4]} k=4: {0: [3], 1: [6, 7, 8, 9], 2: [0, 1, 2, 4], 3: [5]} k=5: {0: [3], 1: [7], 2: [0, 1, 2, 4], 3: [5], 4: [6, 8, 9]} k=6: {0: [3], 1: [7], 2: [0, 2], 3: [5], 4: [6, 8, 9], 5: [1, 4]} k=7: {0: [3], 1: [7], 2: [0, 2], 3: [5], 4: [6, 8, 9], 5: [1], 6: [4]} k=8: {0: [3], 1: [7], 2: [0, 2], 3: [5], 4: [4], 5: [6, 9], 6: [1], 7: [8]} k=9: {0: [3], 1: [7], 2: [2], 3: [5], 4: [4], 5: [6, 9], 6: [1], 7: [8], 8: [0]} k=10: {0: [3], 1: [7], 2: [2], 3: [5], 4: [4], 5: [6], 6: [1], 7: [8], 8: [0], 9: [9]} ``` | K | F1 | Precision | Recall | |----|---------:|----------:|-------:| | 1 | 0.524590 | 0.355556 | 1.0000 | | 2 | 0.461538 | 0.333333 | 0.7500 | | 3 | 0.750000 | 0.750000 | 0.7500 | | 4 | 0.857143 | 1.000000 | 0.7500 | | 5 | 0.720000 | 1.000000 | 0.5625 | | 6 | 0.476190 | 1.000000 | 0.3125 | | 7 | 0.400000 | 1.000000 | 0.2500 | | 8 | 0.222222 | 1.000000 | 0.1250 | | 9 | 0.117647 | 1.000000 | 0.0625 | | 10 | 0.000000 | 0.000000 | 0.0000 | **Component 1: Correct at K=2** Best silhouette K = 2; Elbow Knee K = 1 or 2, Visual Elbow k=2; F1=2 ![klsh_elbow_component1_weight.png](klsh_elbow_component1_weight.png) ```json == Final Clusters: == k=1: {0: [10, 11, 12, 13, 14, 15]} k=2: {0: [10, 11, 12, 13, 15], 1: [14]} k=3: {0: [11, 12], 1: [14], 2: [10, 13, 15]} k=4: {0: [11, 12], 1: [14], 2: [10, 13], 3: [15]} k=5: {0: [12], 1: [14], 2: [10, 13], 3: [15], 4: [11]} k=6: {0: [12], 1: [14], 2: [10], 3: [15], 4: [11], 5: [13]} ``` | K | F1 | Precision | Recall | |----|---------:|----------:|-------:| | 1 | 0.800000 | 0.666667 | 1.0 | | 2 | 1.000000 | 1.000000 | 1.0 | | 3 | 0.571429 | 1.000000 | 0.4 | | 4 | 0.333333 | 1.000000 | 0.2 | | 5 | 0.181818 | 1.000000 | 0.1 | | 6 | 0.000000 | 0.000000 | 0.0 | **Component 2: Correct at K=1** Best silhouette K = 2; Elbow Knee K = 1 or 2, Visual Elbow k=2; F1=1 ![klsh_elbow_component2_weight.png](klsh_elbow_component2_weight.png) ```json == Final Clusters: == k=1: {0: [16, 17, 18, 19, 20]} k=2: {0: [17], 1: [16, 18, 19, 20]} k=3: {0: [17], 1: [16, 19, 20], 2: [18]} k=4: {0: [17], 1: [16, 20], 2: [18], 3: [19]} k=5: {0: [17], 1: [20], 2: [18], 3: [19], 4: [16]} ``` | K | F1 | Precision | Recall | |----|---------:|----------:|-------:| | 1 | 1.000000 | 1.0 | 1.0 | | 2 | 0.750000 | 1.0 | 0.6 | | 3 | 0.461538 | 1.0 | 0.3 | | 4 | 0.181818 | 1.0 | 0.1 | | 5 | 0.000000 | 0.0 | 0.0 | We conclude: - The silhouette score is significantly more robust for these runs though not necessarily more accurate. - Component 0 incorrectly classifies record 3. The remaining records are correctly grouped. It is fair to say record 3's ground truth may have also been a singleton with the assumptions we made, perhaps we were too aggressive. - Component 1 Correctly classifies groups and singleton. - Component 2: correctly classifies all records in the same group.--DIVIDER--### Step 4 - Cosine Similarity Overlay to KLSH results **Further research** may benefit this analysis by creating an additional clustering filter based on cosine similarity which removes the influence of magnitude, unlike KMeans ecludian distance. DISADVANTAGE PICK THRESHOLD As shown on the heat map below which shows the cosine similarity for records in component 0, we may have been too aggressive adding typos to the dataset features for record 3 and it may deserve to be an entity by itself and indeed unrelated to records 0,1,2,4. ![heatmap_cossim_compo_0.png](heatmap_cossim_compo_0.png)--DIVIDER--#### Results We conclude that post feature weight optimization, the training set exhibits the following performance metrics: | Component | F1 | Precision | Recall | |-----------|----------|-----------|------------| | 0 | 0.857143 | 1.000000 | 0.750000 | | 1 | 1.000000 | 1.000000 | 1.000000 | | 2 | 1.000000 | 1.000000 | 1.000000 | | Average | 0.952381 | 1.000000 | 0.916667 | Generalization of these results with a larger training dataset and an evaluation dataset may surface considerations not addressed and yield better results.--DIVIDER--## Conclusion and Discussion The resulting F-1/Precision/Recall=0.952/1.00/0.916 is a good start and domain input on blocking rules and feature engineering and standardization can greatly increase these results. A blocking graph approach helps us scale for larger datasets by reducing all record comparisons which is an O(n$$^2$$) to O(n). Smaller datasets may benefit from skipping blocking all together and creating embeddings for those features, e.g. in the case of this publication, a phonetic embedding for record name. The blocking rules approach can be easily expanded and capture nuances specific to the domain with a deeper graph. The pre-processing techniques applied to a dataset, distance measure and clustering algorithm can have a significant impact on the final clustring results. In this project, we found that weights helped us correctly classify one of the two incorrectly classified components. In addition, further distance conditions in pair selection may also help capture outliers and new entities. This provides an interesting opportunity to create ensembles and diversify the way we capture nuances in the dataset. Blocking rule refinement and the optimization of the order how these are applied can generate more accurate component composition results and help in finding singletons pre-KLSH clustering. Elbow and silhouette unsupervised k-determination is difficult. For instance, the K determined by the [KneeLocator](https://kneed.readthedocs.io/en/stable/parameters.html) library requires the user to run the algorithm having previously established the type of curve (convex, concave) and its slope. Computer vision solutions can help us determine the value for these parameters but we find that the user's visual comparison of these metrics and F-1 helps reach a more robust k-determination. We often find that a production pipeline requires the program to determine K without user intervention. There may be scope for **further research** to achieve this by training a classifier such as [XGBoost](https://xgboost.readthedocs.io/en/release_3.0.0/), and using an X-training_dataset composed of the record features, Elbow KneeLocator K result, silhouette score K result, and a desired F-1/Precision/Recall target; the Y-training_dataset the ground truth K. The resulting bayesian optimized weights gives us some insight into the average importance placed by the model across features, components and clusters. We observe resonance and longevity_sin drive a great deal of the result in this analysis whilst tension does not. Domain knowledge may bias these results or justify its meaning. | Feature | Weight | |--------------------|------------| | tension_adj_cos | 0.32922948 | | tension_adj_sin | 0.36130566 | | tension | 0.20008195 | | resonance | 0.82066852 | | longevity_cos | 0.44855293 | | longevity_sin | 0.62657605 | | quality_cos | 0.36378109 | | quality_sin | 0.44053380 | | amt_sold | 0.24136750 | There may be scope for **further research** to further increase the interpretability of our results. On possible approach may entail training a supervised classifier e.g. random forest with x_training=record_norm_features and y=record_cluster. [feature_importances_](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier.feature_importances_) can yield insight onto the drivers for our results since it give us the statistical accumulation of the impurity decrease within each tree. Alternatively, the analysis of the distribution for each feature in each cluster may shed some light how these also affect clustering.--DIVIDER--## Further Research The list below outlines potential further research that was mentioned in this article and that we may enhance once the core methodologies have been completed. - Enhancement to Entity Resolution with KLSH publication: Replacement of KLSH for Hierarchical clustering - Enhancement to Entity Resolution with KLSH publication: Adding record name phonetic encoding and its vectorization or one hot encoding to KLSH feature inputs - Blocking hierarchical order optimization via bayesian optimization: Instead of banking on a single blocking phonetic rule to block, we add additional blocking rules (e.g. numeric bins, etc) to generate a graph. We run bayesian optimization to select the hierarchical order of rules, alter rule values (e.g. sampling such as np.linspace, np.logspace, etc) and combine it with a transitive approach. - Application of multiple similarity methods as an ensemble: Application of different pre-processing methods to run similarity measures including cosine similarity and dot product to capture both angle and magnitude. We take a supervised learning approach using ground truth to determine the feature optimal weights. - Unsupervised k-determination classifier: Train a classifier such as [XGBoost](https://xgboost.readthedocs.io/en/release_3.0.0/), and using X-training_dataset composed of the record features, Elbow KneeLocator K result, silhouette score K result, and a desired F-1/Precision/Recall target; the Y-training_dataset the ground truth K. - KLSH Interpretability - Supervised Learning: Train a supervised classifier e.g. random forest with x_training=record_norm_features and y=record_cluster. [feature_importances_](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier.feature_importances_) can yield insight onto the drivers for our results since it give us the statistical accumulation of the impurity decrease within each tree. - KLSH Interpretability: A violin plot can shed light on the distribution for each feature in each cluster and thus the importance each feature plays on the final clustering classification.--DIVIDER--# References - [LSH Explained](https://medium.com/@hbrylkowski/locality-sensitive-hashing-explained-304eb39291e4) - [Numbers, categories, locations, images, text. How to embed the world?](https://www.youtube.com/watch?app=desktop&v=z75LcE4kcfs&t=0s) - Research [Is Cosine-Similarity of Embeddings Really About Similarity?](https://arxiv.org/abs/2403.05440) - [Use of more algos on phonetic decreases missing matches and increases precision by reducing false matches that might occur with just one algorithm](https://essay.utwente.nl/96658/1/Meijerman_MA_EEMCS.pdf) - [Medium article with approach to setup bayes skopt.callbacks Custom Early Stopper](https://medium.com/sitechassethealthcenter/gaussian-process-to-optimize-hyperparameters-of-an-algorithm-5b4810277527) - ["Roughly Everything You Need to Know About Entity Resolution"](https://faingezicht.com/articles/2024/09/03/entity-resolution/) - [Rebecca Steorts presentation: (Almost) all of entity resolution](https://www.youtube.com/watch?v=hKVUoybhepA) and [research paper](https://arxiv.org/pdf/1810.05497)