3

I have a bunch of company names to match, for example, I want to match this string: A&A PRECISION

with A&A PRECISION ENGINEERING

However, almost every similarity measure I use: like Hamming distance, Levenshtein distance, Restricted Damerau-Levenshtein distance, Full Damerau-Levenshtein distance, Longest Common Substring distance, Q-gram distance, cosine distance, Jaccard distance Jaro, and Jaro-Winkler distance

matches: B&B PRECISION instead.

Any idea which metric would give more emphasis to the preciseness of the substrings and its sequence matched and care less about the length of the string? I think it is because of the length of the string that the metrics would always choose wrongly.

imguessing
  • 377
  • 1
  • 3
  • 9
  • You can check that one is a substring of another. I think you need to use rules, because you seem to be using semantic interpretations of the string -- although Levenshtein distance at the level of words rather than letters would fit this problem. – Gordon Linoff Nov 09 '19 at 17:25

1 Answers1

2

If you really want to "...give more emphasis to the preciseness of the substrings and its sequence...", then this function could work, as it tests wether a string is a substring of another one:

library(data.table)

x <- c("A&A PRECISION", "A&A PRECISION ENGINEERING", "B&B PRECISION")
y <- x

We want to expand the grid. For that I'd use the CJ function in data.table. Then, we will check each pair and see if x is a substring of y (this doesn't work the other way round):

CJ(x, y)[, similarity := apply(.SD, 1, function(x) x[2] %like% x[1]), .SDcols = c("x", "y")][x != y, ]
                           x                         y similarity
1:             A&A PRECISION A&A PRECISION ENGINEERING       TRUE
2:             A&A PRECISION             B&B PRECISION      FALSE
3: A&A PRECISION ENGINEERING             A&A PRECISION      FALSE
4: A&A PRECISION ENGINEERING             B&B PRECISION      FALSE
5:             B&B PRECISION             A&A PRECISION      FALSE
6:             B&B PRECISION A&A PRECISION ENGINEERING      FALSE

Please keep in mind that you'll need to make sure that the strings are as neat as possible for this to work, and even then it might fail.

There are some things I'll check to clean your strings:

  • Remove multiple spaces,
  • Remove spaces at the beginning / end of string
  • Ensure the same encoding
  • Ensure the same CASE

You can achieve that with the stringi package.

PavoDive
  • 6,322
  • 2
  • 29
  • 55
  • I was thinking that you may want the 3rd row in the answer to return `TRUE` too. If that's the case, you can use this code: `CJ(x, y)[, similarity := apply(.SD, 1, function(x) x[2] %like% x[1] || x[1] %like% x[2]), .SDcols = c("x", "y")][x != y, ]` – PavoDive Nov 09 '19 at 18:08
  • Thank you so much for your help! Sorry, I dont really get your code, it is pretty advanced for me. is SD a df? I tried running the above code but getting Error in CJ(x, y) : object 'y' not found. – imguessing Nov 10 '19 at 01:37