This Julia package computes various distances between strings.

Distances

Edit Distances

• Hamming Distance
• Levenshtein Distance
• Damerau-Levenshtein Distance

Q-Grams Distances

Q-gram distances compare the set of all substrings of length q in each string.

• QGram Distance
• Cosine Distance
• Jaccard Distance
• Overlap Distance
• Sorensen-Dice Distance

Others

• Jaro Distance
• RatcliffObershelp Distance

Syntax

evaluate

The function evaluate returns the litteral distance between two strings (a value of 0 being identical). While some distances are bounded by 1, other distances like Hamming, Levenshtein, Damerau-Levenshtein, Jaccard can be higher than 1.

using StringDistances
evaluate(Hamming(), "martha", "marhta")
#> 2
evaluate(QGram(2), "martha", "marhta")
#> 6

compare

The higher level function compare returns a similarity score between two strings. The similarity score is always between 0 and 1. A value of 0 being completely different and a value of 1 being completely similar. The output of compare is generally 1 - normalized distance, with some care for NaN values.

using StringDistances
compare(Hamming(), "martha", "marhta")
#> 0.6666666666666667
compare(QGram(2), "martha", "marhta")
#> 0.4

Modifiers

The package defines a number of ways to modify similarity scores:

• Winkler boosts the similary score of strings with common prefixes

  compare(Jaro(), "martha", "marhta")
  #> 0.9444444444444445
  compare(Winkler(Jaro()), "martha", "marhta")
  #> 0.9611111111111111
  

  The Winkler adjustment was originally defined for the Jaro similarity score but this package defines it for any string distance.

  compare(QGram(2), "william", "williams")
  #> 0.9230769230769231
  compare(Winkler(QGram(2)), "william", "williams")
  #> 0.9538461538461539
  

• The Python library fuzzywuzzy defines a few modifiers for the RatcliffObershelp similarity score. This package replicates them and extends them to any string distance:

  • Partial returns the maximal similarity score between the shorter string and substrings of the longer string.

    compare(Levenshtein(), "New York Yankees", "Yankees")
    #> 0.4375
    compare(Partial(Levenshtein()), "New York Yankees", "Yankees")
    #> 1.0
    

  • TokenSort adjusts for differences in word orders by reording words alphabetically.

    compare(RatcliffObershelp(), "mariners vs angels", "angels vs mariners")
    #> 0.44444
    compare(TokenSort(RatcliffObershelp()),"mariners vs angels", "angels vs mariners")
    #> 1.0
    

  • TokenSet adjusts for differences in word orders and word numbers by comparing the intersection of two strings with each string.

    compare(Jaro(),"mariners vs angels", "los angeles angels at seattle mariners")
    #> 0.559904
    compare(TokenSet(Jaro()),"mariners vs angels", "los angeles angels at seattle mariners")
    #> 0.944444
    

  • TokenMax combines scores using the base distance, the Partial, TokenSort and TokenSet modifiers, with penalty terms depending on string lengths.

    compare(TokenMax(RatcliffObershelp()),"mariners vs angels", "los angeles angels at seattle mariners")
    #> 0.855
    

• You can compose multiple modifiers:

  compare(Winkler(Partial(Jaro())),"mariners vs angels", "los angeles angels at seattle mariners")
  #> 0.7378917378917379
  compare(TokenSet(Partial(RatcliffObershel())),"mariners vs angels", "los angeles angels at seattle mariners")
  #> 1.0
  

Tips

• Each distance is tailored to a specific problem. Edit distances works well with local spelling errors, the Ratcliff-Obsershelp distance works well with edited texts, the Jaro Winkler distance was invented for short strings such as person names, the QGrams distances works well with strings composed of multiple words and fluctuating orderings.

• Most distances perform poorly when comparing company or individual names, where each string is composed of a few words.

  • While word order is mostly irrelevant in this situation, edit distances heavily penalize different orderings. Instead, either use a distance robust to word order (like QGram distances), or compose a distance with TokenSort, which reorders the words alphabetically.

    compare(RatcliffObershelp(), "mariners vs angels", "angels vs mariners")
    #> 0.44444
    compare(TokenSort(RatcliffObershelp()),"mariners vs angels", "angels vs mariners")
    #> 1.0
    compare(Cosine(3), "mariners vs angels", "angels vs mariners")
    #> 0.8125
    

  • General words (like "bank", "company") may appear in one string but no the other. One solution is to abbreviate these common names to diminish their importance (ie "bank" -> "bk", "company" -> "co"). Another solution is to use the Overlap distance, which compares the number of common qgrams with the length of the shorter strings. Another solution is to use the Partial or TokenSet modifiers.

  TokenMax(RatcliffObershelp()), corresponding to the WRatio function in the Python library fuzzywuzzy, solves these two issues and may work best in this situation.

• Standardize strings before comparing them (lowercase, punctuation, whitespaces, accents, abbreviations...)

References

• The stringdist Package for Approximate String Matching Mark P.J. van der Loo
• fuzzywuzzy blog post