Installation

The package is registered in the General registry and so can be installed at the REPL with ] add StringDistances.

Supported Distances

Distances are defined for AbstractStrings, and any iterator that define length() (e.g. graphemes, AbstractVector...)

The available distances are:

• Edit Distances
  • Jaro Distance Jaro()
  • Levenshtein Distance Levenshtein()
  • Damerau-Levenshtein Distance DamerauLevenshtein()
  • RatcliffObershelp Distance RatcliffObershelp()
• Q-gram distances compare the set of all substrings of length q in each string.
  • QGram Distance Qgram(q::Int)
  • Cosine Distance Cosine(q::Int)
  • Jaccard Distance Jaccard(q::Int)
  • Overlap Distance Overlap(q::Int)
  • Sorensen-Dice Distance SorensenDice(q::Int)
• Distance "modifiers" that can be applied to any distance:
  • Partial returns the minimum of the normalized distance between the shorter string and substrings of the longer string.
  • TokenSort adjusts for differences in word orders by returning the normalized distance of the two strings, after re-ordering words alphabetically.
  • TokenSet adjusts for differences in word orders and word numbers by returning the normalized distance between the intersection of two strings with each string.
  • TokenMax combines the normalized distance, the Partial, TokenSort and TokenSet modifiers, with penalty terms depending on string lengths. This is a good distance to match strings composed of multiple words, like addresses. TokenMax(Levenshtein()) corresponds to the distance defined in fuzzywuzzy
  • Winkler diminishes the normalized distance of strings with common prefixes. The Winkler adjustment was originally defined for the Jaro similarity score but it can be defined for any string distance.

Basic Use

Evaluate

You can always compute a certain distance between two strings using the following syntax:

evaluate(dist, s1, s2)
dist(s1, s2)

For instance, with the Levenshtein distance,

evaluate(Levenshtein(), "martha", "marhta")
Levenshtein()("martha", "marhta")

Compare

The function compare is defined as 1 minus the normalized distance between two strings. It always returns a Float64 between 0 and 1: a value of 0 means completely different and a value of 1 means completely similar.

evaluate(Levenshtein(),  "martha", "martha")
#> 0
compare("martha", "martha", Levenshtein())
#> 1.0

Find

• findnearest returns the value and index of the element in itr with the lowest distance with s. Its syntax is:

  findnearest(s, itr, dist::StringDistance; min_score = 0.0)
  

• findall returns the indices of all elements in itr with a similarity score with s higher than a minimum value (default to 0.8). Its syntax is:

  findall(s, itr, dist::StringDistance; min_score = 0.8)
  

The functions findnearest and findall are particularly optimized for Levenshtein and DamerauLevenshtein distances (as well as their modifications via Partial, TokenSort, TokenSet, or TokenMax).

References

• The stringdist Package for Approximate String Matching Mark P.J. van der Loo
• fuzzywuzzy