minhash_bands
Splits a MinHash signature into bands of `rows_per_band` positions and returns one key per band, as a dynamic array of longs, for LSH bucketing with `mv-expand` and `summarize by`.
Two sets with Jaccard similarity s share at
least one band key with probability 1 - (1 - s^r)^b, where r is
rows_per_band and b the number of bands.
A band key is a candidate bucket, not a cluster: a row lands in b buckets.
rows_per_band must be a positive constant that divides the signature's
length. When binding can see the length (a direct minhash(..., k) call or a
constant array) a length it does not divide is a query error; otherwise it is
an error when the row is evaluated. An empty or null signature, or one with a
position that is not an int or long, returns null.
Syntax
minhash_bands(signature, rows_per_band)Parameters
Prop
Type
Returns: dynamic
Examples
Example 1
print minhash_bands(minhash("the quick brown fox", 8), 2)| print_0 (dynamic) |
|---|
| [-8538493499026222176,-8683894995503639820,-2719021822603344213,-6964628045477729000] |
Example 2
datatable(text:string)[
"error on node 1",
"error on node 2",
"all good"
]
| mv-expand band = minhash_bands(minhash(text, 16), 2) to typeof(long)
| summarize texts = make_set(text) by band
| where array_length(texts) > 1| band (long) | texts (dynamic) |
|---|---|
| 1168030393910279707 | ["error on node 1","error on node 2"] |
minhash
Returns a MinHash signature of a set: a dynamic array of `k` longs whose positions agree between two sets with probability equal to their Jaccard similarity.
minhash_similarity
Returns the fraction of positions at which two MinHash signatures agree: an estimate of the Jaccard similarity of the sets they came from.