minhash
Returns a MinHash signature of a set: a dynamic array of `k` longs whose positions agree between two sets with probability equal to their Jaccard similarity.
Use it to find near-duplicates: compare signatures with
minhash_similarity(), or bucket them with minhash_bands().
A string's set is its tokens, the maximal runs of ASCII letters, digits and
_, case-sensitive. A dynamic array's set is its elements, null included.
Elements match when they are the same value of the same type: 1 and "1"
differ, as do 1 and 1.0, and an int and a long. A string token and
the same string as an array element hash alike, so minhash("a b", k) equals
minhash(dynamic(["a", "b"]), k).
Null, an empty string, a string with no tokens and an empty array return null:
an empty set has no signature. k must be a constant between 1 and 256: a
literal, a calculation over literals, or toscalar().
Signatures are deterministic across workers, queries and releases with the same semantic version, so they can be stored and compared later.
Syntax
minhash(source, k)Parameters
Prop
Type
Given a dynamic field
This parameter is string-only. In permissive mode — the default — almost every field arrives as a dynamic, so passing one works because Berserk injects asstring: extract-or-null, the value when it really is a string and null otherwise, so the call yields null (a predicate yields false). A property bag or array is therefore not matched, and tostring() is never applied implicitly — use it explicitly to match a bag's JSON text, keys included. Strict mode rejects the dynamic instead of coercing it. To search the strings inside an object or array, write body.* contains "x" (also has, hasprefix and hassuffix). See String coercion and the asXXX family.
Returns: dynamic
Syntax
minhash(source, k)Parameters
Prop
Type
Returns: dynamic
Examples
Example 1
print minhash("the quick brown fox", 8)| print_0 (dynamic) |
|---|
| [3159476979880464507,1429234371446826077,5097220477338837324,3733831081471518593,-4862306691207948953,2756491118856744969,627153400513632819,176151646767040178] |
Example 2
print minhash(dynamic(["a", "b", "c"]), 8)| print_0 (dynamic) |
|---|
| [3508827723414499431,-6178191346546239874,3281409427696077258,3428563401114183193,1904788514007754420,1208974314325466741,4104173205064672066,7726621793539914212] |