Learn more about the Cortex Query Language dedup stage that removes duplicate occurrences of field values.
Syntax
dedup <field1>[,<field2>, ...] by asc | desc <field>
Description
The dedup stage removes all records that contain duplicate values (or duplicate sets of values) from the result set. The record that is returned is identified by the by clause, which selects the record by either the first or last occurance of the field specified in this clause.
Note
The dedup stage can only be used with fields that contain numbers or strings.
Examples
Return unique values for the actor_primary_username field. For any given field value, return the first chronologically occurring record.
dataset = xdr_data | fields actor_primary_username as apu | filter apu != null | dedup apu by asc _time
Return the last chronologically occurring record for any given actor_primary_username value.
dataset = xdr_data | fields actor_primary_username as apu | filter apu != null | dedup apu by desc _time
Return the first occurrence seen by for any given actor_primary_username. field value.
dataset = xdr_data | fields actor_primary_username as apu | filter apu != null | dedup apu by asc apu
Return unique groups of actor_primary_username and os_actor_primary_username field values. For each unique grouping, return the pair that first appears on a record with a non-NULL action_file_size field.
dataset = xdr_data
| fields actor_primary_username as apu,
os_actor_primary_username as oapu,
action_file_size as afs
| filter apu != null and afs != null
| dedup apu, oapu by asc afs