I was reading about Elasticsearch and the engine it’s using for full-text indexing and search, Apache Lucene. The concept of query and filter context intrigued me. It’s a simple concept, really, but I realized I haven’t mentally separated my queries into two categories before:

  • Query: How well does this match?
  • Filter: Does this match?

Query context is all about relevance. A score is calculated for each eligible result and the set is ranked based on the scores.

Filter context, on the other hand, is to get “yes” or “no” answers. It allows the engine to include/exclude results in/from the set.

An example from Elasticsearch:

{
  "query": {
    "bool": {
      "must": [
        { "match": { "title": "distributed systems" } }
      ],
      "should": [
        { "match": { "abstract": "consensus" } }
      ],
      "filter": [
        { "term":  { "status": "published" } },
        { "range": { "published_at": { "gte": "2024-01-01" } } },
        { "terms": { "language": ["en", "de"] } }
      ],
      "must_not": [
        { "term": { "retracted": true } }
      ]
    }
  }
}

Here, bool.must (required) and bool.should (optional) belong to the query context. They both contribute to a score. On the other hand, bool.filter and bool.must_not belong to the filter context. They must be taken at face value. When I take a close look, I also notice a difference in the clauses: match is used for relevance analysis while term is used for exact values. If I loosely translate this example query to everyday language:

We’re looking for an academic paper (document) whose title is relevant to distributed systems and its abstract mentions “consensus” (algorithm). The paper must be published no earlier than Jan 1st, 2024. It cannot be in any language other than English or German. Any retracted paper is excluded from the result set.

To humanize the terms query and filter contexts, I think it’s okay to say transform them as ranking (query) vs matching (filter). While matching (filter) reduces the candidate set, ranking (query) orders the surviving set.