System design / Search & storage

Choosing a Search Engine: Elasticsearch vs Postgres vs Redis vs OpenSearch, Solr and Typesense

How to choose a search engine in system design: what a separate search cluster really costs, real full-text search in Postgres with tsvector, GIN and pg_trgm, Redis Search and MongoDB Search, the Lucene family (Elasticsearch, OpenSearch, Solr), Typesense, Meilisearch and Vespa, and a decision flow you can defend in an interview.

On this page
  1. The real question: one more system, or not?
  2. Postgres full-text search: the one you already have
  3. Redis Search and MongoDB Search: search inside the store you run (members only)
  4. The Lucene family: Elasticsearch, OpenSearch, Solr (members only)
  5. A short note on licences (members only)
  6. Search-first engines: Typesense, Meilisearch, Vespa (members only)
  7. When nothing off the shelf fits (members only)
  8. The decision table and flow (members only)

Your Postgres database already ships with a real full-text search engine, built on the same inverted index idea as Lucene. Most teams never try it before adding a whole new cluster. This blog is about that choice: when the search you already have is enough, and which engine to reach for when it isn't.

Where this fits: Part of the Search & storage series. The journey of a query has a box called "search cluster". This blog asks whether you need a separate one at all, and if so, which.

Before you start: How search engines work (inverted index, analyzers, BM25) is required. Elasticsearch shows what you take on when you add a cluster, and Types of search gives the feature list we compare against.

After reading this you will be able to

  • List what a separate search engine costs you, and what it buys.
  • Build real full-text search in Postgres with tsvector, a GIN index and ts_rank, and say where it stops scaling.
  • Explain why "search built into my database" is not always consistent with that database.
  • Place Elasticsearch, OpenSearch, Solr, Typesense, Meilisearch and Vespa on one map.
  • Walk an interviewer through a short decision flow and defend your default.

For this blog, we compare what each engine is, how it indexes, and what it costs you to run. There are no benchmarks here. Speed depends on your data and hardware, and vendor numbers are marketing until you test them yourself.

01The real question: one more system, or not?

You don't buy a second fridge because you bought more milk. You first check whether the one you have has room. Search engines are the same.

A separate search engine costs you more than its server bill:

  • A second copy of your data, which can drift from the first.
  • A sync pipeline (an outbox or CDC feed) to keep that copy fresh. We built one in Elasticsearch for system design.
  • Lag between "saved" and "searchable".
  • Another thing to page someone at 3 a.m.

In return, you get richer relevance tuning, typo tolerance, facets, scale-out across many machines, and heavy search traffic kept off your primary database. Those are real wins, when you need them.

The default

Start with the search your database already has. Move to a dedicated engine when a concrete need appears: a feature you can't build, a query you can't make fast, or search load that hurts your primary. "We might need it someday" isn't one.

02Postgres full-text search: the one you already have

Postgres has had full-text search built in for years. The introduction to its text search chapter names the pieces:

  • tsvector: a document turned into normalised words (lexemes) with their positions. "Running shoes" becomes 'run':1 'shoe':2.
  • tsquery: the parsed search query, with AND, OR and NOT.
  • @@: the match operator. "Does this tsvector match this tsquery?"
  • Dictionaries, including Snowball stemmers: Postgres's version of analyzers, deciding that "running" and "run" are the same word.

The index is the part that matters. Postgres's page on preferred index types for text search says GIN is the preferred choice, because it's an inverted index with one entry per lexeme. GIN literally stands for Generalized Inverted Index. The alternative, GiST, is lossy: it can return false matches, so Postgres must recheck each candidate row. Here's the whole thing in a few lines:

-- One column that holds the analysed text, kept up to date by Postgres
ALTER TABLE products ADD COLUMN search tsvector
  GENERATED ALWAYS AS (
    to_tsvector('english', coalesce(title, '') || ' ' || coalesce(body, ''))
  ) STORED;

-- The inverted index
CREATE INDEX products_search_idx ON products USING GIN (search);

-- Match with @@, rank with ts_rank
SELECT id, title, ts_rank(search, q) AS rank
FROM products, websearch_to_tsquery('english', 'running shoes') AS q
WHERE search @@ q
ORDER BY rank DESC
LIMIT 10;

It's the same pipeline as Lucene, inside your database, as the diagram shows: analyse the text, invert it, match, rank.

Fig. 1

Postgres full-text search: analyse, invert, match, rank

Postgres full-text search: analyse, invert, match, rankIndex time, top row: a row with title and body goes through to_tsvector, which runs the parser, dictionaries and stemmer, producing a tsvector column. The GIN index maps each lexeme to the rows that contain it. Query time, bottom row: the user query goes through to_tsquery, the @@ operator matches it using the GIN index, ts_rank orders the matches, and the top results are returned.Index timeQuery timelookupRowtitle, bodyto_tsvectorparser + dictionaries+ stemmertsvector'run':1 'shoe':2GIN indexlexeme -> rowsUser queryrunning shoesto_tsquery'run' & 'shoe'@@ matchvia GINts_rankorderTop 10Same pipeline as Lucene, inside your database
Analyse, invert, match, rank: the Lucene pipeline, inside Postgres. The tsvector is the analysed document and the GIN index is the inverted index.

Where it hurts. Ranking. The text search controls page warns that ts_rank and ts_rank_cd can be expensive, because they read the tsvector of every matching document. Finding 50 matches is cheap; ranking 2 million matches to show the top 10 is not. Also, the built-in ranking isn't BM25, typo tolerance is limited, there are no facets out of the box, and scaling out across machines is your problem.

Substring search. For "find '%shoe%' anywhere in a name" or fuzzy-ish name matching, the pg_trgm extension indexes three-letter slices (trigrams) with GIN or GiST. Those indexes support LIKE, ILIKE and regex matches, and unlike a B-tree, the search string need not be anchored at the start.

CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE INDEX products_title_trgm ON products USING GIN (title gin_trgm_ops);
-- Now this can use the index instead of scanning every row:
SELECT id, title FROM products WHERE title ILIKE '%shoe%';

The strengths are big. It's transactional: the row and its index change together, so there's no sync pipeline and no lag. You can join search results with the rest of your data in one query. And it's one system to run, not two.

A real example: Instacart moved its full-text search to sharded Postgres, using GIN indexes and a modified ts_rank, and later added pgvector for hybrid search in the same engine. They report a 10x drop in write workload, because a normalised schema meant far fewer partial updates than their denormalised search index needed.

Seen this before?

GIN is the inverted index from How search engines work, in a Postgres costume. Lucene, Postgres GIN and Redis Search all share it.

Members-only article

Keep reading with the Article Pass

You've read the free part. The other 6 sections of this guide, with every diagram and worked example, are for members. One Article Pass unlocks every members-only article on Engineering Atlas: the ones live today and every one I publish while your pass is active.

  • Redis Search and MongoDB Search: search inside the store you run
  • The Lucene family: Elasticsearch, OpenSearch, Solr
  • A short note on licences
  • Search-first engines: Typesense, Meilisearch, Vespa
  • When nothing off the shelf fits
  • The decision table and flow

From ₹49 · one-time payment, no auto-renewal · separate from course passes

Interview practice