The search infrastructure most teams run today is Elasticsearch, and for good reason. It handles full-text search at scale, has a mature ecosystem, and its API has become the de facto standard. But Elasticsearch carries architectural baggage that becomes painful in specific scenarios, particularly when you need to manage large numbers of separate indexes. SeaSearch, an open source project from the seacloud-lab team, tackles that problem head-on with a design built around shared storage and a Go runtime that trades JVM overhead for a smaller footprint.

The Multi-Tenant Index Problem

In a SaaS application serving multiple tenants, the cleanest data isolation model gives each tenant its own index. Queries stay scoped to a single tenant's data, there is no risk of cross-tenant leakage through filter bugs, and each tenant's index can be managed, backed up, and deleted independently. The problem is that Elasticsearch was not designed to handle thousands or tens of thousands of indexes efficiently.

Each Elasticsearch index carries overhead: shard allocation, segment merging, cluster state updates, and memory consumption all scale with the number of indexes. When you store all tenants in a single index with a tenant ID field, you avoid the index proliferation problem but introduce a different one. The index grows until it requires manual sharding, query performance degrades as irrelevant tenant data gets scanned, and operations like reindexing or deleting a tenant's data become expensive.

SeaSearch eliminates this tradeoff by design. It supports a practically unlimited number of indexes without the overhead that makes Elasticsearch choke. Each tenant gets its own index, queries stay scoped, and the system does not degrade as the index count climbs into the tens of thousands.

Shared Storage Instead of Data Replication

The architectural choice that makes this possible is shared storage. Elasticsearch clusters replicate data across nodes for availability and read scaling. Every node in the cluster holds a copy of the data it serves, and adding capacity means copying data to new nodes. This model works but creates operational complexity. Cluster rebalancing moves data between nodes, which takes time and network bandwidth. Scaling up requires provisioning new nodes and waiting for data to replicate before they can serve traffic.

SeaSearch takes a different approach. Compute nodes share the same S3-compatible object storage backend. Index data lives in S3, not on individual nodes. When a compute node fails or a new node joins the cluster, the only thing that moves is ownership metadata, not the data itself. The cluster manager maintains a map of which node owns which index partition, stored in Etcd, and recomputes ownership when the cluster topology changes.

This means scaling query capacity is as simple as adding a new compute node and updating the ownership map. No data migration, no replication lag, no rebalancing windows. The node starts serving traffic almost immediately after it joins.

Cluster Architecture

A SeaSearch cluster has four components. Compute nodes handle index read and write requests. A proxy (or gateway) distributes client requests across compute nodes. Etcd stores cluster metadata, including index metadata and the ownership map. A cluster manager monitors node health and redistributes partition ownership when nodes join or leave.

For single-node deployments, the architecture simplifies significantly. SeaSearch uses a local key-value database (bbolt) for index metadata and the local filesystem for index data. This makes local development and testing straightforward without requiring S3 infrastructure.

The partitioning scheme groups indexes into a fixed number of partitions based on a hash of their names. The cluster manager maintains the ownership map in Etcd, and the proxy routes requests to the node that owns the relevant partition. When a node fails, the cluster manager recomputes ownership and transfers partition responsibility to surviving nodes. Because only metadata moves, this process completes quickly even with a large number of indexes.

Mutable Ownership, Immutable Data

SeaSearch organizes index data into immutable segments. Once written, a segment cannot be modified. This design decision has cascading benefits. It makes caching straightforward because a cached segment will never change, eliminating cache invalidation complexity. It simplifies distributed queries because each node can safely read segments without coordination. It makes backups and replication simpler because immutable data is trivially versioned.

The tradeoff is that updates and deletes require writing new segments and marking old ones as obsolete. This is the same approach Elasticsearch uses internally with its Lucene segments, so the pattern is well-understood. The difference is that SeaSearch leans harder into immutability to enable its shared-storage architecture.

Caching and Latency

Retrieving data from S3 introduces latency compared to reading from local disk. SeaSearch addresses this with a local disk cache on each compute node. Index segments are cached on first access and served from cache on subsequent requests. When the cache fills up, older segments are evicted to make room for new ones.

The first request after a segment eviction or node startup will hit S3 and experience higher latency. Once the cache warms up, performance approaches local storage speeds. SeaSearch accelerates warm-up by loading multiple segments in parallel from S3, taking advantage of the high network bandwidth available in modern data center environments.

For queries against indexes larger than a single node's disk capacity, SeaSearch distributes the query across multiple compute nodes. Each node loads and searches a portion of the index in parallel, then results are aggregated. This reduces cache pressure on individual nodes and enables serving indexes that significantly exceed local storage limits.

Built-In Vector Search

SeaSearch includes vector search support with three index types: Flat (brute force), HNSW (hierarchical navigable small world), and IVFPQ (inverted file product quantization). The Flat index type provides exact nearest-neighbor search at the cost of query time. HNSW offers a balance between recall and speed for most use cases. IVFPQ compresses vectors to reduce memory and storage requirements at the cost of some accuracy loss.

Embedding vector search directly into the search engine eliminates the need to maintain a separate vector database for applications that combine keyword search with semantic search. The same index can support both full-text queries and vector similarity queries, and the shared-storage architecture applies to vector data the same way it applies to text data.

Elasticsearch Compatibility as a Strategy

SeaSearch provides an API compatible with Elasticsearch, which is a pragmatic choice. The Elasticsearch query DSL is widely understood, well-documented, and supported by every major programming language client. By matching this API, SeaSearch lets teams migrate existing applications with minimal code changes. The compatibility is not just a convenience feature; it is the adoption strategy. Teams evaluating alternatives to Elasticsearch can test SeaSearch against their existing queries and clients without rewriting application code.

The compatibility also means SeaSearch benefits from the tooling ecosystem built around Elasticsearch. Log shippers, monitoring integrations, and data pipelines that speak the Elasticsearch protocol can target SeaSearch without modification.

When SeaSearch Makes Sense

SeaSearch is not a general-purpose replacement for Elasticsearch. Elasticsearch has a richer feature set, a larger ecosystem, and more mature tooling for many use cases. Where SeaSearch differentiates is in scenarios requiring many independent indexes with efficient resource utilization. Multi-tenant SaaS applications, per-tenant or per-project data isolation, and workloads where index count matters more than single-query latency are the sweet spots.

The shared-storage model also makes sense for teams that want to simplify cluster operations. Eliminating data replication and rebalancing removes entire categories of operational burden. For teams running on cloud infrastructure with S3-compatible storage already available, the storage cost is predictable and the scaling model is straightforward.

The project is open source, which matters for infrastructure that sits between your application and your data. The ability to inspect the code, understand the caching behavior, and verify that no unexpected data flows exist is a baseline requirement for search infrastructure in production.