Quickwit takes on Elasticsearch with an open-source search engine for large datasets

Leave him OSS Enterprise Newsletter guide your open source journey! register here.

Search plays a fundamental role in nearly every modern app, from Amazon and Netflix to Slack and Salesforce. In addition to this, each application generates chunks of log data, which includes timestamped information about events inside the software – these can be details of resources accessed, runtime characteristics of a application and everything related to the operation of this system. .

Being able to search and make sense of all these computer-generated logs is important because it helps businesses troubleshoot errors and bugs, resolve bottlenecks and latency issues, comply with regulations or internal security policies and better understand what’s going on under the hood. It’s all part of what’s known as “observability,” which is the ability to measure the internal state of a software system by analyzing the raw outputs.

Typically, companies consolidate all of their log data into a centralized database system such as ClickHouse. But administering and managing it all comes with many challenges, while the costs associated with storing log data can lead companies to abandon parts of it. It is a problem that Live spirit sets out to solve, with an open-source, cloud-native search and analytics engine built for large datasets.

Founded in 2020, Quickwit touts its ability to perform sub-second queries on terabytes of data on object storage services such as Amazon S3 – and it promises to do so up to “ten times cheaper” than Elasticsearch. But Quickwit isn’t necessarily designed to directly replace all the scenarios covered by search incumbents like Elasticsearch — it has more limited, targeted use cases in mind.

To continue the momentum he created since its first release last July, the company today announced $2.6 million in seed funding co-led by FirstMark and firstminute, with participation from a number of notable angel investors, including MongoDB co-founder Eliot Horowitz; Florian Douetteau and Clément Stenac, CEO and CTO of Dataiku; and SendGrid founder Isaac Saldana.

‘Profitable’

Built on top of Rust based open source search engine library Tantivy, which was created by Quickwit CEO Paul Masurel five years ago, Quickwit is a distributed search engine that offers additional functionality on top of Tantivy’s “low-level building blocks” for searching. Quickwit provides a REST API for indexing data, performing search queries, managing indexes, and managing clusters, with a suite of pre-built connectors covering data sources such as Apache Kafka, Amazon Kinesis, and Amazon S3.

At its core, Quickwit is intended for so-called “immutable” datasets (data that is never deleted or updated), which makes it perfect for companies wishing to store and search log data. It also promises sub-second latency of as little as 140 milliseconds, which is great for log management searches. However, of course milliseconds matter in the online world, which is why Quickwit does not target use cases such as e-commerce websites that require lower latency.

“A search query requires at least two round trips to object storage. However, object storage systems have higher latencies than the local disks used by systems such as Elasticsearch,” said François. Massot, co-founder of QuickWit, to VentureBeat. “Ultimately, Quickwit can never respond faster than 130-140ms, which is acceptable for the use cases we’re targeting, but not for those in e-commerce where higher latencies correlate with loss. of sales.”

That said, Quickwit competes with Elasticsearch for certain purposes, including searching logs, cloud storage backups, and providing full-text search functionality for online analytical processing databases ( OLAP) like ClickHouse. But it’s in these use cases that Quickwit hopes to differentiate itself enough to win the hearts and minds of small businesses and enterprises.

For starters, object storage is cheaper to store data than hard drives, while Quickwit is written in Rust, which is known to consume less memory than Java, which Elasticsearch is based on. Additionally, Quickwit’s decision to separate compute and storage may set it apart in the analytics space – it claims to be the first open-source search engine with such an architecture in place.

“Quickwit instances are stateless and can be started or stopped in seconds. You don’t need to move data around like in Elasticsearch, because storage is separate from compute,” explained Quickwit co-founder Adrien Guillo.

In theory, this all translates to faster and cheaper for Quickwit’s target use cases. And given this promise of profitability, companies may be more inclined to keep more log data, which will improve the insights they get into their system’s performance.

“A lot of companies end up reducing their log retention to reduce costs,” Massot added. “Quickwit makes it unnecessary to throw away this valuable data.”

When it comes to the types of businesses Quickwit targets, Guillo says it will suit businesses of all sizes. Smaller companies will want to use Quickwit as a log search observability building block, while larger companies will build entire applications on Quickwit, covering application and log management, search analytics, search and data lake analysis, and more.

“Companies are struggling to make existing search systems work at scale and must mobilize significant resources and capital to do so, especially for the ever-increasing number of logs generated by applications, systems and business events. “Guillo said. “Quickwit offers unparalleled profitability.”

While Quickwit didn’t give away much regarding its first customer base, Guillo did confirm that they had worked with French unicorn Contentsquare on a proof of concept.

For now, Quickwit’s business model is based on a simple dual-licensing approach – an open source AGPL license for free use and a commercial license that includes support and gives the licensee a “voice in our roadmap” , said Guillo.

Although the company plans to offer software as a service (SaaS) in the future, this is not currently on Quickwit’s agenda.