Announcing WebhookDB

We're proud to announce that our second software product, WebhookDB, is available for an open beta. WebhookDB allows you to query data from any supported API, in real-time, using SQL, by mirroring information from the API into a dedicated Postgres database.

The genesis of WebhookDB is a service we've had to build many times: endpoints that accept webhooks from, say, Stripe, and mirror them into a database for use in both production applications and analytics systems.

Why bother using a database? Well, there are some serious benefits to using a database over an API:

• All APIs are slow. Look, I love Stripe (I even have a featured testimonial for Stripe Connect), but I don't want to have to page through customers for 30 minutes looking for some that match a certain field I cannot search via the API. I'd much rather write a SQL query and find what I'm looking for almost instantly. There are types of application and ETL workloads that are impossible using an API directly.
• Every API is queried differently. Each has different conventions for searching and filtering. Instead, you can use standard SQL operators. We try and be smart about denormalizing and indexing fields most commonly used in queries, but the full document available is also available in every WebhookDB row. You can always fall back to Postgres' JSONB operators.
• APIs are unreliable. APIs naturally have many moving parts and are inherently more complex than something like a database connection. Performance can be unpredictable and intermittent failures not uncommon.
• And finally, APIs are hard to test with. We have extensive experience with WebMock/VCR and similar libraries in other languages. But even for us, it's much easier to fixture data by inserting a row into a database during test setup.

Those are good reasons to use a database, but why use WebhookDB? It can be tempting to write this sort of thing yourself. With a single API that is easy to use, it may be a couple days of work. The devil, though, is always in the details.

Here's why I use WebhookDB rather than building this for clients anymore.

• Get going instantly. If we already support the API you need, the time from "I want to sync data from this API" to "show me the data" is almost instantaneous. We've already done the work to handle denormalizing and indexing the right columns, verifying signed webhooks, supporting lossless backfilling, etc. If we don't already support the API you need, please shoot us an email and we'll have it within a couple days.
• Webhook support in APIs is spotty. Many APIs have limited or no webhook support; this means mirroring data from them requires repeated backfilling in the best case, or a complex series of API calls in others. Backfilling these APIs in particular is a lot more complex than simply "iterate over every page and item."
• You can use WebhookDB in your application. Very few products that sync 3rd party data allow you to use it how you want; it only meant for analytics use, available via their app or API (almost never SQL!). This is both so you stick on their platform, but also because it is hard to the make guarantees needed to use webhook data in your application. We give you a PostgresQL connection so you can use your data how you want, and make sure it is kept as up-to-date as possible.
• UX gets attention. Internal services like webhook databases are usually hidden from external use, so are difficult for anyone to monitor or debug. WebhookDB is entirely driven through a CLI that we've worked really hard on (it's is open source!), so you can easily add integrations, change config, replay backfills, view stats, and much more.
• You can use it from the web. The CLI is compiled to WASM and you can run the terminal from the website. It's actually the primary way we use it, even internally. Being able to enable integrations and query your data from any computer, with no local requirements, is a game changer. Even easier than Docker!
• It's private and secure. We really push ourselves to approach security and privacy from all levels. Along with the webhook payload data, we store the absolute minimum required and give you API-specific instructions about how to lock down permissions where possible. We also have extensive isolation between customer environments, so no sort of cross-account exfiltration is possible.

I hope you will give WebhookDB a try! Please check out WebhookDB or let us know what you think!