Strange Loop

Temporal Databases for Streaming Architectures

Time is intrinsic to information and yet it is usually an afterthought in database designs. We present Crux, a general purpose open source document database with bitemporal graph queries.

This talk will explore the journey of how Crux was conceived from JUXT's consulting experiences of building global system integrations and providing temporal query capabilities for financial services projects.

Jon and Jeremy from the core team will outline the overarching goals for Crux as an "unbundled", pluggable, and layered database with bitemporal ACID transactions. Bitemporality cleanly separates the time when the system learns about a fact (transaction time) from the times relating to the actual life-cycle of a fact (valid time). Modelling valid time is essential in a streaming architecture for maintaining a consistent temporal view of data. Bitemporality in Crux provides efficient and horizontally scalable point-in-time Datalog queries whilst enabling programmatic use of the database "as a value" with globally consistent queries.

The team will also discuss the design choices made to support a Datalog query engine on-top of Key/Value stores such as RocksDB and LMDB, the use of semi-immutable log storage to support eviction of sensitive data, clustering with Kafka, and the use of Clojure for constructing a layered and extensible system. Finally, they will share a view on future directions and how JUXT plans to collaborate with the open source community.

Jeremy Taylor

JUXT LTD

Jeremy Taylor is an Offering Manager at JUXT where he leads the development of Crux as an open source database product. Following 7 years of technical pre-sales for IBM's eCommerce systems business, Jeremy transformed his interest in Clojure and Datalog from a hobby into a career evolution by joining JUXT, the UK's foremost Clojure consultancy. As an Offering Manager, Jeremy is responsible for leading the growth of Crux towards long-term viability as a supported open source product in the enterprise database marketplace. Jeremy is passionate about database technologies and their use for knowledge representation, and he has contributed to the DataScript library through the course of his personal research.

Jon Pither

JUXT LTD

Jon Pither is a tech lead by trade having led projects for JUXT at multiple Tier-1 Investment Banks, an online newspaper website, a major property portal, and an international public electric bikes scheme. Jon is a co-founder of JUXT; a software engineering firm that uses Clojure as its primary weapon of choice. Before co-founding JUXT Jon worked as a senior consultant at ThoughtWorks where he became well versed in Agile software development techniques. During his time as a consultant Jon worked at large institutions such as the BBC and Sky, a major publishing house and a fashion retail giant. Jon writes a series of blog articles for JUXT including a series of reference Clojure adoption stories.