Compiling Streaming SQL to Byte Code with Esper Complex Event Processing
Today, the Esper team is proud to announce the first compiler that turns Streaming SQL into JVM byte code, available with the just-released Esper version 8.0.0 as of November 21, 2018.
Streaming SQL has been around since the year 2003, when commercial vendors started providing Complex Event Processing products [1, see Streambase in 2003, Coral8 and Aleri in 2006].
Esper is the first open-source implementation of Streaming SQL and Complex Event Processing and had its first release in 2006 [1, 2]. Esper is open-source under the GPLv2 license.
The Esper compiler compiles Streaming SQL into native JVM byte code. The Esper runtime loads the byte code, dynamically and at runtime, to execute for arriving events. Internally, the Esper runtime manages data structures and indexes to quickly match incoming events and time to queries and to process the raw data, allowing queries to produce output to callbacks with near-zero latency and excellent throughput.
Esper’s implementation of Streaming SQL is called the Event Processing Language (EPL). It is a declarative, data-oriented language for dealing with high frequency time-based event data. EPL is compliant to the SQL-92 standard and extended for analyzing series of events and in respect to time [3].
Complex Event Processing is technology that combines data from multiple sources and considers how time passes in order to infer events or patterns that suggest more complicated circumstances. The goal is to take in raw data and identify meaningful events (such as opportunities or threats) and respond to them as quickly as possible (source: Wikipedia).
Streaming SQL is related to Complex Event Processing in that SQL is the foundation for the language that ingests streams and analyzes events and time.
Compiling Streaming SQL to JVM byte code allows optimization to take place at compile-time and at runtime. Compile-time optimizations include co-location of objects, which is reducing heap memory footprint, and elimination of casts. Compiling to byte code allows the Java just-in-time compiler to turn byte code into native code [4]. Esper allows compiling, deploying and un-deploying queries at runtime.