Short Case Study

In this short case study we will learn to process a raw market data feed. We will learn to report throughput statistics and to detect when the data rate of a feed falls off unexpectedly. A rate fall-off may mean that the data is stale and we want to alert when there is a possible problem with the feed.

Input

Our input stream consists of 1 event stream that contains 2 simulated market data feeds. Each individual event in the stream indicates the feed that supplies the market data, the security symbol and some pricing information:

String symbol;
FeedEnum feed;
double bidPrice;
double askPrice;

Computing Rates Per Feed

For the throughput statistics and to detect rapid fall-off we calculate a ticks per second rate for each market data feed.

We can use an aggregation function by name "rate" that outputs an average rate given a number of seconds of events to consider. We specify the feed and a count of events per feed as output values. To make this data available for further processing, we insert output events into the TicksPerSecond event stream:

insert into TicksPerSecond
select feed, rate(10) as cnt
  from MarketDataEvent
 group by feed

Detecting a Fall-off

We define a rapid fall-off by alerting when the number of ticks per second for any second falls below 75% of the average number of ticks per second over the last 10 seconds.

We can compute the average number of ticks per second over the last 10 seconds simply by using the TicksPerSecond events computed by the prior statement and averaging the last 10 seconds. Next, we compare the current rate with the moving average and filter out any rates that fall below 75% of the average:

select feed, avg(cnt) as avgCnt, cnt as feedCnt
  from TicksPerSecond#time(10 seconds)
 group by feed
having cnt < avg(cnt) * 0.75

Summary

This tutorial showed how to compute throughput and detect a rapid fall-off on a single event stream. We calculated the ticks per second and a rolling 10-second average. We alerted when a rates fell below an arbitrary 75% threshold.