Traditional static data sources are old hat to some of our customers, but many people have business requirements that require visualizing dynamic data. For example, the ability to handle real-time streams is a "must have" in algorithmic trading applications. The developers of these systems often implement support for multiple data layers, including traditional databases and real-time streaming sources. They utilize cutting edge techniques, including messaging middleware, to provide streams of dynamic data to a wide range of different business-critical applications. In my experience, popular third-party sources are most frequently incorporated into algorithmic trading systems that in turn connect directly to low-latency real-time financial applications. We also see developers using homegrown in-house streaming systems with proprietary APIs that are designed to push out updates as they happen in real-time to trading systems, and so on.

CEP vendors are coming into the market with some really interesting systems these days and many financial institutions are beginning to replace their homegrown systems with these vendor-supplied systems. I recommend having a look at http://magmasystems.blogspot.com/ for some interesting posts on the Complex Event Processing topic. Marc Adler, the owner of this Blog, will can give you an interesting take on what is going on in this space. If you follow the Blog you will get a good feel for the mindshare that various CEP vendors are staking out for themselves within the Financial Services market.

Relational and multidimensional databases are familiar and well-known technologies. Their query languages, SQL and MDX respectively, are more or less standardized. Most quantitative and structured data is kept in such static data repositories or will eventually at least end up there. However, this type of quantitative and structured data can only provide part of the whole picture when supporting informed decisions based on data. Ever more unstructured or semi-structured data is required to feed algorithmic trading engines, and this sort of data can be difficult to access even using the latest search engines. And then we have the critical element of live streaming data sources, which is the topic of this post.

How can all of these data sources — structured, unstructured, and streaming — be combined? How can you monitor and analyze them? We have designed our Visual BI platform and our rendering pipeline to support these sources. We use the concept of a data plug-in (also called a connector); this is an adapter that provides access the data, regardless of whether it is static or streaming. These connector modules are available out of the box for some of our products, providing simple point-and-click or drag-and-drop access in applications where our end-users connect to static data sources such as relational databases and multidimensional OLAP databases. We also offer these connectors for a range of dynamic data sources or streams, such as Reuters feeds, Bloomberg terminals and SonicMQ and the ubiquitous Excel format.

What about customers who need to leverage their in-house IP and investments they have made into their own custom built system or a non-standardized proprietary API? By definition, we can’t offer a pre-built data plug-in for these situations, but we can solve the problem. Let’s say the customer doesn’t want pipe the data stream into some standardized data source that we can already access. We can develop a custom data plug-in in cooperation with the customer and then slot that module into one of our existing out-of-the box applications for visual analysis and monitoring. This data plug-in can be embedded into the customer’s existing applications as a component when Panopticon Developer SDK is used to integrate visual analysis and monitoring on top of that app (or to embed it within the app).

We continue to build more data plug-ins to cover more and more dynamic data sources. Please feel free to contact us and let us know what dynamic data sources you would like to see supported in the future!

Markus Skyttner
CTO
Panopticon Software