Our customers have been very excited about the new Visualization Plug-In functions we’ve implemented in our SDK. I want to clarify the key concepts on how the Plug-Ins work as well as how you can use 3rd party visualizations (like Chart FX) with our SDK.

Beginning with Panopticon Developer 5.0 (yes, that’s 5.0, not 5.1), the SDK has a plug-in mechanism for visualizations. This is what it looks like:


We offer four different visualization plug-ins in the SDK — Treemaps, Horizon Graphs, Barseries, and Stack Graphs. (Sometimes we refer to them simply as “visualizations”.) You can think of these visualizations all having the same nice rounded peg that allow them to slot into a matching hole in the SDK Framework. The shape of this peg is dictated by the corresponding hole in the Framework component. (In actual practice, this is specified by our visualization API.)

It’s important to understand that the SDK Framework itself doesn’t know anything about how to present data. It handles almost everything else — OLAP functionality, real-time data, filtering, and so on — but without a visualization plug-in you cannot actually see anything. Therefore, you must select a visualization and plug it into the Framework, like so:


This gives the Framework the “missing piece” it needs in order to take the processed data and turn it into a picture. The visualization plug-in — in this case our Treemap — only contains logic, so it’s like slotting that special knowledge about Treemaps into the framework.

Here’s another key concept: The Treemap visualization plug-in contains all logic needed to determine the size and position of the boxes in a Treemap that gets displayed on the screen. Think about what our four out-of-the-box visualizations have in common and what things are different; you should be able to get an idea about what logic goes into the Framework box and what needs to be in the Treemap box.


What’s new in Panopticon Developer 5.1?

Everything said in the previous section applies both to both 5.0 and 5.1 of our .NET SDK. The plug-in architecture has not changed in 5.1, so the shape of the visualization “peg” is still the same. However, in 5.1, we also added this to both the .NET and the Java editions of the SDK:


We’ve bundled in a new example visualization plug-in that is actually a set of text documents. It is comprised of a bunch of C# source code file in the .NET edition (and Java source code in the Java edition of the SDK) along with a tutorial document in HTML.

This is really a major revolution for the SDK. What a developer can now do with 5.1 is compile the example code (using Microsoft Visual Studio in the .NET edition or Eclipse in the Java edition), and the result will be this:

The example source code files contain all the information needed to create a visualization plug-in, plus it’s completely open. The tutorial document has thorough step-by-step instructions that we’ve tried to keep as generic as possible. A good developer can follow them regardless of what kind of visualization he or she wants to create, whether it’s a pie chart, a line graph, a spark line, a bullet graph, or scatter plot. With this new function, developers can add any sort of visualization they like to their implementations, including proprietary visualizations that they may have created themselves!

The compiled plug-in is a proper SDK visualization plug-in, so you use it exactly as you would use the Treemap or any of the other three standard visualizations available for the SDK. All the SDK Framework cares about is the shape of the “peg”, and the example code shows you how to create a “peg” that fits into the Framework.

Stefan Odelfalk
Senior Consultant and SDK Architect
Panopticon Software AB

Panopticon Enterprise is our server product that enables web deployment of Panopticon’s visual business intelligence technology. Our latest release uses “Streaming AJAX” technology to push data in real-time from the server to its web front-ends. This allows us to deliver live data with real-time updates over the Internet to web clients using standard HTTP connections. This is a great feature that enables companies to easily expose real-time data streams to their employees and/or customers. The system works equally well on a company Intranet or on the public Internet.

This new real-time functionality is available in both editions of Panopticon Enterprise — Java and .NET.

Previous versions of the server product used a more traditional stateless delivery model; clients polled the server to receive updates. The new Streaming AJAX solution is quite different. We’re no longer polling the server for updates — the server is delivering real-time updates to the clients as they happen, when they happen.

You can still configure Panopticon Enterprise in the traditional stateless “polling” mode if you like. In this case, the refresh rate is set for the client applications. The clients make a background request for updates at defined time intervals and new data that has accumulated since the last request is delivered to the client in a batch. Polling mode may the best way to use the system in some application scenarios. It offers great scalability; a single server can handle huge number of users without requiring a lot of hardware. However, it is better suited for static data than for highly dynamic real-time feeds since the network bandwidth consumed using polling mode can become large in deployments with large numbers of clients and frequent data updates. For example, you might see problems if you had tens of thousands of clients requesting updates every few seconds.

The new Streaming AJAX technology in the current version of Enterprise provides an excellent alternative to polling mode and also provides users with a greatly enhanced experience. Real-time streams of data are delivered to the clients with very low latency and with minimal bandwidth consumption. The Streaming AJAX engine we use from Lightstreamer has been optimized for real-time distribution of data through HTTP connections based on a publish/subscribe paradigm.

The Streaming AJAX mode is highly scalable. It can support thousands of connections per CPU. The system also allows you to add additional servers and use load balancing tools. The Lightstreamer Streaming AJAX engine includes some really handy features that are particularly useful to Panopticon users:

• Bandwidth and frequency control function designed to avoid problems with unpredictable network traffic
• Adaptive streaming function designed to avoid data bursts and data aging; the system adapts in order to not send more data than the network can handle

This technology allows our web client applications with web-based visualizations to provide a richer and more responsive user experience, one closer to what is possible in richly interactive desktop applications for streaming data.

Displaying real-time streaming data in a web client presents some interesting design issues. The screen can get very busy unless you carefully design the software to make it easy for people to interpret the real-time updates. The designer has to take high update frequencies into account and present changes in an understandable way. We have developed and evaluated a range of alternative solutions to address these challenges, in order to come up with a design solution that allowed users to take effective advantage of this great new real-time streaming technology. We developed a new set of “visual change effects” that helps users to perceive frequent changes. These graphical effects are unique to Panopticon. They animate changes to the data by calling attention to the changing value and the event type. We’ve had very promising user acceptance of this approach. With this technique, it’s quite easy to see the real-time updates, but you can still make sense of what’s happening to the big picture as well. Give us a call and we can show you how it works!

Markus Skyttner
CTO
Panopticon Software

We often get questions related to how to handle large datasets when building applications using our Developer SDK. The simple answer is that there are no hard-coded limits in the SDK that restrict how many items an application can load into a dataset and visualize. Of course, there are some practical limits when you display data in a visualization, including:

1. Intrinsic visualization capacity

Visualizations that utilize the pre-attentive processing capabilities of the human psycho-visual system, like Treemaps, can display in excess of tens of thousands of items simultaneously. Some time series visualizations can show thousands of time series values per data element on a single screen.

Other visualizations are not designed to be pre-attentive and can display a few thousand simultaneous values at most. For example, a tabular report can easily handle up to 50 rows with 50 attributes simultaneously in a single display. However, with more rows, the user needs to scroll and pan to see all the data, which makes it very difficult — almost impossible really — to compare values. With more than 50 attributes, a table becomes very confusing. In either case, a table with too much data a very ineffective way to visualize data.

The lesson is that the developer must take the capabilities and limitations of the people who have to view and use the visualizations into account. A well designed implementation will provide users with a fun, easy-to-use, easy-to-understand set of visualizations that allow them to very quickly understand their data, see patterns, compare values, and make well-informed decisions for their organization.

2. Hardware capacity

The computing capacity available to the software sets other practical limits. These include available memory, CPU speed, bus speeds, and so on. For example, this limits how much data can be kept in memory by the application and how many updates in the data can be processed per second.

3. System architecture and design

Applications must be supported by an architecture designed to support the task at hand. Many systems are designed to utilize servers that keep huge data sets in repositories (such as OLAP servers), with middleware responsible for staging relevant parts of the data — sometimes using clever caching strategies. This provides access to the client applications that can — on an “as necessary” basis — access the datasets in the repositories that is needed by the application at any given time.

It is generally not wise to send a complete copy of a very large database over the network to the client; it may not be possible given the hardware limitations of the client and it can tie up a huge proportion of your network and server resources unnecessarily. Query languages exist to provide practical, high speed access to large datasheets.

Having said that, however, some visualizations — like the interactive Treemaps included in our SDK — can handle and display very large amounts of data effectively. In the right circumstances, you can really push the limits of your hardware.

Markus Skyttner
CTO
Panopticon Software

We recently released a new version of our desktop Visual Business Intelligence product, Panopticon Explorer. This post discusses some of my own favorite new features in Explorer.

One of the most exciting new features in Explorer is that the software now supports data analysis and monitoring using our new Barseries visualization. This visualization can be used for displaying time series; that is, data that changes over time. Unlike the Treemap, this new visualization utilizes two axes, and you can use the X-to display a time dimension. See my example below. In this picture we see that from January 2004 and onwards, the relative number of pedestrians crossing the border between Canada and Alaska started to increase, especially during the first few months of the year.

If visualizations are the “Tip of the Iceberg”, what’s beneath the surface?

Mostly when we first talk to customers, we show them our visualizations and how these can support visual analysis and monitoring based on their business requirements. This isn’t surprising, of course, since the visualizations are very prominent in the GUIs for our software. It may therefore seem surprising that the visualizations actually account for a very small part of our technology – in fact they account for less than a tenth of our total codebase in our product platform.

If the visualizations are the tip of the iceberg, what is beneath the surface? One important and significant part of our technology involves our powerful data model, which is at the core of our offering. It provides rich OLAP functionality that supports fast analysis through the visualizations. The data model is actually a “stream cube” that is capable of doing traditional multidimensional analysis for static data (like those specified by the MDX language) and also for dynamic data streams. It can also be used for monitoring scenarios, triggering rich events when certain conditions are fulfilled.

There are a range of other data models that provide multidimensional analytical OLAP functionality. Why don’t we use one of these, such as the Microsoft UDM? First of all, most other data models are not optimized for real-time performance. Ours is designed for making fast calculations on continuously updating data. Also, our model is designed to match the requirements that live visual interactive displays poses to a data back-end. This involves being able to bind aspects of the data to visuals and other requirements. Most other third party offerings are not specifically designed for such use and start to degrade under heavy streams of live data.

The Panopticon data model conforms to the original rules defined by Codd for OLAP data models by offering support for multidimensional fast analysis. Specific features include:

• Multidimensional conceptual views capable of breaking down dimensions as required by interactive reports.
• Supports intuitive data manipulation, such as interactive direct manipulation (filtering, grouping, and so on.)
• Can be used as a standalone middleware if required, or can be deployed in a client/server architecture – either on the server, on the client or both.
• Can be used both for batch and interpretative use.
• Supports switching analysis models – since the data model is “pluggable”.
• Provides transparency against data sources – provides live access to heterogeneous data sources and can be used to unify structured and unstructured data sources.
• Supports storing aggregates (OLAP results) separately from the source data.
• Supports extraction and treatment of missing values.Provides uniform reporting performance and does not degrade significantly in performance when number of dimensions are increased.

The Panopticon data model can be used on its own to “OLAPify” any data source. Because it provides transparency against data sources, you can also “tunnel” MDX to it; that is, use other “cubes” as data sources.

Since the architecture of our SDK is based on a “pluggable” data model, we can even bypass our data model entirely and tie the visualizations directly to other data models. In this case, the third party data model used must be able to guarantee support for a range of traditional OLAP analysis functions, including grouping, interactive filtering, calculations and some other things expected from OLAP capable data models.

Markus Skyttner
CTO
Panopticon Software

The SDK is the core of Panopticon Software’s technology. It is designed to be used by OEMs, financial services companies, telecoms companies and other firms who need to develop and build applications that embed or integrate sophisticated interactive visual analysis and monitoring functionality into enterprise applications. It's a particularly good fit for companies that need to display and analyze streaming data. We use the SDK extensively ourselves; in fact, our Explorer desktop product and Enterprise web-deployed products are built using this same SDK. So yes, we eat our own dog food. We have added a lot of very cool functionality to the SDK over the past seven years and our development team has extensive experience with what is required from an SDK to support a wide range of applications. This new 5th gen SDK encapsulates years of sometimes hard-learned lessons that we have picked up both from customers who have used our SDK to build their own applications and from our own experiences in supporting different application scenarios over the years.

What is new, then? This is a major new generation of the SDK and therefore involves several significant changes. A couple of the most important ones are outlined below.

Firstly, a significant new architectural feature is “plug and play” both for visualizations on the display side (output) and for connectors on the data side (input). This may sound abstract, but it means that new visualizations and data plug-ins can be created and dropped into the framework during run-time. In application scenarios, this can be used to dynamically extend the set of display media and data connectivity options available in the application.

Secondly, another significant new architectural feature is that the SDK now adds the concept of shared state for visualizations. This is useful in many ways, for example in information dashboard application scenarios where multiple interactive visualizations work closely together using coupled filtering, brushing and linking etc.

Thirdly, there are also a couple of new implementations both for visualizations and for data models. On the visualization side there are two new visualizations particularly geared towards time series analysis, called Stack Graph and Horizon Graph. Together with the previously available bar chart visualization that we named Barseries, these new visualizations provide support for a range of time series analysis and monitoring scenarios.

Here are a couple of example screenshots that help explain what type of applications that can be created using the SDK. They come from a small application specifically designed for time series analysis of data. This application was quickly created a couple of weeks ago by a student doing an internship here. It makes use of the new SDK to enable an application to display real-time streaming data for technical analysis and monitoring of time series information. It utilizes the plug and play features by creating a visualization — in this case a traditional line graph — and by adding a custom connector that provides access to various types of time series data (historical stock data and something completely different — heartbeat data from a human subject.)




This first screenshot that shows stock data which uses a prediction model to forecast the price and a display that visualizes where the forecast model deviates from the actual values for the stock price.





This second screenshot shows heart beat data and how patterns can be defined to capture slight variations in the time series — something that may warrant an alert or warning notification to be triggered by the application.

We have a number of partners who are doing some really interesting things with our SDK and we’ll be talking about them in more detail in future posts. Watch this space!

Markus Skyttner
CTO

Traditional static data sources are old hat to some of our customers, but many people have business requirements that require visualizing dynamic data. For example, the ability to handle real-time streams is a "must have" in algorithmic trading applications. The developers of these systems often implement support for multiple data layers, including traditional databases and real-time streaming sources. They utilize cutting edge techniques, including messaging middleware, to provide streams of dynamic data to a wide range of different business-critical applications. In my experience, popular third-party sources are most frequently incorporated into algorithmic trading systems that in turn connect directly to low-latency real-time financial applications. We also see developers using homegrown in-house streaming systems with proprietary APIs that are designed to push out updates as they happen in real-time to trading systems, and so on.

CEP vendors are coming into the market with some really interesting systems these days and many financial institutions are beginning to replace their homegrown systems with these vendor-supplied systems. I recommend having a look at http://magmasystems.blogspot.com/ for some interesting posts on the Complex Event Processing topic. Marc Adler, the owner of this Blog, will can give you an interesting take on what is going on in this space. If you follow the Blog you will get a good feel for the mindshare that various CEP vendors are staking out for themselves within the Financial Services market.

Relational and multidimensional databases are familiar and well-known technologies. Their query languages, SQL and MDX respectively, are more or less standardized. Most quantitative and structured data is kept in such static data repositories or will eventually at least end up there. However, this type of quantitative and structured data can only provide part of the whole picture when supporting informed decisions based on data. Ever more unstructured or semi-structured data is required to feed algorithmic trading engines, and this sort of data can be difficult to access even using the latest search engines. And then we have the critical element of live streaming data sources, which is the topic of this post.

How can all of these data sources — structured, unstructured, and streaming — be combined? How can you monitor and analyze them? We have designed our Visual BI platform and our rendering pipeline to support these sources. We use the concept of a data plug-in (also called a connector); this is an adapter that provides access the data, regardless of whether it is static or streaming. These connector modules are available out of the box for some of our products, providing simple point-and-click or drag-and-drop access in applications where our end-users connect to static data sources such as relational databases and multidimensional OLAP databases. We also offer these connectors for a range of dynamic data sources or streams, such as Reuters feeds, Bloomberg terminals and SonicMQ and the ubiquitous Excel format.

What about customers who need to leverage their in-house IP and investments they have made into their own custom built system or a non-standardized proprietary API? By definition, we can’t offer a pre-built data plug-in for these situations, but we can solve the problem. Let’s say the customer doesn’t want pipe the data stream into some standardized data source that we can already access. We can develop a custom data plug-in in cooperation with the customer and then slot that module into one of our existing out-of-the box applications for visual analysis and monitoring. This data plug-in can be embedded into the customer’s existing applications as a component when Panopticon Developer SDK is used to integrate visual analysis and monitoring on top of that app (or to embed it within the app).

We continue to build more data plug-ins to cover more and more dynamic data sources. Please feel free to contact us and let us know what dynamic data sources you would like to see supported in the future!

Markus Skyttner
CTO
Panopticon Software