November 23, 2009
What Do You Want?
Help Us Design The Next Release of Panopticon EX
We’ve just released version 5.3 of Panopticon EX after several months of development. It includes three big areas of new functionality (Real-Time Streaming, New Visualizations & Time Analysis), plus a large set of usability enhancements. I went over the real-time streaming functionality in a recent webinar and plan to go through the time analysis features in a another webinar soon.
The requirements for this new version, especially in relation to usability, have generally come from our existing clients. These improvments range from the ability to add custom help messages to each visualization in an EX Dashboard, through disabling the data export function, to time period calculations like moving averages.
July 7, 2009
Optimizing System Performance With
Panopticon Developer SDK
Are you using the Panopticon Developer SDK in your software development projects and thinking about best practices for tuning performance in high throughput scenarios? If so, you should contact us request our best practice guidelines for the SDK. This material describes how to get the best performance for your data visualization applications built using the SDK and covers a range of common use cases.
For example, let's say you have an application that receives data updates from one or two streaming sources and that you also have one or more states set up to synchronize visualizations in your application. How is the performance affected if your application plays it safe and source events are marshaled immediately to the UI thread? How much performance can be gained by doing the processing in separate threads and making sure your application is thread safe?
June 23, 2009
BACK TO THE ROOTS OF THE TREEMAP DATA VISUALIZATION
In the 1990's, treemaps were not very well known and most people looked at them as an exotic and unusual type of data visualization. Treemaps let you focus on the elements that are most important to you, make it obvious which elements need immediate attention and help you spot outliers and patterns without hardly thinking about it.
The first thing to understand about treemaps is that these graphs don’t use a traditional rows and columns-oriented matrix layout. They display a data tree rather than a data table with rows and columns. The x- and y-coordinates of the position where a rectangle is located in a treemap doesn’t really mean anything. Instead it is the containment structure of the treemap that carries meaning. Recursively nested rectangles illustrate the hierarchy in the data tree — enclosure of one rectangle inside another denotes that we’re looking at different levels of the tree simultaneously.
June 9, 2009
What does a typical Panopticon EX deployment require from your hardware, your software and your users?
You may wonder if Panopticon EX requires platforms and user skills that are within reach to your organization. What is required in terms of hardware and software infrastructure to support Panopticon EX? What skills must users and administrators posses to manage the system?
The Panopticon EX platform contains three application level components – a) EX Server, which includes the b) EX Rich Web Client and finally c) EX Designer - a Windows application used to create and publish interactive dashboards to EX Server. A typical deployment of Panopticon EX looks like this:
- One EX Server instance running on a Windows server accessible to the users
- Several installations of EX Designer for the power users who wll create and publish BI Dashboards to the server
- Most users will access the system using the EX Rich Web Client Java-enabled web browsers
May 27, 2009
Do you need an SDK for information visualization –
or will just any chart library do?
When developing your custom interactive dashboard solution or embedding visualizations into your new or existing applications, is a traditional chart library good enough or should you use a software development kit (SDK)? Some people don’t understand the big differences between chart libraries and SDKs.
The answer is, of course, that it depends on what you're trying to accomplish. Chart libraries are sometimes a good choice, although they are completely inadequate in many circumstances. Here’s a few questions you can ask yourself to help sort out the best route to take:
- Do you need to clarify complex data with your visuals?
- Are you working with data that is changing frequently?
- Do you plan to embed capabilities in different platforms?
- Is your application mission critical?
If you answer "yes" to any of these questions, you probably need an SDK rather than a chart library.
May 18, 2009
The History of Data Visualization
Jeffrey Heer is a well respected researcher in the information visualization research community. He teaches information visualization at Stanford University and I recommend you check out his homepage. He has recently posted a movie with one of his lectures that he gave at Stanford that's truly fascinating about the history of data visualization:
By the way, you can see on his home page that he counts his study on the Horizon Graph, a data visualization available in Panopticon’s products, to be among some of his latest and greatest research. And don’t forget to check out his other work too; it is very inspiring — all of it!
Markus Skyttner
CTO
Panopticon Software
May 7, 2009
"loose coupling" or "deep embedding"?
At Panopticon Software, we see different needs from different types of users:
- Software developers may need the full freedom and flexibility to work with and customize components. They often want to embed our tools at the code level in their own existing applications.
- Business users don’t look for the same freedom that you get with a componentized offering. They want more of a self-service application suite with all the necessary functionality already in place. They want to roll it out quickly and start using it to support improved decision-making in their organization. They want to avoid coding and lengthy deployment projects.
We offer different products to address both these needs...
January 20, 2009
Horizon Graph under scrutiny – and passes the test!
Jeffrey Heer at Stanford University, together with Nicholas Kong and Maneesh Agrawala from Berkeley, have now published their research study where they compare the performance of the Horizon Graph to that of a line graph. The study is entitled "Sizing the Horizon: The Effects of Chart Size and Layering on the Graphical Perception of Time Series Visualizations". In brief, they come to the conclusion that the Horizon Graph is more effective than the traditional line graph when graph sizes are small. The Horizon Graph results in fewer estimation errors, with similar or slightly longer estimation times. The paper will be presented at the CHI 2009 (Computer-Human Interaction) conference in April and has even been nominated for the CHI Best Paper award!
This is of course very rewarding for Hannes Reijner in R&D at Panopticon and all of his colleagues who have spent a lot of time developing the Horizon Graph. And I would say the satisfaction is two-fold: it is both seeing this attention to our innovations, and seeing that the visualization stands the test. Heer, Kong, and Agrawala were introduced to the Horizon Graph through Stephen Few and his article "Time on the Horizon" published in his Perceptual Edge newsletter in June, 2008 – so many thanks to him for giving it his attention.
For a quick introduction to the Horizon Graph, I recommend Hannes Reijner’s own write-up, "The Development of the Horizon Graph" which he wrote for Maureen Stone’s workshop From Theory to Practice: Design, Vision and Visualization at the 2008 Infovis conference.
Theodor S. Klemming
VP Corporate Development
Panopticon Software AB
December 23, 2008
A reply by Director of Product Management Brian O'Keefe to the article entitled "Building on BI with Advanced Analytics" by Steve Palmer, published in the DM Review Special Report dated December 23, 2008.
Usability and Next Generation Data Visualization Tools for Operational BI Applications
Steve notes that, according to a 2008 IDC report, advanced analytics software includes data mining and statistical software. “It uses technologies such as neural networks, rule induction and clustering among others to discover relationships in data and to make predictions that are hidden, not apparent or too complex to be extracted using query, reporting and multidimensional analysis software.”
This is a very specialized niche and Panopticon is heading in another direction. We are focusing on the Operational BI world. We believe that our software can become pervasive in the corporate environment, including everything from retailers to commodity manufacturers. Our tools can be easily deployed for a wide variety of users, including executives, salespeople, mid-level managers, financial people and analysts. We strongly believe that great data visualization tools like those in our new Developer SDK 5.2 belong on everyone's desktop; they are not just for power users or Quants.
Those power users and quants can use our Enterprise and Explorer products to design dashboards, but the results of their work will be shared throughout the firm.
Panopticon's data visualization tools are applicable to a wide variety of business problems. It's much easier to spot problems early using the right data visualization than it is with reports or spreadsheets, no matter how complex they are.
It's important to remember that the number one criteria for pervasive software is ease of use! A highly statistical analysis of data pushed to ‘normal’ people won’t yield useful results. In fact, it will quickly be disregarded by business users as they don’t want to openly show that they don’t understand the techniques.
An algorithm is only as good as the variables that make up the function. Business users must be able to define the data that is useful to them and then structure it so that they can easily interpret the resulting visualization. This is exactly what our new tools allow you to do, and we're seeing excellent uptake from our users.
Our goal is to arm each and every business user with the tools to make informed decisions rather than only providing an executive tool or quant tool for a few people who may or may not be effective influencers of upper management when it comes to corporate strategy.
Brian O'Keefe
Director of Product Management
Panopticon Software AB
December 4, 2008
A reply by Director of Product Management Brian O'Keefe to the blog post, "Economic Crisis to Analytics" by Bhupendra Khanal, Business Analytics — Analysis of the World of Business".
Analytics to the Rescue?
Analytics and business decisions must move down the corporate ladder to the front line; that is, business users. Today, your description of business decisions coming exclusively from the 'top' is correct. A person with the highest title or largest paycheck exclusively make all the decisions which are largely based on a mound of data that is aggregated up in a traditional BI report. As evidenced by crisis after crisis, this 'top down' method isn't working.
From my days of derivatives trading I realize that often times, the sum of the parts is greater than the whole. So, inform and empower the enterprise. By delivering manageable and relevant sets of business data to decision makers spread across the institution then more informed (i.e. better) decisions can be made. The idea is to have an interactive BI tool that is pervasive. And two key tenets of pervasive software are:
- Ease of Use
- Extensibility
Today's Big 4 BI providers don't meet these requirements. What's needed is a BI tool with a flexible data model, highly configurable framework and it must be interoperable with other key business systems. Additional requirements in this new paradigm of BI include: interactive visualizations, monitoring, in-memory analytics, CEP (or alerts) and the ability to seamlessly embed any or all of these features in a firm's current systems architecture. BI must change to empower business users.
Brian O'Keefe
Director of Product Management
Panopticon Software AB
Thursday, November 20, 2008
Some like it smooth, some like it crunchy
When I was a kid, peanut butter was available only of the smooth kind in our store. The crunchy kind, with pieces of crushed peanuts mixed in, I first saw many years later. Perhaps this is why I prefer rich client front-ends to thin clients.
When doing analysis work with information visualization tools, a great part of the advantage is attached to functionality such as smooth, stepless zooming, smooth direct manipulation filter slider controls and animated transitions. These are all properties of visualization software that helps you understand what’s going on: Where am I going when zooming or drilling? Which items go away at the different filter levels? What happens to my objects of interest when I skip to a different visualization? Personally, I want my analysis client to be as smooth as peanut butter.
Of course, animated transitions and direct manipulation requires more software intelligence to be put into the client, gives it a larger footprint and creates some usage requirements. This is probably why others prefer crunchy peanut butter, or should I say zero foot-print clients. They are typically built up by DHTML, images and javascript, and eliminates the need for any client side software installations. However, when drilling or filtering, there are no smooth changes. Instead, the entire page reloads – hopefully fast – causing an analysis flow disruption where you lose orientation. And you wonder “what happened?”. Much like when you hit a piece of a peanut when eating crunchy peanut butter.
Theodor S. Klemming
VP Corporate Development
Panopticon Software AB
Wednesday, November 5, 2008
Election Year Equals Geo-Viz Prime Time
The US presidential election is over and Barack Obama in the new president of the United States.
The past few months have been a veritable geo-viz feast in all the media, from prime-time television coverage of the election to newspapers, blogs, websites, posters, and even billboards! Electoral maps showing each state in the Union color-coded by predicted and actual results have been everywhere. Starting today, these geographic visualizations are showing the actual political opinion of the Americans, rather than pre-election survey data.
 It has been interesting to compare the different maps that are out there. I find that the online, interactive versions are preferable to the printed ones or even the very sophisticated maps used on the major television networks. The interactive maps give me really useful features like popup boxes with additional details, the ability to drill-down or zoom to county detail level, and links from each mapped area to an in-depth fact page. This approach is so much more useful and effective for communicating the important information compared to static presentations.
Many of the maps I've seen are excellent examples of the power of information visualization software. I have even seen one or two with a time slider, allowing you to compare the election results in an area over the past few elections.
We have a really nice interactive Treemap visualization showing previous election results in our Demo Gallery. It's a great way to understand trends in US election history, spot the really close elections, and drill down to get more data. Click here to check out our US Election Treemap.
Here a few examples of some of the better maps that I found on the web today. I may be prejudiced, but I think the interactive map from Svenska Dagbladet is the most informative. It combines geographic information with clear indications of population and voting results in a single image. Very clever and easy to interpret.
Another map that I like is the one from Wall Street Journal. They use color gradients to show how votes were distributed between Obama and McCain all the way down to county level. Most of the other maps give you the impression that Texas is 100% Republican, but the Wall Street Journal map shows you clearly that there is wide diversity in Texas in terms of vote percentages, with some counties reaching over 90% in favor of McCain and others reaching over 80% in favor of Obama.
Here are the maps - the links will take you to the interactive maps. Check them out!
Theodor S. Klemming
VP Corporate Development
Panopticon Software AB
Wednesday, August 6, 2008
Visualization Plug-Ins in Panopticon Developer SDK v5.1
Our customers have been very excited about the new Visualization Plug-In functions we’ve implemented in our SDK. I want to clarify the key concepts on how the Plug-Ins work as well as how you can use 3rd party visualizations (like Chart FX) with our SDK.
Beginning with Panopticon Developer 5.0 (yes, that’s 5.0, not 5.1), the SDK has a plug-in mechanism for visualizations.
We offer four different visualization plug-ins in the SDK — Treemaps, Horizon Graphs, Barseries, and Stack Graphs. (Sometimes we refer to them simply as “visualizations”.) You can think of these visualizations all having the same nice rounded peg that allow them to slot into a matching hole in the SDK Framework. The shape of this peg is dictated by the corresponding hole in the Framework component. (In actual practice, this is specified by our visualization API.)
It’s important to understand that the SDK Framework itself doesn’t know anything about how to present data. It handles almost everything else — OLAP functionality, real-time data, filtering, and so on — but without a visualization plug-in you cannot actually see anything. Therefore, you must select a visualization and plug it into the Framework.
This gives the Framework the “missing piece” it needs in order to take the processed data and turn it into a picture. The visualization plug-in — in this case our Treemap — only contains logic, so it’s like slotting that special knowledge about Treemaps into the framework.
Here’s another key concept: The Treemap visualization plug-in contains all logic needed to determine the size and position of the boxes in a Treemap that gets displayed on the screen. Think about what our four out-of-the-box visualizations have in common and what things are different; you should be able to get an idea about what logic goes into the Framework box and what needs to be in the Treemap box.
What’s new in Panopticon Developer 5.1?
Everything said in the previous section applies both to both 5.0 and 5.1 of our .NET SDK. The plug-in architecture has not changed in 5.1, so the shape of the visualization “peg” is still the same.
We’ve bundled in a new example visualization plug-in that is actually a set of text documents. It is comprised of a bunch of C# source code file in the .NET edition (and Java source code in the Java edition of the SDK) along with a tutorial document in HTML.
This is really a major revolution for the SDK. What a developer can now do with 5.1 is compile the example code (using Microsoft Visual Studio in the .NET edition or Eclipse in the Java edition). The example source code files contain all the information needed to create a visualization plug-in, plus it’s completely open. The tutorial document has thorough step-by-step instructions that we’ve tried to keep as generic as possible. A good developer can follow them regardless of what kind of visualization he or she wants to create, whether it’s a pie chart, a line graph, a spark line, a bullet graph, or scatter plot. With this new function, developers can add any sort of visualization they like to their implementations, including proprietary visualizations that they may have created themselves!
The compiled plug-in is a proper SDK visualization plug-in, so you use it exactly as you would use the Treemap or any of the other three standard visualizations available for the SDK. All the SDK Framework cares about is the shape of the “peg”, and the example code shows you how to create a “peg” that fits into the Framework.
Stefan Odelfalk
Senior Consultant and SDK Architect
Panopticon Software AB
Tuesday, April 29, 2008
Displaying real-time data in web clients with Streaming AJAX using Panopticon Enterprise
Panopticon Enterprise is our server product that enables web deployment of Panopticon’s visual business intelligence technology. Our latest release uses “Streaming AJAX” technology to push data in real-time from the server to its web front-ends. This allows us to deliver live data with real-time updates over the Internet to web clients using standard HTTP connections. This is a great feature that enables companies to easily expose real-time data streams to their employees and/or customers. The system works equally well on a company Intranet or on the public Internet.
This new real-time functionality is available in both editions of Panopticon Enterprise — Java and .NET.
Previous versions of the server product used a more traditional stateless delivery model; clients polled the server to receive updates. The new Streaming AJAX solution is quite different. We’re no longer polling the server for updates — the server is delivering real-time updates to the clients as they happen, when they happen.
You can still configure Panopticon Enterprise in the traditional stateless “polling” mode if you like. In this case, the refresh rate is set for the client applications. The clients make a background request for updates at defined time intervals and new data that has accumulated since the last request is delivered to the client in a batch. Polling mode may the best way to use the system in some application scenarios. It offers great scalability; a single server can handle huge number of users without requiring a lot of hardware. However, it is better suited for static data than for highly dynamic real-time feeds since the network bandwidth consumed using polling mode can become large in deployments with large numbers of clients and frequent data updates. For example, you might see problems if you had tens of thousands of clients requesting updates every few seconds.
The new Streaming AJAX technology in the current version of Enterprise provides an excellent alternative to polling mode and also provides users with a greatly enhanced experience. Real-time streams of data are delivered to the clients with very low latency and with minimal bandwidth consumption. The Streaming AJAX engine we use from Lightstreamer has been optimized for real-time distribution of data through HTTP connections based on a publish/subscribe paradigm.
The Streaming AJAX mode is highly scalable. It can support thousands of connections per CPU. The system also allows you to add additional servers and use load balancing tools. The Lightstreamer Streaming AJAX engine includes some really handy features that are particularly useful to Panopticon users:
- Bandwidth and frequency control function designed to avoid problems with unpredictable network traffic
- Adaptive streaming function designed to avoid data bursts and data aging; the system adapts in order to not send more data than the network can handle
This technology allows our web client applications with web-based visualizations to provide a richer and more responsive user experience, one closer to what is possible in richly interactive desktop applications for streaming data.
Displaying real-time streaming data in a web client presents some interesting design issues. The screen can get very busy unless you carefully design the software to make it easy for people to interpret the real-time updates. The designer has to take high update frequencies into account and present changes in an understandable way. We have developed and evaluated a range of alternative solutions to address these challenges, in order to come up with a design solution that allowed users to take effective advantage of this great new real-time streaming technology. We developed a new set of “visual change effects” that helps users to perceive frequent changes. These graphical effects are unique to Panopticon. They animate changes to the data by calling attention to the changing value and the event type. We’ve had very promising user acceptance of this approach. With this technique, it’s quite easy to see the real-time updates, but you can still make sense of what’s happening to the big picture as well. Give us a call and we can show you how it works!
Markus Skyttner
CTO
Panopticon Software
Tuesday, April 8, 2008
Handling large datasets with the Panopticon SDK
We often get questions related to how to handle large datasets when building applications using our Developer SDK. The simple answer is that there are no hard-coded limits in the SDK that restrict how many items an application can load into a dataset and visualize. Of course, there are some practical limits when you display data in a visualization, including:
1. Intrinsic visualization capacity
Visualizations that utilize the pre-attentive processing capabilities of the human psycho-visual system, like Treemaps, can display in excess of tens of thousands of items simultaneously. Some time series visualizations can show thousands of time series values per data element on a single screen.
Other visualizations are not designed to be pre-attentive and can display a few thousand simultaneous values at most. For example, a tabular report can easily handle up to 50 rows with 50 attributes simultaneously in a single display. However, with more rows, the user needs to scroll and pan to see all the data, which makes it very difficult — almost impossible really — to compare values. With more than 50 attributes, a table becomes very confusing. In either case, a table with too much data a very ineffective way to visualize data.
The lesson is that the developer must take the capabilities and limitations of the people who have to view and use the visualizations into account. A well designed implementation will provide users with a fun, easy-to-use, easy-to-understand set of visualizations that allow them to very quickly understand their data, see patterns, compare values, and make well-informed decisions for their organization.
2. Hardware capacity
The computing capacity available to the software sets other practical limits. These include available memory, CPU speed, bus speeds, and so on. For example, this limits how much data can be kept in memory by the application and how many updates in the data can be processed per second.
3. System architecture and design
Applications must be supported by an architecture designed to support the task at hand. Many systems are designed to utilize servers that keep huge data sets in repositories (such as OLAP servers), with middleware responsible for staging relevant parts of the data — sometimes using clever caching strategies. This provides access to the client applications that can — on an “as necessary” basis — access the datasets in the repositories that is needed by the application at any given time.
It is generally not wise to send a complete copy of a very large database over the network to the client; it may not be possible given the hardware limitations of the client and it can tie up a huge proportion of your network and server resources unnecessarily. Query languages exist to provide practical, high speed access to large datasheets.
Having said that, however, some visualizations — like the interactive Treemaps included in our SDK — can handle and display very large amounts of data effectively. In the right circumstances, you can really push the limits of your hardware.
Markus Skyttner
CTO
Panopticon Software
Friday, March 28, 2008
New Explorer Visual Business Intelligence Software
We recently released a new version of our desktop Visual Business Intelligence product, Panopticon Explorer. This post discusses some of my own favorite new features in Explorer.
One of the most exciting new features in Explorer is that the software now supports data analysis and monitoring using our new Barseries visualization. This visualization can be used for displaying time series; that is, data that changes over time. Unlike the Treemap, this new visualization utilizes two axes, and you can use the X-to display a time dimension. See my example below. In this picture we see that from January 2004 and onwards, the relative number of pedestrians crossing the border between Canada and Alaska started to increase, especially during the first few months of the year.
This improved support for the time dimension allows you to work with date or time columns in a much richer way than you could using previous versions of Explorer. When creating your visualization, you can choose how you want to interpret your time dimension. For example, you can interpret it to be a quantitative number such as an age measured in days from today; then you can use other criteria for size and color. Or you can choose to interpret the time in your visualization as a categorical entity – this is especially useful when you want to break down your data by days to make it easier to spot repeating patterns. The next screenshot illustrates this type of analysis using a number of “small multiples” barcharts:
The picture shows the number of incoming persons that crossed the border into four states - Arizona, California, New York and Texas – during 2005 and 2006. We can see that the scales used in the small multiples of the graphics are the same. Texas and California are the two states with bars that go beyond 20 million people per weekday - a relatively large number of persons passing the border. We can see that during 2005, an unusually large amount of people crossed the border on foot on Tuesdays in California. Fridays were also popular in 2005, but in 2006, Tuesdays were not popular anymore while Fridays stood out as an even more popular day to enter California on foot. We also see that few people enter the state of New York on foot, but the opposite holds for Arizona.
Panopticon Explorer works with live data, so you can hook it up to streaming real-time data delivered by services like Reuters. A status indicator indicates whether you are looking at streaming data or static data.
This example shows a screenshot of a live display of a portfolio of investments, grouped by sectors, in total 186 different instruments. The portfolio holdings were kept in an Excel sheet that was dropped onto Panopticon Explorer. By connecting to Bloomberg for market data updates, live values are fed directly into the display in true real-time. In the picture above, the value or holding is represented by the size of the rectangles, so we can see that Exxon is a large holding in this portfolio. The color communicates the one day relative price change – blue means price change is upwards, white means no or very small change and red color indicates downward price change.
There are also overlays on top of the rectangles. These overlay symbols indicate that update changes occur in this instant for the elements in the display. For example, the Exxon rectangle has a symbol which shows a ring of outwards bound arrows; this means that the value of our Exxon holdings is increasing in this instant. Note also that Procter & Gamble has received a data update where both the size and the price goes up in the same instant – indicated by the blue upward pointing arrow – and this happens at the same time as the total size of the holding increases. In this display, the color communicates the relative one day price change. Shades of red tell us which stocks are currently falling in price; for example, we can easily see that the price of Yahoo stock is falling.
Panopticon Explorer works with static data too, of course, and in this version we added yet another plug-in which allows you to connect to RSS feeds using the “Connect” toolbar button.
Panopticon Explorer can work as a completely standalone application running on your desktop; this means that you can work while disconnected from your servers. You can share your work using email with a simple click of the mouse. Another great way to use Explorer is to share work done using a Panopticon Enterprise server. You can connect Explorer to an Enterprise server in a completely secure way using Integrated Windows, Basic, NTLM or Digest authentication modes.
Monday, March 24, 2008
Beneath The Surface: Panopticon's Stream Cube Data Model Supports OLAP Functionality
If visualizations are the “Tip of the Iceberg”, what’s beneath the surface?
Mostly when we first talk to customers, we show them our visualizations and how these can support visual analysis and monitoring based on their business requirements. This isn’t surprising, of course, since the visualizations are very prominent in the GUIs for our software. It may therefore seem surprising that the visualizations actually account for a very small part of our technology – in fact they account for less than a tenth of our total codebase in our product platform.
If the visualizations are the tip of the iceberg, what is beneath the surface? One important and significant part of our technology involves our powerful data model, which is at the core of our offering. It provides rich OLAP functionality that supports fast analysis through the visualizations. The data model is actually a “stream cube” that is capable of doing traditional multidimensional analysis for static data (like those specified by the MDX language) and also for dynamic data streams. It can also be used for monitoring scenarios, triggering rich events when certain conditions are fulfilled.
There are a range of other data models that provide multidimensional analytical OLAP functionality. Why don’t we use one of these, such as the Microsoft UDM? First of all, most other data models are not optimized for real-time performance. Ours is designed for making fast calculations on continuously updating data. Also, our model is designed to match the requirements that live visual interactive displays poses to a data back-end. This involves being able to bind aspects of the data to visuals and other requirements. Most other third party offerings are not specifically designed for such use and start to degrade under heavy streams of live data.
The Panopticon data model conforms to the original rules defined by Codd for OLAP data models by offering support for multidimensional fast analysis. Specific features include:
- Multidimensional conceptual views capable of breaking down dimensions as required by interactive reports.
- Supports intuitive data manipulation, such as interactive direct manipulation (filtering, grouping, and so on.)
- Can be used as a standalone middleware if required, or can be deployed in a client/server architecture – either on the server, on the client or both.
- Can be used both for batch and interpretative use.
- Supports switching analysis models – since the data model is “pluggable”.
- Provides transparency against data sources – provides live access to heterogeneous data sources and can be used to unify structured and unstructured data sources.
- Supports storing aggregates (OLAP results) separately from the source data.
- Supports extraction and treatment of missing values.Provides uniform reporting performance and does not degrade significantly in performance when number of dimensions are increased.
The Panopticon data model can be used on its own to “OLAPify” any data source. Because it provides transparency against data sources, you can also “tunnel” MDX to it; that is, use other “cubes” as data sources.
Since the architecture of our SDK is based on a “pluggable” data model, we can even bypass our data model entirely and tie the visualizations directly to other data models. In this case, the third party data model used must be able to guarantee support for a range of traditional OLAP analysis functions, including grouping, interactive filtering, calculations and some other things expected from OLAP capable data models.
Markus Skyttner
CTO
Panopticon Software
Monday, February 11, 2008
Introducing our 5th generation SDK: Panopticon Developer .NET v 5.0
The SDK is the core of Panopticon Software’s technology. It is designed to be used by OEMs, financial services companies, telecoms companies and other firms who need to develop and build applications that embed or integrate sophisticated interactive visual analysis and monitoring functionality into enterprise applications. It's a particularly good fit for companies that need to display and analyze streaming data. We use the SDK extensively ourselves; in fact, our Explorer desktop product and Enterprise web-deployed products are built using this same SDK. So yes, we eat our own dog food. We have added a lot of very cool functionality to the SDK over the past seven years and our development team has extensive experience with what is required from an SDK to support a wide range of applications. This new 5th gen SDK encapsulates years of sometimes hard-learned lessons that we have picked up both from customers who have used our SDK to build their own applications and from our own experiences in supporting different application scenarios over the years.
What is new, then? This is a major new generation of the SDK and therefore involves several significant changes. A couple of the most important ones are outlined below.
Firstly, a significant new architectural feature is “plug and play” both for visualizations on the display side (output) and for connectors on the data side (input). This may sound abstract, but it means that new visualizations and data plug-ins can be created and dropped into the framework during run-time. In application scenarios, this can be used to dynamically extend the set of display media and data connectivity options available in the application.
Secondly, another significant new architectural feature is that the SDK now adds the concept of shared state for visualizations. This is useful in many ways, for example in information dashboard application scenarios where multiple interactive visualizations work closely together using coupled filtering, brushing and linking etc.
Thirdly, there are also a couple of new implementations both for visualizations and for data models. On the visualization side there are two new visualizations particularly geared towards time series analysis, called Stack Graph and Horizon Graph. Together with the previously available bar chart visualization that we named Barseries, these new visualizations provide support for a range of time series analysis and monitoring scenarios.
Here are a couple of example screenshots that help explain what type of applications that can be created using the SDK. They come from a small application specifically designed for time series analysis of data. This application was quickly created a couple of weeks ago by a student doing an internship here. It makes use of the new SDK to enable an application to display real-time streaming data for technical analysis and monitoring of time series information. It utilizes the plug and play features by creating a visualization — in this case a traditional line graph — and by adding a custom connector that provides access to various types of time series data (historical stock data and something completely different — heartbeat data from a human subject.)
This first screenshot that shows stock data which uses a prediction model to forecast the price and a display that visualizes where the forecast model deviates from the actual values for the stock price.
This second screenshot shows heart beat data and how patterns can be defined to capture slight variations in the time series — something that may warrant an alert or warning notification to be triggered by the application.
We have a number of partners who are doing some really interesting things with our SDK and we’ll be talking about them in more detail in future posts. Watch this space!
Markus Skyttner
CTO
Panopticon Software
Wednesday, January 30, 2008
Tapping into the live stream of data for visual analysis and monitoring
Traditional static data sources are old hat to some of our customers, but many people have business requirements that require visualizing dynamic data. For example, the ability to handle real-time streams is a "must have" in algorithmic trading applications. The developers of these systems often implement support for multiple data layers, including traditional databases and real-time streaming sources. They utilize cutting edge techniques, including messaging middleware, to provide streams of dynamic data to a wide range of different business-critical applications. In my experience, popular third-party sources are most frequently incorporated into algorithmic trading systems that in turn connect directly to low-latency real-time financial applications. We also see developers using homegrown in-house streaming systems with proprietary APIs that are designed to push out updates as they happen in real-time to trading systems, and so on.
CEP vendors are coming into the market with some really interesting systems these days and many financial institutions are beginning to replace their homegrown systems with these vendor-supplied systems. I recommend having a look at http://magmasystems.blogspot.com/ for some interesting posts on the Complex Event Processing topic. Marc Adler, the owner of this Blog, will can give you an interesting take on what is going on in this space. If you follow the Blog you will get a good feel for the mindshare that various CEP vendors are staking out for themselves within the Financial Services market.
Relational and multidimensional databases are familiar and well-known technologies. Their query languages, SQL and MDX respectively, are more or less standardized. Most quantitative and structured data is kept in such static data repositories or will eventually at least end up there. However, this type of quantitative and structured data can only provide part of the whole picture when supporting informed decisions based on data. Ever more unstructured or semi-structured data is required to feed algorithmic trading engines, and this sort of data can be difficult to access even using the latest search engines. And then we have the critical element of live streaming data sources, which is the topic of this post.
How can all of these data sources — structured, unstructured, and streaming — be combined? How can you monitor and analyze them? We have designed our Visual BI platform and our rendering pipeline to support these sources. We use the concept of a data plug-in (also called a connector); this is an adapter that provides access the data, regardless of whether it is static or streaming. These connector modules are available out of the box for some of our products, providing simple point-and-click or drag-and-drop access in applications where our end-users connect to static data sources such as relational databases and multidimensional OLAP databases. We also offer these connectors for a range of dynamic data sources or streams, such as Reuters feeds, Bloomberg terminals and SonicMQ and the ubiquitous Excel format.
What about customers who need to leverage their in-house IP and investments they have made into their own custom built system or a non-standardized proprietary API? By definition, we can’t offer a pre-built data plug-in for these situations, but we can solve the problem. Let’s say the customer doesn’t want pipe the data stream into some standardized data source that we can already access. We can develop a custom data plug-in in cooperation with the customer and then slot that module into one of our existing out-of-the box applications for visual analysis and monitoring. This data plug-in can be embedded into the customer’s existing applications as a component when Panopticon Developer SDK is used to integrate visual analysis and monitoring on top of that app (or to embed it within the app).
We continue to build more data plug-ins to cover more and more dynamic data sources. Please feel free to contact us and let us know what dynamic data sources you would like to see supported in the future!
Markus Skyttner
CTO
Panopticon Software
Wednesday, January 9, 2008
Thoughts on “What’s My Fill?”
I introduced Panopticon’s new series of webinars yesterday. Marcus Eder, one of our most senior management consultants, produced yesterday’s event with me. Marcus has over ten years of experience in the implementation of software within financial services organizations and I’ve been working in financial services for over 20 years now, including a stint as a portfolio manager at Putnam Investments.
I’ll start by saying: The integration of static data sources with real-time streaming data is powerful stuff!
The response to the webinar was great! We had over 100 participants from all over the world, including people from the US, the UK, Continental Europe, the Middle East, South Africa, and India. We even had someone from Australia who couldn’t attend since it would have been in the middle of the night for him to watch it live; he’ll watch the recorded version on our website. Based on this fantastic response to the topic, I’m even more confident in saying that interactive visualization tools are the “in” thing for the financial community right now — the most reliable way to gain a true competitive advantage!
In the “What’s My Fill?” webinar, we introduced our tools for Trade Monitoring and showed how they could be used to efficiently visualize the status and performance of their orders on different levels — by sector, trader, country, firm, client, and so on. During the webinar, attendees could type questions to us in a chat room. In the middle of the presentation, one participant from a large financial institution simply typed “So, what IS my fill”? I was a bit surprised at first since that was our main topic for the webinar and we were definitely going to answer that question. What I realized is that he didn’t have time to watch the whole webinar, and why? Because in this market, not only minutes, but seconds separates a winner from a loser. He needed to get back to his business and yet he was so interested in the interactive visualization tools we presented that he took some time to participate in the webinar! That’s why our clients use our interactive visualization tools; they don’t have that (extra) time to spend on reading reports or browsing through tables to find what they are looking for. The information needs to be there, in an instant and in a format that makes it easy to see what you are looking for. This is true for any market these days, whether you’re dealing with commodities, equities, fixed income instruments, CDOs or whatever.
Back in the day, before joining Panopticon, I worked with several large financial institutions and banks. Our main challenges were to find new ways to gain competitive advantages and that was also when I first laid my eyes on visualization tools. That was before streaming real-time was a reality and before there were visualization tools that could be plugged in to any existing system. Today, all I can say is that interactive visualization tools are better than ever before and they will definitely make your life a lot easier. So to answer the question, “What’s My Fill?”, I say that the fill is the tool that gives you that competitive advantage and saves you that extra time. You can use the time you save by using effective visualization tools to give your clients better service. It might sound as a cliché but it’s even more true today – time is money!
Thanks to everyone who participated in yesterday’s webinar and I’d also like to hear from you! Please send me an email (brian.okeefe _at_ panopitcon.com) with your follow up questions. We’re hosting a series of these events over the next couple of months; we’ll announce them on the site and in our newsletters.
Brian O’Keefe
Director of Product Management
Panopticon Software
Sunday, December 16, 2007
Infovis 2007 and the awakening of infovis
For the third year in a row, I visited the yearly Infovis conference with a small team of software engineers from the company. This was about a month ago now, and this year it was located in Sacramento, California. Back again we’re now deep into a number of release cycles. As always, the yearly conference for information visualization research provided us with inspiration and new perspectives. It also boosted our confidence in our own technology being ahead of both the competition and in some cases ahead of the latest research being made into some areas of information visualization. Saying that, I particularly have in mind the exciting and challenging area of presentation of streaming abstract data, that is, “live information visualization”.
The conference not only boosted our confidence, but also gave us a number of new ideas that we intend to make use of when we continuously improve our VBI software platform – which is in essence an industry strength commercialization of various information visualization techniques. Sometimes working with innovative software, you feel like you are “the world’s best kept secret” and wonder, just like many of our customers do, why the market penetration for this type of technology hasn’t progressed more. Well, if you take a look at many of the applications coming out of research, they lack testing and design and continuity and support, and are by nature more proof of concepts than examples of wide spread industry strength stable software. Not surprisingly, such artifacts will not be able to “bridge the gap” even if the idea behind the software is great. However, the recent consolidation trend among specialist vendors that commercialize these types of technologies proves that this is now turning from an emerging market into a more mature market.
One of the themes on the conference was just that, “how to develop software that reaches a large audience”. In a mature market, people using your software are not experts or statisticians or computer engineers, they are ordinary people trying to do their jobs and need software to provide humane interfaces to support their work. This is the first time I have seen a clear and strong presence in the infovis conference also from some of the largest companies that produce software that reach large audiences. Dinosaurs of software, such as Microsoft and Google, were represented at the conference, not only with their researchers presenting papers, but also with some product people. In most cases the dinosaurs are years behind, which is not surprising with complex products with long release cycles such as our all time favorite example Excel. Especially for desktop software, there is the “tyranny of the installed base” which comes into play, effectively slowing things down as anyone faced with maintaining multiple versions of the same software simultaneously knows. Not to mention the requirements of keeping backward compatibility and by doing that also often keeping old and bad habits alive. Despite new “software as a service” models and web applications coming into play now in a big way, this long time truth still holds in many ways.
So although change happens slower than you’d expect if you’re a technologist, I am getting the feeling that we have already entered a new phase where databases and their dual – query languages - now are standardized and commodities. Therefore the market is now turning towards new value creating technologies in the stack, such as “visual analysis and monitoring”. Software specifically designed for “visual analysis and monitoring” resides in the topmost layer of the technology stack, the layer closest to the human eyes and brains of the people out there. Data tucked away in non-visual databases is not utilized efficiently in organizations, because query languages are not people-friendly. The market is understanding that data, this extremely important asset in any business, can be utilized a million times better if it can be turned easily into actionable information. But then it needs to become understandable somehow, which is the big challenge. So how can we fill that need? By creating software that supports people-friendly visual ways of working with the data we can tap into the amazing bandwidth of the human visual system and unleash the tremendous parallel processing power of the human mind. It is extremely exciting and challenging to develop software that delivers on this challenge. In our marketing we use the term Visual Business Intelligence when explaining our technology. This is a term used also by Stephen Few, author of the excellent book Information Dashboard Design and the keynote speaker at Infovis 2007. I recommend a visit at his blog Perceptual Edge which very much deals with some of the design aspects surrounding this promising technology.
Today I just learned that Professor Anders Ynnerman, one of the visualization researchers presenting at the conference, and a researcher at NVIS – Norrköping Visualization and Interaction Studio – Sweden’s centre of excellence for visualization research (http://nvis.itn.liu.se/) won the Swedish IT-person of the year price for his work in the visualization area. This is really exciting. Panopticon Software has NVIS as our research partner and collaborate closely with this institution. One student from NVIS is currently doing his master’s thesis as Panopticon Software, specializing within the area of predictive analytics. It is great that such a small country as Sweden has such an edge within the exciting emerging software technologies for visualization. We also congratulate Anders to being elected into the Swedish Research Council.
Markus Skyttner
CTO
Panopticon Software |
