Panopticon Enterprise is our server product that enables web deployment of Panopticon’s visual business intelligence technology. Our latest release uses “Streaming AJAX” technology to push data in real-time from the server to its web front-ends. This allows us to deliver live data with real-time updates over the Internet to web clients using standard HTTP connections. This is a great feature that enables companies to easily expose real-time data streams to their employees and/or customers. The system works equally well on a company Intranet or on the public Internet.
This new real-time functionality is available in both editions of Panopticon Enterprise — Java and .NET.
Previous versions of the server product used a more traditional stateless delivery model; clients polled the server to receive updates. The new Streaming AJAX solution is quite different. We’re no longer polling the server for updates — the server is delivering real-time updates to the clients as they happen, when they happen.
You can still configure Panopticon Enterprise in the traditional stateless “polling” mode if you like. In this case, the refresh rate is set for the client applications. The clients make a background request for updates at defined time intervals and new data that has accumulated since the last request is delivered to the client in a batch. Polling mode may the best way to use the system in some application scenarios. It offers great scalability; a single server can handle huge number of users without requiring a lot of hardware. However, it is better suited for static data than for highly dynamic real-time feeds since the network bandwidth consumed using polling mode can become large in deployments with large numbers of clients and frequent data updates. For example, you might see problems if you had tens of thousands of clients requesting updates every few seconds.
The new Streaming AJAX technology in the current version of Enterprise provides an excellent alternative to polling mode and also provides users with a greatly enhanced experience. Real-time streams of data are delivered to the clients with very low latency and with minimal bandwidth consumption. The Streaming AJAX engine we use from Lightstreamer has been optimized for real-time distribution of data through HTTP connections based on a publish/subscribe paradigm.
The Streaming AJAX mode is highly scalable. It can support thousands of connections per CPU. The system also allows you to add additional servers and use load balancing tools. The Lightstreamer Streaming AJAX engine includes some really handy features that are particularly useful to Panopticon users:
- Bandwidth and frequency control function designed to avoid problems with unpredictable network traffic
- Adaptive streaming function designed to avoid data bursts and data aging; the system adapts in order to not send more data than the network can handle
This technology allows our web client applications with web-based visualizations to provide a richer and more responsive user experience, one closer to what is possible in richly interactive desktop applications for streaming data.
Displaying real-time streaming data in a web client presents some interesting design issues. The screen can get very busy unless you carefully design the software to make it easy for people to interpret the real-time updates. The designer has to take high update frequencies into account and present changes in an understandable way. We have developed and evaluated a range of alternative solutions to address these challenges, in order to come up with a design solution that allowed users to take effective advantage of this great new real-time streaming technology. We developed a new set of “visual change effects” that helps users to perceive frequent changes. These graphical effects are unique to Panopticon. They animate changes to the data by calling attention to the changing value and the event type. We’ve had very promising user acceptance of this approach. With this technique, it’s quite easy to see the real-time updates, but you can still make sense of what’s happening to the big picture as well. Give us a call and we can show you how it works!
Markus Skyttner
CTO
Panopticon Software
We often get questions related to how to handle large datasets when building applications using our Developer SDK. The simple answer is that there are no hard-coded limits in the SDK that restrict how many items an application can load into a dataset and visualize. Of course, there are some practical limits when you display data in a visualization, including:
1. Intrinsic visualization capacity
Visualizations that utilize the pre-attentive processing capabilities of the human psycho-visual system, like Treemaps, can display in excess of tens of thousands of items simultaneously. Some time series visualizations can show thousands of time series values per data element on a single screen.
Other visualizations are not designed to be pre-attentive and can display a few thousand simultaneous values at most. For example, a tabular report can easily handle up to 50 rows with 50 attributes simultaneously in a single display. However, with more rows, the user needs to scroll and pan to see all the data, which makes it very difficult — almost impossible really — to compare values. With more than 50 attributes, a table becomes very confusing. In either case, a table with too much data a very ineffective way to visualize data.
The lesson is that the developer must take the capabilities and limitations of the people who have to view and use the visualizations into account. A well designed implementation will provide users with a fun, easy-to-use, easy-to-understand set of visualizations that allow them to very quickly understand their data, see patterns, compare values, and make well-informed decisions for their organization.
2. Hardware capacity
The computing capacity available to the software sets other practical limits. These include available memory, CPU speed, bus speeds, and so on. For example, this limits how much data can be kept in memory by the application and how many updates in the data can be processed per second.
3. System architecture and design
Applications must be supported by an architecture designed to support the task at hand. Many systems are designed to utilize servers that keep huge data sets in repositories (such as OLAP servers), with middleware responsible for staging relevant parts of the data — sometimes using clever caching strategies. This provides access to the client applications that can — on an “as necessary” basis — access the datasets in the repositories that is needed by the application at any given time.
It is generally not wise to send a complete copy of a very large database over the network to the client; it may not be possible given the hardware limitations of the client and it can tie up a huge proportion of your network and server resources unnecessarily. Query languages exist to provide practical, high speed access to large datasheets.
Having said that, however, some visualizations — like the interactive Treemaps included in our SDK — can handle and display very large amounts of data effectively. In the right circumstances, you can really push the limits of your hardware.
Markus Skyttner
CTO
Panopticon Software
