Historize or Roll-Up

Information Technology is all about acquiring, processing, storing and presenting the “data” to the right people at right time to enable them to derive some meaningful information and in some cases useful intelligence or insight out of that data.

In usual business, a data point is captured only when there is a transaction that changes the data. So, each change to the data is captured, validated and stored. Such systems are called OLTP or On-Line Transaction Processing systems.

But, when the data is acquired at regular intervals in typical process control systems, not all the data points required to be stored; For example
a. the utilization of a system processor at every 5 seconds interval
b. Temperature of a steam turbine scanned at every 400 ms interval

In a typical process control system there will be several thousands of such data points scanned at very high frequency, typically every second.

What to do with all this data?
In a standard relational database storing all this data in raw format will be simply impractical. So, there are two methods to make some sense out of such “time series” data.

1. Historize the data using a process historian. A process historian uses a compression algorithm that only stores a data point only when there is a deviation beyond a set limit using variety of algorithms like straight line interpolation (SLIM1, 2, 3) or swinging door compression etc., to achieve a high degree of compression in storing the time series data.

2. Roll-up data on a periodical (i.e., hourly) basis to store max, min, average, standard deviation etc., values in one single record per data point. This can be used for historical trending purposes.

There are advantages of both methods. Recently I have seen a patent on dynamic compression of system management data which is interestingly putting the Historization with multiple compression algorithms for storing the systems management data.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: