I love industry trade shows. It is a great opportunity to meet so many different people and get a chance to see different and new technologies under one roof!

Recently I was in a trade show in Vegas and had a chance to talk to a CEO of a network device manufacturer who had been growing steadily. We were discussing the challenges that came with rapid growth and the discussion turned to management of their device - my favorite subject (in case you couldn’t guess!). He pointed out to their management application which was visible in a monitor behind him and was kind enough to give me a quick tour of their management application as well. He seemed to be satisfied with the application - well almost. The almost part is because he confided in me that the application was developed in a hurry (no surprise here) and he felt that the usability could be better. He also felt that adding new functionality to the management application was not easy and in addition it was becoming a bottleneck because of its inability to expose the functionality that had been added to the device after the first version of the EMS was released.

From our experience we have noticed that a good EMS has to balance a few things, at the very least three things. First it should be able to expose the new features that get added on to the device. Most often we’ve noticed that when the device manufacturers release the first version of their device, they tend to limit it’s feature set, since the primary goal here is to get the device out to the market with acceptable level of features. More features are progressively added in subsequent releases of the device and the management application therefore should be capable of being enhanced easily so that it can expose these additional features as well.

Second, the value of the EMS is in allowing the user to interact with the device in a natural and easy manner (Several years ago, Apple used to run television commercials making fun of Microsoft user interface, letting viewers know that the productivity of a Mac user was significantly better because of it’s intuitive and easy to use interface!). This unfortunately often times gets overlooked. It appears the focus in developing an EMS is most often placed in getting the EMS done as quickly as possible (and as cheaply as possible, I might add) as opposed to getting it done right. Therefore in the case mentioned above, unfortunately there was no proper focus on the usability and it showed.  In our opinion this is a key reason why product management has to be very closely involved throughout the development of the management application.

Third important criteria that needs to be looked into when developing an EMS is what I would call as technology issues. The EMS should be designed and developed in a way that it is possible to incorporate newer technologies into the EMS as and when they become available. For example in the case of an EMS built with a webGUI, it should be possible to incorporate web 2.0 technologies easily.

Admittedly to balance the above mentioned requirements and also get the management application to the market in a timely and cost effective manner is definitely a significant challenge for the device manufacturer.

Dhyan Infotech Inc is a software and services company that helps Equipment Vendors differentiate their products in the market place. Dhyan's EMS technology, DNMS (Dhyan  Network/Element  Management System), is a component based Element Management System, which has been widely deployed by Dhyan's customers at many Service Providers both small and large

If you have questions on EMS or related topic that you would like to get answered or would like to write a guest column in this newsletter, please send an email to ems-energizer@dhyanit.com. This is a complimentary newsletter that was sent to you because of your interest in EMS. If you do not wish to receive this newsletter please click here to unsubscribe

In the last edition we covered some of the best practices in collecting data from the network elements (NE). Managing and analyzing the collected data from the NEs is even more challenging as it requires knowledge of various statistical techniques, database performance tuning methods, etc.,. This section provides some insights into the challenges involved in managing the collected data and some of the best practices in handling that data.

The number of performance metrics (PM) to be monitored by the EMS depends upon the complexity of the device. In this discussion, to get an idea of the challenges involved we will analyze a simple scenario wherein only 20 PM need to be monitored.

This number will increase ten fold if the number of devices that need to be managed increases to 1000 NE’s (which is not very uncommon in large networks) and we could be soon dealing with extremely large volumes of data (it is to be noted that in a lab environment the above mentioned volumes will not occur)

Even though we have millions of records in the database, when a user requests a report, the expectation is that the report would be available within a few seconds. However these reports have to be generated by processing many millions of records and if the request is not handled properly it could take a long time to generate the required reports. As an example to determine the total bandwidth that is being used by a video application the entire traffic data has to be processed by summing up all the bandwidth utilization entries for video traffic on a per-employee basis. Such reports might take few minutes to execute, which might be unacceptable from a user perspective.

To reduce the reporting time, EMS designers should have a very good knowledge of the database and be intimately familiar with database tuning methods. As a general rule, to improve the performance of a data base it is advisable to restrict the number of rows in a single database table to a few million records. Therefore as a basic step it makes sense to group the PM and create a single table with various metrics as individual columns. Even though this is a fundamental rule that people use when they design an EMS from scratch, many times it does not happen when frameworks are being used to build an EMS. By default frameworks typically store each metric in a separate row which would require the use of custom tables instead of generic tables.

Another common principle used to limit the database size is to purge the data that is older than a few days; however this may not be desirable since users might want to get historical reports. The issue of databases becoming large and unwieldy could be overcome by splitting the data into multiple tables. However, identifying the right approach to splitting the data is the key in differentiating a successful reporting engine from those that fail. Following are two ways in which the data could be split:

1. Creating a table for each day/week depending on the amount of data being collected
2. Creating a new table once a certain size is reached

Even though it appears that this approach might solve the performance problem, it does not because of the following reasons:

For example if we need a report that tells us how much bandwidth was used for various services (video, audio, voip, etc.,) by each user over the last month then we have to go through the following steps

1. Identify the individual table(s) which contains the data (it is quite possible that the data may be spread across multiple tables)
2. Frame the query and execute it across all the individual tables
3. Merge the results from the multiple queries by grouping the usage on a per user basis.

Unfortunately, it is quite possible that the time taken to do the above steps will end up taking up more time and thus defeating the purpose of splitting the data across multiple tables.

In some cases, it becomes practically very difficult to frame a query across two tables (for Instance if we have to get the bandwidth usage of the Top 10 users, we need the entire data set to derive the right report), the above mentioned reason is why most common reporting engines do not have great support for generating reports from multiple tables.

A far better alternative that solves this problem is to make use of a Trending Engine (explained below), which helps to get historical reports quickly and easily.

In order to manage the data and to process it into meaningful information, modern EMS solutions have a Trending Engine that generates aggregated statistics from the collected data. The Trending Engine typically runs in the background and periodically generates required statistical information about the collected data. For instance a Trending Engine that runs hourly will crunch all the 12 data points (data collected every 5 minutes) collected in an hour and generate an hourly bin (which contains information such as minimum, maximum, average etc., for the 12 data points). Similarly a daily bin might have the statistical measures for the collected data for the entire day. Intelligent Trending Engines also provide the option of generating multiple data points within a single day (say for every 2 hours, 4 hours etc.,) to ensure that there are more data points available so that meaningful statistical analysis can be performed.

Thus by using the Trending Engine, the actual collected data can be purged after a week or so because we could always reconstruct the data (required for generating the monthly report) from the hourly bin/daily bin that was generated by the trending engine. In addition the use of trending engine also allows users to generate the reports in a very short period of time