Current State

Application mapping technologies have continued to mature throughout 2008. Due to the economic issues facing businesses around the world, most vendors in this space have spent 2008 shoring up their current offerings and improving upon features already found in the product. As the economic environment begins to improve over the next 12 – 24 months, vendors will start to add additional functionality into their mapping solutions. These innovative features will include better visibility into virtual server environments, management of high capacity computing environments such as grid computing environments. SOA will start to become a prominent technology that will be implemented in many of the Fortune 500, so firms will require mapping technology in order to have a better understanding of what components are running in their SOA environment and what the interdependencies of the components are.

The major players in this space continue to be IBM with their Tivoli Application Dependency Discovery Manager (TADDM), EMC with their Smarts with their Application Discovery Manager, HP with the Application Mapping component built into their Business Availability Center solution, and Tideway Systems with their Tideway Foundation.

Players in space

IBM Tivoli Application Discovery and Dependency Mapper

IBM’s TADDM solution is a component of their Tivoli Systems Management framework. Although IBM will claim that you can use this solution standalone, the value that it provides is greatly diminished when not used with IBM Tivoli Monitoring. IBM’s solution is an agentless solution, in that it does not require any agents to function. Instead it uses privileged credentials for Unix and Microsoft Windows platforms. Again to derive full value from the solution, TADDM will require privileged access to several commands on the UNIX platform, such as LSOF and on Solaris platforms, the root equivalent permissions for PS so as to see all of the command line arguments from the ps command. In IBM’s defense, this is an issue with all platforms that utilize credentials to pull running process information.

EMC Application Discovery Manager

EMC’s Application Discovery Manager is an elegant solution in that it is an appliance based solution that can truly be a turn key installation. You deploy the appliance into your data center environment, plug it into a switch port and configure the appliance. One must then configure the switch port that the appliance is plugged into to span all of its traffic to that particular port. One that is done, it almost immediately begins assembling relationship information in what EMC calls passive network discovery. EMC utilizes a hybrid approach in that it has the ability to actively go discovery certain network components to “connect the dots” and build a fairly comprehensive application map. One of the critical issues with their solution is that it requires quite an investment in switches so that network engineering groups can span network traffic to a switch infrastructure. Since the solution inspects network packets

HP OpenView Discovery and Dependency Mapper

HP’s Discovery and Dependency Mapping solution utilizes credentials as well. This solution does an adequate job of discovering information about objects running in the environment through the use of elevated privileges on hosts. One of the main complaints of HPs solution is that there is quite a steep learning curve to learn the the TQL language of the solution. It is quite archaic and takes some time to get a group of resources up to speed on how to use the language.

Tideway Foundation

Tideway Foundation is an appliance based solution that also utilizes credentials and elevated privileges to log into and discovery running processes on hosts. Tideway’s solution comes with many commercial software packages already defined in their proprietary TPL or Tideway Pattern Language. Prior to the Foundation 7 release, Tideway utilized XML to create their models, so if one was somewhat fluent in XML, then you could easily write the code necessary to generate visual representations of running software. With release 7, Tideway moved to their proprietary TPL. Much like the other solutions that require credentials, Tideway utilizes elevated privileges to execute many of the commands in its discovery scripts.

Out of all the players in this space, Tideway has done the best job in terms of making their software available to users in order to conduct testing and proof of concept testing.  One can actually navigate to their website and download what the call a Tideway Foundation Community Edition.  This version runs as a VMWare virtual appliance and will allow you to deep scan up to 30 hosts.  To the best of my knowledge, this is the only vendor that has moved to this model of try before you buy.  The other major vendors will only provide this capability to their largest customers or in certain competitive situations.

Use Cases

Several use cases exist for this sort of technology. Several that I have been involved with include data center relocation and data center consolidation efforts. These technologies provide invaluable near real time information in terms of what hosts are communicating with other hosts. These tools provide great insight into the logical dependencies that exist in the data center and help data center architects and project management personnel mitigate the risks involved with a complex large scale data center relocation initiative.

Licensing exercises

These systems come in handy when it comes time to renegotiate a software contract or enter into a new enterprise license agreement with a vendor. With these tools, you now have a programmatic way of determining what software is running where, without having to use an agent to do so. One can then take this data and combine it with a traditional asset management solution to have very quantifiable numbers to use for negotiating these sort of contracts.

Automation of systems management technologies

I have also designed solutions where these technologies assist with the overall management of monitored devices and ensuring those devices are programmatically instrumented in various monitoring technologies. Integrations with tools such as IBM Tivoli’s Netcool, SL Corp’s RT View, and integrations with Netuitive Service Navigator are also under way.

Making sense of SOAs

These solutions have also proven that they can provide additional insight into the running processes and software that exist within a service oriented architecture (SOA). I have been involved with initiatives to also map the complete application portfolio of a firm, using Tideway as a critical component of that strategy.

Implementation Challenges

Implementing these solutions is not necessarily a walk in the park. As with any major IT Service management initiative, these is an element of planning and architectural discussions that need to take place prior to the first major roll out. In my experience, a pilot implementation of the technology is the best way to go. With a pilot, you can place the technology in your lab or development / test environment and evaluate how the technology works given the constraints of your environment. There will be several issues to overcome as there is with any implementation of this magnitude which involves discovering the configuration of devices on the network. With that being said, here are the top three issues that you will need to contend with if considering deploying this sort of technology.

Credentials and security

Since all of the technologies discussed here do not use an piece of software, or agent, to gather their information, they all use credentials or packet analysis or a combination of both. In most cases, some sort of elevated privileges will be required to gather the complete set of data that each of the vendors proport to gather. For example, on Sun Solaris, one of the commands that Tideway Foundation runs is a ps axe command. Sounds like a relatively harmless command but it is one that will require elevated privileges on a Sun Solaris 8 or 9 systems running the latest patch bundles, or on a Solaris 10 system by default. The technical reasons behind this are outside the scope of this article, but suffice it to say, if you want a complete set of running processes on a Solaris box, you will need to grant the “Tideway” user elevated privileges to run this command. Tideway is not alone, Tivoli’s Application Mapping solution has a long list of elevated commands that need to be run. Again, they don’t HAVE to be run, but you will not realize the full value of the solution without those elevated commands. HP’s solution has similar constraints on it as well.

For a Windows host, the same sort of issue exists, but in a different light. You will need to figure out the best way to deploy and Administrator equivalent credential for these tools to use as well. Since most of the rely on the Windows WMI interface to gather their statistics, there is usually a combination of local Admin rights and the appropriate either global or local group to be able to push the necessary rights out to hundreds if not thousands of hosts. Again, the technical details of this are beyond the scope of this document, but I would be more than happy to talk to anyone seriously considering the deployment of one of these technologies. Credentials will be one of the most challenging aspects of deployment for you and I have been through it.

Buy in from senior management

This goes without saying but it bears repeating. The deployment of these technologies will require a lot of support and various levels of interaction from senior leadership. Invariably you will run into a situation where a business unit or group will not want to push out elevated credentials to their systems. You will support from leadership to breakdown these barriers and work with the other business units through this process. In fact, try and find a senior leader that is willing to conduct a pilot of the technology on their systems. Once they see the level of detail that comes back and the potential for them to have a much better handle of their IT estate, they should become a strong advocate of deploying the techni9logy company wide.

Buy in from Application owners and application developers

Last but not least, you are going to need a lot of help and assistance from the application owners and developers. First of all, they will need to be on board with the whole scanning a host and detecting all of the running processes on it. DBAs are also going to need to buy off on the whole concept as well as systems administrators. They tend to be the group that will need the most convincing that the technology is not harmful and will actually help them in the long run. Most will come around eventually, especially when they can use the technology to help them during a major outage or troubleshooting event.

If you plan on doing any sort of application modeling with the technology, then you will most certainly need the experience and expertise of the application developers to help the application modeling team to build a pictorial representation of the application. You may be saying to yourself; wait a minute, I thought these tools did this all automatically. Well, they do, to some extent. They do go out and discover running processes on each of your Windows or Unix based hosts in your environment. They also will tell you what switches those hosts are plugged into (some of these tools require some customization and integration to make that happen). Then they will show you host to host network communication traffic and what processes are generating and receiving that traffic. What they will NOT do is magically tell you that a group of hosts running in a Veritas cluster are part of your order entry application for your Sales department. Nor will the tools magically tell you that a 1000 node compute grid is part of your algorithmic trading platform supporting your investment bank. These inferences require knowledge of the business as well as knowledge of the critical processes to run that order entry or trading platform mentioned above.

Conclusions

The level of insight and transparency that these technologies bring to the table is quite impressive. I have had some level of involvement with each of these technologies over the last several years. They all have their strengths and shortcomings. The Smart’s solution is an elegant and is the most turnkey of all the solutions mentioned here, but it would become quite expensive to deploy this technology across multiple data centers due to its dependency on switches and spanning traffic from a switch to a span port. Also, these packet sniffing technologies are also not going to be able to map HTTPS encrypted traffic. All of the technologies discussed in the article have some level of dependence on credentials and all of them ultimately are going to need some level of elevated privileges to gain the full value from the tool that the vendor has promised you. The single most important factor in choosing to use one of these technologies is to follow the tips identified above, especially when it comes to the credentials and engaging your firm’s security group EARLY in the proof of concept and pilot phase. Furthermore, don’t try to tackle your entire data center or enterprise all at once; it is too much to take on and coordinate at once. Use a crawl, walk, run methodology to get these technologies deployed. Start with a small pilot and get your feet wet. This provides an opportunity to operationalize the technology and get everyone comfortable with its use. Go through the exercise of modeling several applications so that you and leadership have an idea of what it will take to model applications. Once you have this knowledge, you will be able to make much better decisions in terms of how to roll the technology enterprise wide.

These technologies will continue to mature over the next several years. One could expect that the current recession and lack of capital spending by large enterprises will slow down the development efforts of firms like IBM, HP, EMC, and Tideway Systems, but these tools are still very robust in their current state and are invaluable in terms of assisting with green data center efforts and the ensuing IT data center consolidation that firms will be going through over the coming years due to the economic downturn and the need to reduce costs in the data center. Furthermore, there is almost certainly going to be a reinvigorated effort to better understand system to system communications as we move into an era of more regulation and oversight of financial services institutions in the coming years.