Integrated Event Logging for Disparate Applications

Ibridge’s Distributed Event Logging package (IDEL) provides a uniform event log mechanism for distributed applications. IDEL can be used to integrate previously independent applications with very little programming effort.

Background:  Event logging is an integral component of any application development and maintenance, but with distributed implementations it becomes of even greater concern. In distributed computer systems, the dispersed elements must be closely monitored to provide a desired level of service and availability. Observation of global events, with a uniform interface and timestamp, is required for:

  • distributed application debugging
  • comparison of disparate applications performance
  • incident investigation
  • the first step in building fault-tolerance for distributed systems
  • determining global predicates in the distributed application
  • it is the first task when sequencing distributed application tasks

The primary considerations for IDEL were:Scalability

  • Reliability
  • Performance
  • Cross-Platform Support
  • Rapid Integration into existing Application frameworks
  • Support for Event/Action matching
  • Flexibility

Scalability  (The Source/Sink Event Model)A major design goal of IDEL was that it had to be scalable. IDEL must accommodate the addition of applications without the need to modify or restart existing applications. In the context of IDEL, we use the term “source” to talk about a process that is providing data to the event logging system, irrespective of whether it is a client or a server in the traditional sense. When a process is the recipient of event data, we say it is a data sink. Data flows "downstream" from sources to sinks. Given this terminology, we may define the dissemination of event logging data as follows:

    • Source: application which generates an event
    • Sink: application which uses the event

Reliability: Data loss should be avoided and the system should continue to run after disruptions, such as a loss of power or a system crash. IDEL has been designed to minimize the necessity for human intervention.

Performance: It will periodically occur that large numbers of events will be generated virtually simultaneously in a large distributed application (consider the case of a failure in one member that all the other members then detect and report to the central server). Therefore, the issue of I/O performance must be carefully considered in the overall event logging design.

Cross-platform support:  IDEL can provide a uniform event log for applications that are executing on different operating system platforms. These applications may even be written in different programming languages. Currently, the IDEL library supports C, C++, Fortran and Java.

Rapid integration into existing applications:  IDEL is very simple to use. The application library consists of a basic set of functions that developers can call from their code.

Support for Event/Action Sequences:  Pattern matching is used in determining the fate of each event once it has reached a destination sink (i.e. which of a set of general actions are applied to the event). The IDEL sink software uses a configuration file to set up the table of patterns and associated actions for events. There are two action types, FILE and PIPE. The FILE action will simply log the event to the central repository. The PIPE action will propagate the event to a known PIPE. This feature is used to easily and efficiently integrate custom application task scheduling with the global event log system.

Flexibility: Arbitrary run-time reconfiguration of IDEL is easy. The source/sink design of IDEL also makes it possible to distribute the load of ingesting data from multiple sources simultaneously.

Example Applications of IDEL

  • Combining event logs from different database implementations and computer systems platforms.
  • Viewing client application events with server events.
  • Integration of previously disparate applications.
  • Determination of previously unknown global predicates.
  • Basis for building fault-tolerance in distributed applications.
  • Sequencing of distributed application tasks.

Copyright Ibridge Inc., 2010