Log Analysis

Poirot is a set of tools and libraries for log collection and analysis

Dec 2013 - Present
InSTEDD, Data, Tools
3 min

1 /3

Client

Poirot is a set of tools and libraries for log collection and analysis developed for InSTEDD.

2 /3

Approach

Poirot includes loggers for multiple platforms, which post log entries to a log receiver that indexes them in an ElasticSearch backend, which can then be queried and navigated in a web frontend.

3 /3

Results

Activities

Poirot log entries can be grouped in activities. An activity is a set of log entries grouped by a certain criteria, such as being related to the same task or to the same web request. Activities can be hierarchical, and Poirot tracks the spawning of different activities across different threads, processes or even applications. This can be used for linking a background process activity with the web request that triggered it, or for complex interactions between processes in Erlang OTP applications.

Activity detail

Detail of a single activity, showing its process hierarchy

Metadata

Both log entries and activities can be enriched with arbitrary metadata. For example, a web request can be indexed with all the query parameters sent by the client, allowing for easy lookup of a request based on any of its parameters; or a modification to a sensitive resource in the application can be logged with detailed information on the user who altered it, thus using Poirot as an audit trail as well. Poirot indexes all metadata by default, and relies on the power of ElasticSearch queries to provide easy search functionality.

Storage

All data is stored in ElasticSearch indices. Poirot automatically creates one index per day, allowing for easy rotation of indices by making use of ElasticSearch snapshots. Each log entry is indexed with common fields such as `@message`, `@pid`, `@level`, `@source`, `@timestamp` and `@activity`; while applications keep track of a `@description`, `@pid`, `@source`, `@start` and `@end`, as well as any other metadata set by the client applications.

Open source

This project is Open Source, we invite you to collaborate and join us in the development of a better world through the use of technology.


https://github.com/instedd/poirot
 

Let's work together




Start your project