Client
Poirot is a set of tools and libraries for log collection and analysis developed for InSTEDD.
Approach
Poirot includes loggers for multiple platforms, which post log entries to a log receiver that indexes them in an ElasticSearch backend, which can then be queried and navigated in a web frontend.
Results
Activities
Poirot log entries can be grouped in activities. An activity is a set of log entries grouped by a certain criteria, such as being related to the same task or to the same web request. Activities can be hierarchical, and Poirot tracks the spawning of different activities across different threads, processes or even applications. This can be used for linking a background process activity with the web request that triggered it, or for complex interactions between processes in Erlang OTP applications.
Activity detail
Detail of a single activity, showing its process hierarchy
Metadata
Both log entries and activities can be enriched with arbitrary metadata. For example, a web request can be indexed with all the query parameters sent by the client, allowing for easy lookup of a request based on any of its parameters; or a modification to a sensitive resource in the application can be logged with detailed information on the user who altered it, thus using Poirot as an audit trail as well. Poirot indexes all metadata by default, and relies on the power of ElasticSearch queries to provide easy search functionality.
Storage
All data is stored in ElasticSearch indices. Poirot automatically creates one index per day, allowing for easy rotation of indices by making use of ElasticSearch snapshots. Each log entry is indexed with common fields such as `@message`, `@pid`, `@level`, `@source`, `@timestamp` and `@activity`; while applications keep track of a `@description`, `@pid`, `@source`, `@start` and `@end`, as well as any other metadata set by the client applications.
Open source
This project is Open Source, we invite you to collaborate and join us in the development of a better world through the use of technology.
https://github.com/instedd/poirot