Session Name: Log Management: From grep to Full-Text Search and Back
Logs are everywhere, but they have gone through an interesting development over the years: (1) grep: This works well as long as you have a single instance to search on. Once you need to SSH into many machines and try to piece together the results of multiple grep commands, things tend not to work that well anymore. (2) Splunk: Centralizing those logs and letting users search through them with a piped language in Splunk is the logical step to fix that issue. However, the more data you centralize, the slower this will get. (3) ELK: The solution to that idleness is using full-text search. Elasticsearch, in combination with Logstash and Kibana (plus Beats), gave logs a major performance boost. But at what cost? (4) Loki: Reducing the scope and going back to a smart data structure combined with grep gives Loki the possibility to reduce costs while still providing good performance. (5) Closing the gap: So what are the tradeoffs between the different systems, and are they potentially closing some gaps between performance and cost? Join the discussion after the talk for "the right amount" of features, costs, and speed.
Philipp lives to demo interesting technology. Having worked as a web, infrastructure, and database engineer for over ten years, Philipp is now a developer advocate and community team lead in EMEA at Elastic — the company behind the Elastic Stack consisting of Elasticsearch, Kibana, Beats, and Logstash. Based in Vienna, Austria, he is constantly traveling Europe and beyond to speak and discuss open source software, search, databases, infrastructure, and security.