How (not) to Scale Elasticsearch for Data Analytics!


Duration: 25 mins
Aishwarya Sankaravadivel
Senior Software Engineer, PayPal

Search is ubiquitous - From booking a cab on Ride sharing platforms to searching for a job on LinkedIn, Elasticsearch has emerged as a prominent solution for extensive data analytics and searches. Due it’s distributed nature and ability to offer powerful visualizations, it serves as a key tool for deriving actionable insights. And, the recent advancements in Elasticsearch using ML stands out due to its simplicity in data analytics.

While Elasticsearch’s out of the box defaults works for smaller implementations, the nuances of handling a massive deployment in order to scale for petabytes of data and billions of ingests & searches per day requires a deeper knowledge on Elasticsearch internals.

In this talk, we will cover:

  • Learnings from our experiences in managing and scaling enterprise grade
  • Elasticsearch clusters with specific details around the things that can go wrong and how we could mitigate them.
  • Our inhouse security plugin that saved us 1 Million$ licensing cost.
  • Gaining actionable insights using ML
  • Demo of the features that we built on the top of Elasticsearch for scaling it.

You may also be interested in

50 mins
Do You Know Da Wae

We build development teams based on individual ability to write code but development of a software project of any significance...

25 mins
How Non-violent Communication Can Help Keep the Peace on your Team

Non-violent communication will help you communicate with your coworkers in a manner that enables productivity and helps you understand how...

25 mins
Remote Working - Is this the New Normal?

In this talk, Nainesh discusses the pros and cons of this new way of working that we have all had...

180 mins
Modern Software Development

Our industry never stops changing, but sometimes those changes are trivial and fluffy and we can ignore them. Sometimes they...

50 mins
Definition of Ready & Done - A Guide to Achieving Predictability

Delivering software often takes longer than we anticipate. Why is that? Part of the reason is not understanding the nuances...

30 mins
Up Your Engineering Game: Adopting the Good Parts of Startup Culture

We all know the hallmarks of startup culture: “act first, apologize later,”  “work hard, play hard,” "growth, growth and more...