How (not) to Scale Elasticsearch for Data Analytics!


Duration: 25 mins
Aishwarya Sankaravadivel
Senior Software Engineer, PayPal

Search is ubiquitous - From booking a cab on Ride sharing platforms to searching for a job on LinkedIn, Elasticsearch has emerged as a prominent solution for extensive data analytics and searches. Due it’s distributed nature and ability to offer powerful visualizations, it serves as a key tool for deriving actionable insights. And, the recent advancements in Elasticsearch using ML stands out due to its simplicity in data analytics.

While Elasticsearch’s out of the box defaults works for smaller implementations, the nuances of handling a massive deployment in order to scale for petabytes of data and billions of ingests & searches per day requires a deeper knowledge on Elasticsearch internals.

In this talk, we will cover:

  • Learnings from our experiences in managing and scaling enterprise grade
  • Elasticsearch clusters with specific details around the things that can go wrong and how we could mitigate them.
  • Our inhouse security plugin that saved us 1 Million$ licensing cost.
  • Gaining actionable insights using ML
  • Demo of the features that we built on the top of Elasticsearch for scaling it.

You may also be interested in

50 mins
Definition of Ready & Done - A Guide to Achieving Predictability

Delivering software often takes longer than we anticipate. Why is that? Part of the reason is not understanding the nuances...

180 mins
Modern Software Development

Our industry never stops changing, but sometimes those changes are trivial and fluffy and we can ignore them. Sometimes they...

25 mins
How Non-violent Communication Can Help Keep the Peace on your Team

Non-violent communication will help you communicate with your coworkers in a manner that enables productivity and helps you understand how...

25 mins
Writing Professionally

The most important thing you do in your job is write. It's in every email you send, every commit you...

180 mins
Leading a Team of Subject Matter Experts with Confidence

As a leader, it is impossible to be an expert on all aspects of your delivery - this is why...

25 mins
Uncovering your Personal Values

We regularly review our code and attend retros, but what about tracking and reviewing our personal identity? We all know...