Word Embeddings - An Alternative and Efficient Approach to Search for Documents

Duration: 50 mins
Ananth Gundabattula
Senior Architect, Commonwealth Bank of Australia

Searching for documents in a collection is typically implemented via a TF/IDF principle in open source document search engines. However recent developments in the field of NLP has shown positive results in representing text into more concise vector representations as opposed to a bag of words construct. In addition to this, these approaches also add richness to the information models like taking care of analogies and semantics of the words. This talk would walk through an end to end data workflow to enable such a construct.

The first part of the session would describe the typical flow of how a search query is processed by default in any of the lucene powered search engines today. The concept of TF/IDF is also introduced in this part of the session.

The session then proceeds to describe the concept of word embeddings using a library like Facebooks fasttext.

Subsequently, a representative data pipeline is discussed as to how an incoming stream of data can be turned into vector representations and made amenable for searching with a few seconds of turn around time.

The session would close with a few references to the more recent developments in this space.

You may also be interested in

50 mins
Top 10 Algorithms for Coding Interviews

So we are all the best coders, but have you see the run time of the code you are writing?...

50 mins
Effecting Change—The Art Of Leading Teams

As leaders we want our teams to pursue great ideas and change directions to realize the goals. However, often we...

25 mins
Designers + Developers = Best Friends Forever?

How is the relationship between your design team and your development team? Is it highly functional? Or 'just professional'? Maybe...

50 mins
Mental Bookmarks and the Fractal Nature of Success

Good discussions are supposed to diverge from their intended path. Free association is a feature, not a bug, and helps...

50 mins
Systems Thinking

Albert Einstein once said — “We cannot solve our problems with the same thinking we used when we created them.” As...

50 mins
Sailing through Digital Transformation

The current health crisis is rapidly reshaping, for the better, what was already a key agenda for organizations in 2020...