When developers or architects hear “Elastic Search (ES)” or think about using it, they often associate it with “adding search functionality” and label it as a “Search Engine.”
When dealing with database access performance, a basic plan would be an evaluation of indexes, leveraging replicas to spread the load, and splitting read from writes. A more experienced answer may entail a deep dive into how the information is being used and developing NoSQL solutions leveraging Hadoop (Hive), MongoDB, Postgres (Redshift), or MongoDB to offload reads.
In the many situations where I’ve seen a “data performance” problem, the technology department at least took the first steps of adding an additional service layer for search and updating their database architecture as mentioned above — but they miss one really cool idea.
I’m talking about leveraging your already existing search service, and not only using it for “searching data,” but also as a flattened index of your most common “read” data. I don’t have official metrics, but from my own experience when comparing a well-indexed database versus an ES representation of that same index, I’ve found ES performs significantly better — while still providing the benefits of using Elastic Search. You now have search capability, very fast indexes of your front-end data, and — with the addition of Kibana — a data warehouse for basic Business Intelligence (BI) and other analytics with no real increase in “tech overhead.”
This doesn’t mean that data warehousing solutions like Redshift, Spark, or BigQuery are not needed. They also play a crucial role in scalable architecture — solving problems around very complex queries and high-performance on a magnitude of even more data. They are not mutually exclusive.
The point I’m trying to make is that there is often a middle-ground to providing a solid solution to a very common problem without having to commit to a data warehouse solution that will have a financial and resource impact. Since you most likely have a “Search Engine,” you may as well expand how you currently use that service. The same DevOps engineers that currently manage this are usually more than capable of building a worker process that flattens out the data and pushes your search engine to handle near real-time updates. (And there are several open source libraries across most programming languages.)
If it’s later determined through use and/or product requirements that ES / Kibana won’t suffice as a data warehouse, you’ll be in a better position to make the right decision. There’s no loss — you’ll still use this solution as a representation of your customer-facing data.
Note: Although I’m saying Elastic Search, just about any “Search Engine” can be used in the way I’ve described.
Moral of the story: if you need a data warehousing solution but don’t have a team of data scientists and engineering DevOps with that specialty, then look at what you may already have as a solution: Elastic Search. Move as many of your data reads (especially client-side) to ES and leverage Kibana to build your dashboard. You will gain enormous benefits with the least cost and fastest time to market.