Stack of code

Improving product analytics with Elasticsearch

It’s an exciting time to be a developer. Concepts and traditions that used to be firmly in the domain of software development, like Agile and the underlying principles of growth hacking, have made their way into operations and management workflows. In our little corner of the economy (the ‘agency space’), I’ve noticed what we offer has become equal parts product development and service delivery.

While it’s true we are providing creative and development services, the way we execute our deliverables and evaluate our performance is actually more reminiscent of a product development lifecycle. We collect requirements, strategize on possible solutions we can deliver, actualize these solutions, and document the internals in what is effectively a vehicle owner’s manual, but personalized for your own website/software solution. One need not worship at the altar of growth hacking to understand the value in systematically measuring the performance of a product’s features and, even more importantly, reacting to the evidence with new fixes, features, and offerings.

Gartner’s IT glossary contends that product analytics is:

a specialized application of business intelligence (BI) and analytical software that consumes service reports, product returns, warranties, customer feedback and data from embedded sensors to help manufacturers evaluate product defects, identify opportunities for product improvements, detect patterns in usage or capacity of products, and link all these factors to customers.

In this modern information economy, products are no longer just tangible, manufactured items; product analytics has evolved in practice, yet the spirit of the definition remains true.

One platform embodying this new age information economy is Elasticsearch by Elastic — an open-source search and analytics engine based on Apache’s Lucene that has taken the software world by storm.

Elastic Search

The unique value of Elasticsearch is the ability to handle both structured and unstructured data simultaneously. With this platform, documents are indexed according to a strategy defined by the user and the complementary tools in the ELK stack–like Logstash (parse and perform ETL from a variety of sources), Beats (lightweight data shippers sending system logs to Elasticsearch clusters), and Kibana (extensible data visualization dashboard)–provide an end-to-end solution for establishing an analytics pipeline.

Due to its versatility, the ELK stack has been leveraged for a wide variety of projects. Anything you can imagine, from server monitoring, to social media analysis of Game of Thrones tweets, is within the realm of possibility with this platform.

Server Log Breakdown

Server log monitoring to improve performance, preempt attacks, and prevent downtime (courtesy of Saavius)


Game of Thrones Analytics

Analysis of Game of Thrones tweets (courtesy of

You may be thinking “so what?!,” but any product analytics team worth their salt knows the value in the following capabilities:

1. UX improvements: Is our UI and IA conducive to users performing our ‘critical event’ (“an action that users take within your product that aligns closely with your core value proposition” Source: Amplitude’s Product Analytics Playbook)?

2. Infrastructure analysis and auditing: Are we equipped to deliver the best possible experience to users? Why or why not? (Amazon Web Services originated from a tremendous capability to scale infrastructure during the holiday season, a result likely due to an investment in enhanced logging & analytics)

3. Engagement and retention: How effectively are we moving users through the conversion funnel? What holes exist in the funnel? How are our ads performing in concert with our other initiatives?

4. Operational: How can we prioritize improvements, endeavors, and requirements resulting from 1-3? Which changes are most critical?

With the ELK stack at your disposal, the possibilities are endless. Consider for a moment all the logs generated by your product: the Apache and MySQL logs on your server, the system.log files on all your organization’s machines, the WiFi & Bluetooth logs on your IoT devices, even the commerce.log file chronicling all the App/iTunes Store interactions taking place on your MacBook. With a little creative engineering, these logs can paint a comprehensive picture of the state-of-affairs at any level of your organization, network, or product.

The ELK stack empowers organizations to filter, segment, and aggregate information to their hearts content. Truly forward-thinking enterprises, like Netflix, eBay, Facebook and Verizon, have already adopted the open-source platform. We believe your organization (regardless of size; after all, it’s a FREE software!) would be wise to follow suit. Let’s talk about how analytics can help you move the needle.

Joe Haaga, Former Full Stack Developer at Grafik
Joseph Haaga

Related Articles

Humans vs. AI: Four contests reveal key traits AI can’t copy
The question of “man versus machine” has fascinated people for centuries. Many works of literature across cultures and history involve competition between humans and gadgets, from folk tales such as John Henry to movies such…
Read More
What’s different about Google’s GA4?
Google Analytics has been the industry-standard web analytics tool for more than a decade, helping businesses of all sizes to measure the performance of their websites and gain insights into their audience’s behavior. But, as…
Read More
The AI invasion: Keep calm and create on
If you’re as chronically online as I am, you couldn’t open a social app in December without seeing artificial intelligence portraits of your friends and peers. Many of the portraits come from the app Lensa,…
Read More
Top considerations for your 2023 marketing strategy
The past year saw marketing spending return to near pre-pandemic levels as consumers regained confidence. But the landscape remains complex, ever-changing, and competitive. So, what can help you rise above the noise? Here are three…
Read More