Stack of code

Improving product analytics with Elasticsearch

It’s an exciting time to be a developer. Concepts and traditions that used to be firmly in the domain of software development, like Agile and the underlying principles of growth hacking, have made their way into operations and management workflows. In our little corner of the economy (the ‘agency space’), I’ve noticed what we offer has become equal parts product development and service delivery.

While it’s true we are providing creative and development services, the way we execute our deliverables and evaluate our performance is actually more reminiscent of a product development lifecycle. We collect requirements, strategize on possible solutions we can deliver, actualize these solutions, and document the internals in what is effectively a vehicle owner’s manual, but personalized for your own website/software solution. One need not worship at the altar of growth hacking to understand the value in systematically measuring the performance of a product’s features and, even more importantly, reacting to the evidence with new fixes, features, and offerings.

Gartner’s IT glossary contends that product analytics is:

a specialized application of business intelligence (BI) and analytical software that consumes service reports, product returns, warranties, customer feedback and data from embedded sensors to help manufacturers evaluate product defects, identify opportunities for product improvements, detect patterns in usage or capacity of products, and link all these factors to customers.

In this modern information economy, products are no longer just tangible, manufactured items; product analytics has evolved in practice, yet the spirit of the definition remains true.

One platform embodying this new age information economy is Elasticsearch by Elastic — an open-source search and analytics engine based on Apache’s Lucene that has taken the software world by storm.

Elastic Search

The unique value of Elasticsearch is the ability to handle both structured and unstructured data simultaneously. With this platform, documents are indexed according to a strategy defined by the user and the complementary tools in the ELK stack–like Logstash (parse and perform ETL from a variety of sources), Beats (lightweight data shippers sending system logs to Elasticsearch clusters), and Kibana (extensible data visualization dashboard)–provide an end-to-end solution for establishing an analytics pipeline.

Due to its versatility, the ELK stack has been leveraged for a wide variety of projects. Anything you can imagine, from server monitoring, to social media analysis of Game of Thrones tweets, is within the realm of possibility with this platform.

Server Log Breakdown

Server log monitoring to improve performance, preempt attacks, and prevent downtime (courtesy of Saavius)


Game of Thrones Analytics

Analysis of Game of Thrones tweets (courtesy of

You may be thinking “so what?!,” but any product analytics team worth their salt knows the value in the following capabilities:

1. UX improvements: Is our UI and IA conducive to users performing our ‘critical event’ (“an action that users take within your product that aligns closely with your core value proposition” Source: Amplitude’s Product Analytics Playbook)?

2. Infrastructure analysis and auditing: Are we equipped to deliver the best possible experience to users? Why or why not? (Amazon Web Services originated from a tremendous capability to scale infrastructure during the holiday season, a result likely due to an investment in enhanced logging & analytics)

3. Engagement and retention: How effectively are we moving users through the conversion funnel? What holes exist in the funnel? How are our ads performing in concert with our other initiatives?

4. Operational: How can we prioritize improvements, endeavors, and requirements resulting from 1-3? Which changes are most critical?

With the ELK stack at your disposal, the possibilities are endless. Consider for a moment all the logs generated by your product: the Apache and MySQL logs on your server, the system.log files on all your organization’s machines, the WiFi & Bluetooth logs on your IoT devices, even the commerce.log file chronicling all the App/iTunes Store interactions taking place on your MacBook. With a little creative engineering, these logs can paint a comprehensive picture of the state-of-affairs at any level of your organization, network, or product.

The ELK stack empowers organizations to filter, segment, and aggregate information to their hearts content. Truly forward-thinking enterprises, like Netflix, eBay, Facebook and Verizon, have already adopted the open-source platform. We believe your organization (regardless of size; after all, it’s a FREE software!) would be wise to follow suit. Let’s talk about how analytics can help you move the needle.

Joe Haaga, Former Full Stack Developer at Grafik
Joseph Haaga

Related Articles

How to make the most of peer-to-peer fundraising on social media
Is your nonprofit organization feeling the negative effects of a turbulent economy? Are you looking for new and innovative ways to fundraise? With over 4.59 billion users worldwide, it’s no wonder that nonprofits are leaning…
Read More
Google Analytics 4: Why you shouldn’t wait to migrate
Another directive has been cast down from our internet overlords: Google has announced it will be turning off Google Universal Analytics on July 1, 2023 and replacing it with Google Analytics 4, or GA4. This…
Read More
The ABCs of NFTs: A Crypto Crash Course
Although the 2022 Super Bowl is already well in our rearview mirror, trending topics and frequent Google searches reveal that a select group of ads shown during the biggest sports, entertainment, and marketing spectacle of…
Read More
making an online donation
The digital donor: Trends in fundraising behind the screen
2021 was the year of adjusting for nonprofits as the world trudged through the second year of the pandemic. Fundraising efforts continued to lean heavily on virtual events and peer-to-peer campaigns, but they also leveraged…
Read More