Back to blog

Why you don't need huge sample sizes for meaningful insights


Written by

Published March 12, 2020 · Updated June 8, 2020

Why you don't need huge sample sizes for meaningful insights

In a world where data is king, businesses are dealing with a lot of information. Indeed, the Piwik PRO suite helps companies in data-sensitive industries handle customer data responsibly. You can build accurate profiles of your customers, research granular user behaviour patterns, track the customer journey, and integrate first-party data from various online and offline sources.

But sometimes, the prevailing idea is the more the data, the better. Whilst this is often valid, it ignores the obstacle that many companies – especially when starting out – have limited volumes of data with which to make critical decisions about their growth strategy. The dream is to have a wealth of data and a whole team of analysts to sift through it, but the reality is that many organisations don’t have this luxury.

In this article, I will show why you needn’t be paralysed by your limited data volumes. I’ll outline 4 reasons why any amount of data can be useful for your business growth strategy – if it’s meaningful.

Four Reasons Why Small Datasets Have Big Value

1: Because “Small Data” has been pivotal, even for huge organisations

The term, “small data”, was first coined by Martin Lindstrom, who argues that every single human action and emotion can reveal unexpected insights. He advocates watching, listening, and being present with your audience in order to understand small nuances in behaviour, attitudes, and habits.

Lindstrom worked with LEGO to reverse their decline, speaking to customers to understand why young people were losing interest. He spoke to an 11 year-old boy who pointed to his worn-out trainers as the thing he was most proud of – because it was evidence of his skateboarding mastery. This was a lightbulb moment, and one of the stories that convinced LEGO to emphasise the challenge of using their product.

Instead of making bigger bricks and easier constructions, LEGO handed power back to the kids and encouraged them to become masters of their craft. This tapped into what kids wanted – ownership and pride about what they create. This insight wasn’t gleaned by sifting through vast quantities of data, but by observing clues that indicate emotions and desires, and by connecting those to a value proposition.

2: Because simply asking the right questions can uncover valuable insights

Naturally, when there is less quantity, it makes sense to focus on quality. For businesses without access to huge pools of data, qualitative research is the route to securing meaningful insights about an audience pain point and/or your potential solution. Indeed, you can still learn what you need to learn by conducting interviews and asking the right people the right questions – in the right way.

You might want to apply this method when testing people’s experience of a product or service (particularly in its early stages), or when understanding more about your target audience in order to build personas. Alternatively, when you want to know the why behind the what.

As this article by User Interviews outlines, there is a method to screening and interviewing participants for a study. Firstly, you should clearly define your intended participants – beyond broad demographics. And during screening, avoid giving too much away with leading questions or background to your study. Finally, try to seek expressive people who will give deeper answers, and incentivise their participation.

But how do you get the most insight out of a research interview? Here are some tips:

  1. Structure with warm-up questions, “big” questions, and cool-down questions
  2. Encourage storytelling through “why?” and “tell me more about…” questions
  3. Follow up on responses to learn more, rather than sticking to the script rigidly
  4. Beware of your own product-centric mindset when you’re preparing the interview
  5. Keep the research goal in mind at all times, even when going off on tangents

Ultimately, an interview is an opportunity to get deep qualitative data, but method is still important in order to get access to the hidden insecurities, challenges, concerns, wishes, and habits. It’s not as simple as getting in a room and having a conversation – you need to have a robust structure to your approach.

3: Because the law of diminishing returns applies to usability testing

A usability test is when you evaluate the ease of use of a product – typically with a digital interface such as an app or a website. These tests tend to occur with real users, rather than with developers or testers.

According to this excellent article by Jakob Nielsen, renowned computer scientist, PhD, and Principal at Nielsen Norman Group, you only need 5 participants for a meaningful usability study. He advocates spending extra budget on more tests, rather than on recruiting more participants for a single large study. For solely quantitative studies, he says that 20 users will allow for statistically significant numbers.

These are both realistic numbers of participants for any business to get hold of, yet Nielsen says that many researchers are tempted to add bigger volume to make their studies (and resulting conclusions) more credible to internal stakeholders. This is a trap, because as costs go up the return actually tails off. He argues that for usability studies tied to iterative design, you can get great results from small-n testing.

4: Because web analytics tools can help you identify patterns without millions of users

You don’t need big data to reach meaningful conclusions from your web analytics. For example, if you’re running a Google Ads campaign that gets a lot of clicks but no conversions. You don’t need masses of data to prove that your offer isn’t satisfying the searcher. In fact, this should become clear quickly: either you’re advertising on the wrong keyword set, or your landing page doesn’t instil confidence in the user.

Tools like Piwik PRO can teach you how users interact with your website, app, or digital platform – right from the start of your journey as a business. You don’t need millions of users, because you can install this type of software at early stages – when fast learning and product iteration is the priority before scaling growth. Whether you’re pre-product-market-fit or beyond, this continuous learning will help you optimise the user flow and map the whole customer journey as your user base grows.

And once you start building a strong user base (from which you can learn and improve even more), you do not need to crunch colossal numbers. You can keep your marketing strategy on point, improve retention, and enhance customer service using the same information – entry and drop-off points, conversion metrics, funnels, user flow charts etc.

Summary: Final Thoughts

Big data became a buzz amid stories of huge companies crunching the numbers to reach amazing conclusions. However, smaller businesses don’t have access to such a wealth of data – and even if they did, they’d need an army of data scientists to make sense of it.

Instead, it’s entirely legitimate to rely on smaller samples and deep qualitative insights to reach meaningful conclusions. And sometimes this isn’t just about making the best of a bad situation – as the LEGO story shows, sometimes the tiniest details can turn around the fortunes of the biggest organisations. The lesson is clear: human insights matter.


Oren Greenberg

Growth marketer and founder of the Kurve consultancy in London, UK. He helps startups and corporate innovation projects scale their growth, and he has been featured in leading marketing blogs and international press.

See more posts by this author

Core – a new plan for Piwik PRO Analytics Suite

Privacy-compliant analytics, built-in consent management and EU hosting. For free.

Sign up for free