Qualitative data and why it matters more than before

Tim Offor, Co-Founder & CTO, Plaetos Group

In this series of articles about how you can use qualitative workplace data to generate deeper employee insights, I have one clear purpose: to get everyone who is responsible for data-led people management to take a second look at the vast trove of underused qualitative data sitting in their organization’s systems to improve their people insights. And truly lead through their people.

Thanks for tuning in.

Since it’s January 2023, a moment that looks like it’s going to be marked in history, I’m writing this series with the support of my ChatGPT “co-author”. We’re sharing this task transparently, so I’m putting ChatGPT’s contributions in italics. It’s a quasi-rigorous way of showing that what I have to say is not just my opinion, without laboring it.

My conclusion? Reports of (my obsolescence) are greatly exaggerated, to (further) misquote Mark Twain. Go to the end here if you’re interested in this exercise.

Why should you care about qualitative data for workforce intelligence?

Lots of reasons, but the chief one for this piece is that the power shift to workers in a digital world means the winners attracting and keeping top talent are those who listen and respond to their employees in a way that makes them feel “heard”.

Feeling heard is an intrinsic human need. Most often felt sharpest in the absence. In a corporate setting this absence shows up in churn numbers, persistent pockets of culture misalignment, recalcitrance, stubborn Diversity numbers, and at worst, toxic behavior harming individuals and corporate reputation.

...whoever felt heard when they answered survey questions. Who feels heard when they get performance feedback tracked by bots.

Contemporary workforce listening programs set up a false confidence that they can hear, but they can’t. At least not in a way that results in their people feeling heard. Think about it: whoever felt heard when they answered survey questions. Who feels heard when they get performance feedback tracked by bots.

Qualitative insight – and yes the clue is in the root of “quality” – is kernel to transforming a modern workforce listening program into a “hearing, reflecting and responding” program.

So what is qualitative data?

[The “oracle” of ChatGPT:] Qualitative data is a type of data that describes and explores the characteristics and attributes of a phenomenon. It is used to gain a deeper understanding of a topic, often by exploring experiences, attitudes, beliefs, and behaviors. Qualitative data is traditionally collected through methods such as interviews, focus groups, observation, and open-ended surveys, and is analyzed through techniques such as coding, categorizing, and identifying patterns and themes. Unlike quantitative data, which is numerical and can be measured, qualitative data is descriptive and subjective, providing rich and detailed information about a topic.

So that’s the traditional world view of qual data, very social research and very manual sounding. All that coding, categorizing and pattern identification sounds like – and is – hard, and somewhat arcane, work.

Contrast that to what ChatGPT has to say about Quantitative data:

Quantitative data is a type of data that is numerical and can be measured. It is used to quantify and describe the characteristics and attributes of a phenomenon, often through statistical analysis. Quantitative data is collected through methods such as surveys, experiments, and observations using standardized instruments and is analyzed using mathematical and statistical methods. It is used to test hypotheses, make comparisons, and generalize findings to a larger population. Quantitative data provides objective and verifiable information that can be easily compared and analyzed.

So, right from the outset, the biases of each are clear. One’s descriptive and subjective, the other objective and verifiable.

What is qualitative data good for?

[My mate:] Qualitative data is good for understanding opinions, attitudes, beliefs, behaviors, and experiences, as well as exploring and discovering new ideas and concepts. It is used to gain a deeper (my emphasis) understanding of a topic, to identify patterns and themes, and to provide context and detail.

If we’re to be effective, our job as people practitioners is to understand people, get to the heart of their “opinions, attitudes, beliefs, behaviors and experiences”. Understand them as people, their differences, their joys, their frustrations, their aspirations, their needs.

So, what is quantitative data good for?

Quantitative data is good for:

1. Testing hypotheses: Quantitative data can be used to test hypotheses and make statistical inferences about a population.

2. Generalizing findings: Quantitative data can be used to generalize findings to a larger population, making it possible to make broad statements about a phenomenon.

3. Measuring change over time: Quantitative data can be used to measure changes in variables over time, allowing for the examination of trends and patterns.

4. Making comparisons: Quantitative data can be used to compare groups and make statistical inferences about differences and similarities between groups.

5. Assessing relationships: Quantitative data can be used to assess the relationship between variables and to determine the strength and direction of these relationships.

6. Supporting decision making: Quantitative data can be used to inform decision making by providing objective and verifiable information.

7. Easily analyzed: Quantitative data can be easily analyzed using mathematical and statistical methods, allowing for efficient and accurate analysis.

Let’s zero-in on the last one: 7. Easily analyzed.

Quite honestly, isn’t that the truth. Quant is great because it’s easy. It’s why leadership teams and boards want “the number”: is it up or down? Are they happy or unhappy? They might ask why, but a number has done its job - it can’t answer why.

And that’s why – if you search the web for HR tech data and analytics solutions – you’ll find 10X the amount of solutions offering quant analysis as qual.

But let’s pause here – set aside the easy versus hard angle – and dig a little deeper on what it is we as people insight professionals are trying to learn, and how best to do that.

...it is predominantly quant data that informs really complex, nuanced business programs like culture, the equity and inclusion that actually shifts the dial on diversity numbers, stress and wellbeing and why people might leave for a competitor.

I went on to ask my new, worldly assistant about the limitations of our two data sources, and here’s what came back.

What are the limitations of qualitative data?

The limitations of qualitative data include:

1. Subjectivity: Interpretations and conclusions drawn from qualitative data may be influenced by the researcher's own biases and perspectives.

2. Lack of generalizability: Qualitative data may not be representative of a larger population, making it difficult to generalize findings to other populations or contexts.

3. Time-intensive: Qualitative data collection and analysis can be time-consuming and resource-intensive.

4. Potential for researcher bias: The process of collecting and analyzing qualitative data can be influenced by the researcher's own biases and perspectives.

5. Difficulty in quantifying results: Qualitative data may be difficult to quantify and compare, making it challenging to test hypotheses and make statistically significant conclusions.

6. Difficulty in replication: Qualitative research can be difficult to replicate due to the unique and complex nature of the data collected.

I’d summarize these as qual being less generalizable because of the small sample sizes it has traditionally relied on, so it is dangerous to generalize to a larger than research cohort population. Quant’s clearer advantage.

Also that the cognitive limitations of a researcher also presents a weakness in reliability, and so a further shakiness on qual data’s generalizability.

As both a scientist and social scientist I am all too familiar with the mutual contempt each discipline has for the other, each valuing their own discipline.

To be fair on qual, and lay the groundwork for discussion, let’s give quant a turn.

What are the limitations of quantitative data?

The limitations of quantitative data include:

1. Reduced context and detail: Quantitative data may lack the richness and depth of information that qualitative data provides, and may not fully capture the complexity of a phenomenon.

2. Limited scope: Quantitative data collection may be limited to specific variables and may not take into account other factors that may impact the results.

3. Reliance on self-reported data: Surveys and experiments may rely on self-reported data, which can be subject to biases and inaccuracies.

4. Artificiality of experimental conditions: Controlled experimental conditions may not accurately reflect real-world situations, limiting the generalizability of findings.

5. Difficulty in measuring abstract concepts: Some abstract concepts, such as attitudes and beliefs, may be difficult to quantify and measure accurately.

6. Relying on single methods: The use of a single method for collecting data, such as a survey, may limit the types of questions that can be answered and the perspectives that can be captured.

7. Oversimplification: The reduction of complex phenomena to numerical data can result in oversimplification and loss of important information.

Again, let’s zero-in on the last one: 7. Oversimplification.

So even quant has shortcomings in generalization. Because it’s not meant for depth, it risks oversimplification of complex issues.

Yet, the irony: it is predominantly quant data that informs really complex, nuanced business programs like culture, the equity and inclusion that actually shifts the dial on diversity numbers, stress and wellbeing and why people might leave for a competitor.

So what did my LLM assistant “think” was the best approach for our use case – understanding employees. Here’s what they came back with, unedited (although I’ve bolded some points for emphasis):

Is quantitative or qualitative analysis better for understanding employees?

It depends on the specific research question and the type of information being sought. Both quantitative and qualitative analysis have their strengths and limitations and can provide valuable insights into understanding employees.

Quantitative analysis is good for measuring and comparing variables and can be used to make generalizations about a population. It is useful for assessing employee attitudes, behaviors, and perceptions on a large scale and can provide a statistical basis for making decisions.

Qualitative analysis, on the other hand, provides a more in-depth and nuanced understanding of employees and can be used to explore their experiences, attitudes, beliefs, and motivations. It is useful for gaining a deeper understanding of complex issues and exploring the meaning and context behind employee behaviors.

In conclusion, a combination of both quantitative and qualitative analysis can provide a comprehensive understanding of employees, with each method complementing the other to provide a well-rounded and informed perspective.

A no brainer, and a B+ grade response.

“...developments in Natural Language Processing in newly commercial applications are able to put the quant behind qual…”

The world has moved on - Qual can now go head-to-head with Quant

There’s a fundamental fact that ChapGPT isn’t (yet) able to “know” to get to an A+ grade: developments in Natural Language Processing in newly commercial applications are able to put the quant behind qual, generating both deep – able to understand attitudes, feelings, behaviors – and broad – statistically valid – and so generalizable. In real-time, and without invading privacy. And getting more generalizable the more data it draws on.

Qualitative data is very useful for understanding complex workplace issues and establishing the meaning and context of employee behaviors (like churn, inclusion, engagement). But, its general use for employee insight has been held back by the effort and time involved in collecting, analyzing and reporting and concerns that it is neither objective nor generalizable.

What ChatGPT has completely missed – for now – is that the new technologies designed to tap into and make sense of the vast troves of workplace qualitative data – the language and metadata exhaust of the modern enterprise – mean those objections melt away.

NLP-driven passive listening tech for “deep listening” can generate fast, objective, quantifiable, replicable and generalizable insights from qualitative data that are every bit as useful - and very compatible - with quantitative methods for employee insight.

Afterword on ChatGPT

Playing with ChatGPT has been an interesting exercise (and I’m far from alone in this play), one that shows the limitations of Large Language Models (LLMs) that rely on historical knowledge for their “wisdom” in an area of rapid innovation.

LLMs are great for quickly compiling relevant information. The confidence over competence is becoming clearer and many people toy like I have here.

What ChatGPT does provide is a useful “bridge” between my experienced opinion and the venerable academic paper (TL;DR).

In a domain that you know well, LLMs can provide a foundation that you can work from but you will disagree with some of it. It sets up a different – and intriguing – way of working that should improve the speed and depth of technical writing providing the human co-author is willing to pull their weight.

---

Tim Offor is Co-Founder and CTO of Plaetos Group, which builds qualitative people insight software. The PlaetosEQ workforce listening platform specializes in analyzing and visualizing qualitative data - what people think, feel and believe about what and why - so large enterprise customers can get a more nuanced understanding of their company culture and get to alignment faster. Prior to founding Plaetos, Tim consulted to large companies on social strategy, social impact and building stronger relationships with stakeholders.