Saturation score. How do we know how many interviews to run?

Right now you need to define the number of interviews you wish to run.

What we describe in this post is a method whereby our agents will run the necessary interviews to achieve topic saturation. It’s a much more cost effective solution and speaks to the core of what we’re building.

‍

A bit of  history

‍

Foundational Contributions:

‍

It's the swinging '60s, and Barney Glaser and Anselm Strauss are about to drop a research bomb with their book, "The Discovery of Grounded Theory." Imagine them as the dynamic duo of qualitative research, turning the tables on the stuffy old ways and saying, "Hey, let's make theories straight from the data!"

‍

The '90s Revolution:

‍

Fast forward to the '90s, and Janice M. Morse steps onto the scene with her masterpiece, "Critical Issues in Qualitative Research Methods," a treasure map guiding researchers on how to pick participants and decide on the size of their study group with style and substance. Janice was all about keeping things legit and making sure everyone played by the rules of good research.

‍

21st Century Shake-Up:

‍

As we zoom into the 21st century, enter Kathy Charmaz, wielding her "Constructing Grounded Theory". Kathy's all about building theories from the ground up, using real data and constantly revisiting her ideas. It's like she's saying, "Let's keep digging and refining until we strike gold." She brought a fresh, constructivist vibe to the party, making sure everyone understood that creating theories is a journey, not just a one-stop destination.

‍

Theoretical sampling drives our approach

‍

Theoretical sampling is a strategy in qualitative research, particularly in grounded theory, where the selection of participants, cases, or data to analyze is driven by the developing theory.

‍

When we say that analysis is "driven by the developing theory" in the context of qualitative research, especially within grounded theory methodology, it means that the process of data analysis is closely integrated with and informed by the evolving theoretical insights that emerge from the data itself. In other words, it’s an ongoing conversation, as it should be.

‍

Unlike probability sampling in quantitative research, which aims for generalisability through random selection, theoretical sampling focuses on deepening the emerging theory's conceptual development.

‍

In order to fulfil our vision of Continuous Insight, we need to aim towards saturation and branch out until we hit saturation again. Imagine agents having multiple conversations that will end when a saturation threshold is met.

‍

Getting to a saturation number

‍

Yes, you can use OpenAI's API (or any other LLM) to analyze interview transcripts and estimate a saturation score, although the process will be somewhat indirect. The LLMs’s don't directly compute a saturation score as it's a specific application not built into the model. However, it’s fairly straightforward to design a workflow that leverages the model's capabilities in text analysis, summarization, and topic extraction to approximate a saturation score.

‍

Here's how we do it (this is a simplified version):

‍

Step 1: Preprocess and Summarize Interviews

‍

First, our interview transcripts don’t need to be clean and organized. You might want to summarize each interview to focus on key points, especially if they are lengthy, but that’s it. We do this.

‍

Step 2: Extract Topics from Each Interview

‍

Next, we need to identify and list the main topics or themes from each interview summary.

‍

Simple prompt for topic extraction:

‍

"List the main topics or themes discussed in this interview summary:"
[Summarized Interview]



Step 3: Aggregate Topics and Identify New Ones

‍

Aggregate the topics from each interview and keep track of new topics that emerge with each subsequent interview. This step might be more manual or semi-automated, depending on your setup. You’re basically creating your codebook, as they say in qualitative research. This codebook is the basis for this algorithm.

‍

Step 4: Calculate Saturation Score

‍

The saturation score can be approximated by monitoring the rate at which new topics emerge as you process more interviews. A simple approach is to calculate the percentage of new topics in each subsequent interview compared to the total topics identified so far.

‍

Step 5:  Automate the Process

‍

To automate steps 2 and 3 we created a script that sends each interview (or summary) to the API with the prompt for topic extraction. Then, the script can compare the topics from each interview to those from previous interviews to track new topics.

‍

Example Python Script Outline:

‍

import openai

openai.api_key = 'your_api_key_here'

def extract_topics(interview):
   response = openai.Completion.create(
     engine="input your engine here",
     prompt=f"List the main topics or themes discussed in this interview summary:\\\\n{interview}",
     temperature=0.5,
     max_tokens=100,
     top_p=1.0,
     frequency_penalty=0.0,
     presence_penalty=0.0
   )
   return response.choices[0].text.strip().split('\\\\n')

# Assume `interview_summaries` is a list of your interview summaries
all_topics = set()
new_topics_per_interview = []

for summary in interview_summaries:
   topics = extract_topics(summary)
   new_topics = set(topics) - all_topics
   new_topics_per_interview.append(len(new_topics))
   all_topics.update(new_topics)

# Calculate saturation score as needed, e.g., by analyzing the trend in `new_topics_per_interview`



Step 6: Interpret the Saturation Score

‍

A decreasing trend in the number of new topics per interview indicates approaching saturation. When the number of new topics drops to zero or near-zero over several interviews, you've likely reached saturation.

‍

The additional logic involves creating a function to calculate the saturation score. This function takes as input the list of new topics per interview and outputs a score representing the degree of saturation. We define saturation as the point when less than 5% of the topics are new. Here is an example of what this function in Python:

‍

def calculate_saturation(new_topics_per_interview):
   total_topics = sum(new_topics_per_interview)
   saturation_points = [new_topics / total_topics for new_topics in new_topics_per_interview]

   saturation_score = 0
   for point in saturation_points:
       if point < 0.05:
           saturation_score += 1

   return saturation_score / len(saturation_points)



‍

This function calculates the proportion of new topics in each interview and checks if it's less than 5%. The saturation score is then the proportion of interviews that meet this criteria.

‍