ChatGPT for Text Analytics: Why It Might Not Be Your Best Choice
Spending tireless hours meticulously coding open-ended respondent data , seas of text blurred into an incoherent mess. You've been there, and so have I.
It's a time-consuming task that one might reasonably assume could be solved with ChatGPT.
Unfortunately, it’s not quite that simple.
Understanding the Limitations of ChatGPT in Qualitative Coding
The Inaccuracy of ChatGPT
While it's remarkable that conversation models like ChatGPT can provide swift responses, this speed often compromises the accuracy of the output. LLMs are prone to hallucinations, which means that the summaries they provide of the themes can contain significant errors.
In addition to outright errors, there is also a lot of variability in the output -- ask ChatGPT to summarize your verbatims 5 different times and you will get 5 different summaries.
In short, the themes that ChatGPT says are present are frequently different from what a researcher would identify.
No way to verify
This challenge is made worse because ChatGPT is a black box and there is no way to verify whether the outputs from ChatGPT are accurate without coding the entire data set yourself and comparing the results, which obviously defeats the point.
So not only is the system prone to making things up, but it's impossible to efficiently verify when it's doing so.
The Lack of Contextual Understanding
Apart from accuracy issues, ChatGPT struggles with a fundamental understanding of conversation context. It's easy to get amazed by its ability to mimic human-like conversation. However, it often misses the nuances and subtleties that embody human interaction.
This means ChatGPT does not understand the goal or context of the research. It often identifies themes that are not particularly helpful, while missing themes that a researcher would understand as much more important.
This ability to craft a strong code frame that reflects the goals and context of the research is something humans remain uniquely good at.
Large Datasets can’t be Processed by ChatGPT
Due to ChatGPT upload and message limits and the limitations of the length of the context window it can remember, processing large amounts of data all at once is a challenge.
So at a very basic level, it simply cannot process the scale of data needed for most cases in a remotely timely or cost effective manner.
The Inability to Label Every Response
If you ask ChatGPT to identify themes in a data set, it will list a set of themes it says are present, but it does not actually label every response with the presence or absence of every theme.
This means that it is not able to tell you the exact percentage of respondents that mentioned a theme, let alone allow you to see how that theme varies across segments or conduct more advanced statistical analyses as you can with a fully labeled data set.
Why Fathom Is the Superior Alternative for Text Analytics
The challenges posed by ChatGPT in qualitative coding - its inaccuracy, inability to verify results, lack of contextual understanding, challenges with large datasets, and limitations in labeling responses - highlight the need for a more robust and reliable solution.
This is why we built Fathom.
Ensuring Accuracy with Human Review
Fathom integrates human review into its process, ensuring the accuracy of the data analysis. Unlike the AI-driven approach of ChatGPT, Fathom's human element adds a layer of precision and reliability, crucial for qualitative coding.
Verifiability and Transparency
Transparency is key in qualitative analysis. Fathom offers full transparency, allowing users to verify all verbatims associated with any code. This feature directly addresses the 'black box' issue of ChatGPT, providing researchers with the confidence that their data is accurately represented.
Contextual Understanding Through Expert Review
Fathom efficiently leverages expert human guidance to structure the code frame to the specific context and goals of your research. This ensures that the themes identified are relevant and valuable, unlike the often misplaced themes identified by ChatGPT.
Handling Large Volumes of Data
Where ChatGPT struggles with large datasets, Fathom excels. It is designed to handle any volume of responses, from a few hundred to hundreds of thousands, making it suitable for a wide range of projects, from small-scale studies to large-scale research.
Comprehensive Dashboard for Advanced Analysis
Fathom doesn't just identify themes; it also provides a fully labeled dataset with a dashboard that includes summaries, providing you the most actionable take-aways found across your responses - and the option to export and integrate into your BI. This enables detailed frequency, segmentation, and other advanced statistical analyses, offering a depth of insight that ChatGPT cannot match.
Fathom - Your Reliable Partner in Text Analytics
While ChatGPT may offer the allure of a quick fix, its limitations in accuracy, verification, context adaptation, volume barriers and detailed analysis make it unsuitable for rigorous qualitative coding.
The time you supposedly could have saved with ChatGPT is wasted wrangling with data and prompting.
Fathom, with its blend of human expertise and advanced analytics, offers a reliable, transparent, and comprehensive alternative in a truly delightful experience. Because analyzing rich, opend-ended, data should be the most valuable part of your research and the best part of your day.
Whether you're dealing with hundreds or thousands of responses, Fathom equips you with the tools needed for precise, context-aware, and in-depth text analytics.
