Case study
AI question rephrasing and survey bias
Researcher. Manuscript in preparation with Dr. Tina Sutton.
A between-subjects experiment testing whether survey questions rewritten by an AI to be neutral change what people report. This study is the scientific basis for Cogbias AI's technology.
Question wording biases survey answers, and expert review to catch it is manual and slow.
An AI rewrites biased items to be neutral. One answer flipped from 56% to 85% the other way.
Problem
The way a question is worded pushes the answer. Acquiescence, social desirability and availability all creep into survey items, and the usual fix is an expert reading every question by hand. That review is manual and slow.
What I ran
I ran a between-subjects experiment. One group answered the original, biased questions. The other group answered versions that an AI had rewritten to be neutral.
Participants then made a behavioral choice, allocating their own study time, so the survey answers could be checked against what people actually did.
Result
Responses changed significantly on 33% of the targeted questions.
On one question about study preferences, the biased wording suggested 56% of students preferred studying in a group. The rephrased wording showed 85% preferred to study alone.
Answers to the neutral questions lined up more closely with the study-time choices participants made afterwards.
33%of targeted questions changed significantly56% → 85%group study on the biased item; studying alone on the rephrased itemMethod
- Between-subjects experiment
- AI question rephrasing
- Behavioral allocation task
- Survey design
Targeted biases: acquiescence, social desirability, availability.
Manuscript in preparation: Altobelli, M., and Sutton, T. The Impact of AI-Powered Question Rephrasing on Cognitive Bias in Survey Responses.
Visuals
Survey item, before and after
IllustrationOriginal wording
Studying with a group helps you learn more, doesn't it?
YesNoBias detected
Studying with a group helps you learn more, doesn't it?
FlagAcquiescence cue: the question supplies its own answerRephrased to be neutral
When you study, do you prefer to work alone or with a group?
AloneWith a groupWhat people reported
Illustration. The question wording here is paraphrased. The 56% and 85% are real results from my between-subjects study (Altobelli and Sutton, manuscript in preparation). The manuscript is in preparation and is not posted here.