← All sources · 返回文献列表

9. AI–Human Hybrids for Marketing Research: Leveraging Large Language Models (LLMs) as Collaborators

营销研究中的 AI–人类混合模式:把大语言模型(LLM)当作合作者 · Empirical methods paper: two replication studies (qualitative interviews and a quantitative survey) with GPT-4, benchmarked against original human studies 实证方法论文:用 GPT-4 复制两项研究(定性访谈与定量问卷),并以原始人类研究为基准比较 · Ch 2, 8, 6.5, 13 · 30 min

Arora, N., Chakraborty, I., & Nishimura, Y. (2025). AI–human hybrids for marketing research: Leveraging large language models (LLMs) as collaborators. Journal of Marketing, 89(2), 43–70. https://doi.org/10.1177/00222429241276529

The paper in 60 seconds · 60 秒读懂

Why we read it · 为什么要读

20-minute reading path · 20 分钟阅读路线

Pages 页码What to read读什么
43–44Abstract and Table 1: the claim and where an LLM fits in research (5 min).摘要和表 1:核心主张,以及 LLM 在研究流程中的位置(5 分钟)。
54–56LLM as analyst: sentence overlap, Table 12 theme recovery and expert choices. This is the part we cite (8 min).LLM 当分析员:句子重合、表 12 主题找回率和专家选择。这是我们要引用的部分(8 分钟)。
62, 65Table 16 and the context results: what few-shot and RAG fix and what they do not (4 min).表 16 和背景信息结果:少样本和 RAG 能改善什么、不能改善什么(4 分钟)。
67Limitations: bias, human in the loop, no generalisation (3 min).局限:偏见、人工把关、不能推广(3 分钟)。

Abstract and Introduction · 摘要与引言

Pages · 页码 43–45

Background: Text Analysis and Large Language Models · 背景:文本分析与大语言模型

Pages · 页码 45

LLMs and Marketing Research (prior work) · LLM 与营销研究(既有文献)

Pages · 页码 46

Operationalization: Prompts and Incorporating Context · 操作化:提示词与加入背景信息

Pages · 页码 46–49

Study 1 (Qualitative): Design and Evaluation Criteria · 研究 1(定性):设计与评估标准

Pages · 页码 49–51

Study 1: Evaluation of Data Generation · 研究 1:数据生成的评估

Pages · 页码 51–54

Study 1: Qualitative Data Analysis by Human–LLM Hybrids · 研究 1:人机混合的定性数据分析

Pages · 页码 54–56

LLM Road Map for Qualitative Research · LLM 定性研究路线图

Pages · 页码 56–57

Study 2 (Quantitative): Survey Design · 研究 2(定量):问卷设计

Pages · 页码 57–59

Study 2: Results of the Zero-Shot Model · 研究 2:零样本模型的结果

Pages · 页码 59–62

Incorporating Context: Few-Shot Learning and RAG · 加入背景信息:少样本学习与 RAG

Pages · 页码 62–65

LLM Adoption Road Map for Surveys · 问卷研究的 LLM 采用路线图

Pages · 页码 65–66

Conclusions and Limitations · 结论与局限

Pages · 页码 66–67

Figures and tables · 图表解读

Exhibit 图表How to read it怎么看
Table 1 (p. 44)Adoption framework. Rows are research stages; columns are exploratory, descriptive and causal research. Each cell says how an LLM can help.采用框架。行是研究阶段,列是探索性、描述性和因果研究。每格说明 LLM 能如何帮忙。
Table 2 (p. 46)Prior studies compared on five ticks. Only this article ticks qualitative, quantitative, human benchmark and context.用五项打勾比较既有研究。只有本文同时勾选定性、定量、人类基准和背景信息。
Figure 1 (p. 47)System architecture. Personas go in the system role, questions in the user role; answers pass validation checks or are asked again.系统架构。人设放在系统角色,问题放在用户角色;回答须通过校验,否则重问。
Table 3 and Figure 2 (p. 48)Zero-shot, few-shot, RAG and fine-tuning side by side, and how RAG retrieves outside text into the prompt.并列比较零样本、少样本、RAG 和微调,并图示 RAG 如何把外部文本检索进提示。
Table 5 (p. 50)Who does each task (guide, sample, respondents, moderation) in the all-human condition and the four hybrids.列出全人类条件和四种混合模式中,每项任务(提纲、样本、受访者、主持)由谁负责。
Table 8 (p. 52)Worked example of LLM moderation. A first answer scores 71.25, below the threshold, so a probe follows; the second scores 83.75.LLM 主持的实例。第一次回答得 71.25 分,低于门槛,于是追问;第二次得 83.75 分。
Table 10 (p. 53)Mixed-effects results. Read the "LLM generated" row: positive and significant for depth and insightfulness only.混合效应模型结果。看「LLM 生成」一行:只有深度和洞察力为正且显著。
Figure 3 (p. 54)How the expert assessment worked: data source × analyst gives four summaries per question, judged by experts.专家评估流程:数据来源 × 分析者,每题得四份摘要,再由专家评审。
Table 12 (p. 56)Theme recovery and new themes per question. The mean row gives 96%, 86% and 77% recovery.每道题的主题找回率和新主题比例。均值行显示找回率为 96%、86% 和 77%。
Table 13 (p. 56)Each judge's reason for the best summary. Useful for what experts value: examples, storytelling, balance.每位评审选择最佳摘要的理由。可看出专家重视:例子、故事性、详略平衡。
Figure 4 (p. 57) and Figure 11 (p. 65)Road maps for qualitative and survey research. White blocks are human-only, dark grey LLM-only, light grey hybrid.定性研究和问卷研究的路线图。白色块为纯人类,深灰为纯 LLM,浅灰为混合。
Table 15 (p. 59)Demographics of the 605 human respondents, copied into the synthetic personas.605 名人类受访者的人口特征,用来建立合成人设。
Figures 5–8 (pp. 59–62)Human vs. synthetic means, spreads and distributions. Synthetic bars have shorter error bars and bunch at 4–5.人类与合成数据的均值、离散度和分布对比。合成数据误差线更短,集中在 4–5 分。
Table 16 (p. 62)Average absolute gaps from human data for LLM1–LLM3. Smaller is better; the correlation column shows the biggest gain.LLM1–LLM3 与人类数据的平均绝对差。越小越好;相关性一列改善最大。
Figures 9–10 (pp. 63–64)Spread and correlation heat maps. LLM3's correlation map looks most like the human map.离散度与相关性热力图。LLM3 的相关图最像人类数据的相关图。

Key terms · 关键术语

Term 术语Meaning解释
large language model (LLM)
大语言模型
A very large text model, such as the GPT family, that predicts and generates language.像 GPT 系列这样的超大文本模型,能预测并生成语言。
AI–human hybrid
AI–人类混合模式
A workflow where the LLM does repetitive steps and a human supervises and decides.由 LLM 做重复步骤、由人类监督和决策的工作方式。
synthetic respondent
合成受访者
An LLM answering questions while playing a given persona.按照指定人设扮演受访者来回答问题的 LLM。
persona / system role
人设/系统角色
Background traits given to the model that shape all its later answers.预先交给模型的背景特征,决定它之后所有的回答。
zero-shot prompting
零样本提示
Asking the model to do a task with no examples.不给任何示例,直接让模型完成任务。
few-shot learning
少样本学习
Putting a few examples, or earlier answers, into the prompt.在提示中放入少量示例或之前的回答。
retrieval-augmented generation (RAG)
检索增强生成
Fetching relevant passages from a knowledge base and adding them to the prompt.从知识库检索相关段落,加进提示后再生成。
fine-tuning
微调
Further training a model's parameters on task-specific data.用特定任务的数据继续训练模型参数。
embedding
嵌入向量
Text turned into a list of numbers that captures its meaning.把文本转成一串能代表其含义的数字。
cosine similarity
余弦相似度
A closeness score between two embeddings; higher means closer meaning.两个嵌入向量之间的接近程度;分数越高,含义越接近。
theme recovery
主题找回率
Share of the all-human themes that a hybrid process also found.混合流程也找到的全人类主题所占的比例。
response heterogeneity
回答异质性
How much answers vary across respondents; measured by standard deviation.不同受访者回答之间的差异程度,用标准差衡量。
internal consistency
内部一致性
Whether related survey items correlate as they should.相关的问卷题目之间是否有应有的相关性。
mixed-effects model
混合效应模型
A regression that allows for repeated ratings by the same evaluator or question.能处理同一评价者或同一题目多次评分的回归模型。

Numbers we can cite · 可引用的数字

Value 数值What it measures含义Page 页
$84.3 billionSize of the marketing research industry in 20232023 年营销研究行业规模(843 亿美元)p. 43
75%Agreement between human and LLM perceptual maps in prior work (over 75%)既有研究中人类与 LLM 感知图的一致率(超过 75%)p. 46
10New personas the LLM generated in the recruitment condition招募条件下 LLM 生成的新人设数量(10 个)p. 51
80Quality score threshold below which the LLM moderator probesLLM 主持人追问的质量分门槛(80 分)p. 51
250Prolific participants recruited to rate answersProlific 招募的评分参与者人数(250 人)p. 52
1,000Total evaluations from the Prolific studyProlific 研究的总评价次数(1,000 次)p. 52
12 yearsSchooling needed to read LLM-generated text读懂 LLM 生成文本所需的教育年限(12 年)p. 52
.680Increase in depth score for LLM answersLLM 回答在深度得分上的提升(.680)p. 53
.498Increase in insightfulness score for LLM answersLLM 回答在洞察力得分上的提升(.498)p. 53
40–45 minutesTime a human analyst needs for one question and its answers人类分析员处理一道题及其回答所需时间(40–45 分钟)p. 54
20Summaries produced in the expert assessment (5 questions × 4 bins)专家评估中产生的摘要数量(20 份,5 题 × 4 组)p. 54
35Sentences humans highlighted on average人类平均标出的句子数(35 句)p. 55
19Sentences the LLM highlighted on averageLLM 平均标出的句子数(19 句)p. 55
.78Average cosine similarity of human and LLM highlighted sentences人类与 LLM 所标句子的平均余弦相似度(.78)p. 55
96%Theme recovery when the LLM analysed human dataLLM 分析人类数据时的主题找回率(96%)p. 55
23%New themes found when the LLM analysed human dataLLM 分析人类数据时发现的新主题比例(23%)p. 55
86%Theme recovery when humans analysed LLM data人类分析 LLM 数据时的主题找回率(86%)p. 55
77%Theme recovery in the all-LLM process全 LLM 流程的主题找回率(77%)p. 55
605Respondents in the original 2019 survey2019 年原问卷的受访者人数(605 人)p. 58
304Synthetic respondents who rated the resealable bag评价可重封袋装的合成受访者人数(304 人)p. 59
.40Correlation recovery gap for the zero-shot model (LLM1)零样本模型(LLM1)的相关性还原差距(.40)p. 62
.11Correlation recovery gap for few-shot plus RAG (LLM3)少样本加 RAG(LLM3)的相关性还原差距(.11)p. 62
16Respondents whose interview transcripts formed the RAG knowledge base其访谈记录构成 RAG 知识库的受访者人数(16 人)p. 64
.7How much more extreme LLM answers were on a five-point scaleLLM 回答在五分量表上更极端的幅度(约 .7 分)p. 65

What it means for our report · 对报告的意义

Limits and where it does not fit · 局限与不适用之处

Questions for the group · 小组讨论问题

← The co-evolution of two Chinese mobile short video apps: Parallel platformization of Douyin and TikTokResale's Next Chapter: How Fashion and Luxury Brands Can Win in the Secondhand Market →

Written by the group from the full text. Numbers and page references were checked against the source. Licensed cases and reports are summarised, not reproduced.
由小组根据全文整理,数字与页码已对照原文核对。授权案例与报告只作摘要,不转载原文。