营销研究中的 AI–人类混合模式:把大语言模型(LLM)当作合作者 · Empirical methods paper: two replication studies (qualitative interviews and a quantitative survey) with GPT-4, benchmarked against original human studies 实证方法论文:用 GPT-4 复制两项研究(定性访谈与定量问卷),并以原始人类研究为基准比较 · Ch 2, 8, 6.5, 13 · 30 min
Arora, N., Chakraborty, I., & Nishimura, Y. (2025). AI–human hybrids for marketing research: Leveraging large language models (LLMs) as collaborators. Journal of Marketing, 89(2), 43–70. https://doi.org/10.1177/00222429241276529
| Pages 页码 | What to read | 读什么 |
|---|---|---|
| 43–44 | Abstract and Table 1: the claim and where an LLM fits in research (5 min). | 摘要和表 1:核心主张,以及 LLM 在研究流程中的位置(5 分钟)。 |
| 54–56 | LLM as analyst: sentence overlap, Table 12 theme recovery and expert choices. This is the part we cite (8 min). | LLM 当分析员:句子重合、表 12 主题找回率和专家选择。这是我们要引用的部分(8 分钟)。 |
| 62, 65 | Table 16 and the context results: what few-shot and RAG fix and what they do not (4 min). | 表 16 和背景信息结果:少样本和 RAG 能改善什么、不能改善什么(4 分钟)。 |
| 67 | Limitations: bias, human in the loop, no generalisation (3 min). | 局限:偏见、人工把关、不能推广(3 分钟)。 |
Pages · 页码 43–45
Pages · 页码 45
Pages · 页码 46
Pages · 页码 46–49
Pages · 页码 49–51
Pages · 页码 51–54
Pages · 页码 54–56
Pages · 页码 56–57
Pages · 页码 57–59
Pages · 页码 59–62
Pages · 页码 62–65
Pages · 页码 65–66
Pages · 页码 66–67
| Exhibit 图表 | How to read it | 怎么看 |
|---|---|---|
| Table 1 (p. 44) | Adoption framework. Rows are research stages; columns are exploratory, descriptive and causal research. Each cell says how an LLM can help. | 采用框架。行是研究阶段,列是探索性、描述性和因果研究。每格说明 LLM 能如何帮忙。 |
| Table 2 (p. 46) | Prior studies compared on five ticks. Only this article ticks qualitative, quantitative, human benchmark and context. | 用五项打勾比较既有研究。只有本文同时勾选定性、定量、人类基准和背景信息。 |
| Figure 1 (p. 47) | System architecture. Personas go in the system role, questions in the user role; answers pass validation checks or are asked again. | 系统架构。人设放在系统角色,问题放在用户角色;回答须通过校验,否则重问。 |
| Table 3 and Figure 2 (p. 48) | Zero-shot, few-shot, RAG and fine-tuning side by side, and how RAG retrieves outside text into the prompt. | 并列比较零样本、少样本、RAG 和微调,并图示 RAG 如何把外部文本检索进提示。 |
| Table 5 (p. 50) | Who does each task (guide, sample, respondents, moderation) in the all-human condition and the four hybrids. | 列出全人类条件和四种混合模式中,每项任务(提纲、样本、受访者、主持)由谁负责。 |
| Table 8 (p. 52) | Worked example of LLM moderation. A first answer scores 71.25, below the threshold, so a probe follows; the second scores 83.75. | LLM 主持的实例。第一次回答得 71.25 分,低于门槛,于是追问;第二次得 83.75 分。 |
| Table 10 (p. 53) | Mixed-effects results. Read the "LLM generated" row: positive and significant for depth and insightfulness only. | 混合效应模型结果。看「LLM 生成」一行:只有深度和洞察力为正且显著。 |
| Figure 3 (p. 54) | How the expert assessment worked: data source × analyst gives four summaries per question, judged by experts. | 专家评估流程:数据来源 × 分析者,每题得四份摘要,再由专家评审。 |
| Table 12 (p. 56) | Theme recovery and new themes per question. The mean row gives 96%, 86% and 77% recovery. | 每道题的主题找回率和新主题比例。均值行显示找回率为 96%、86% 和 77%。 |
| Table 13 (p. 56) | Each judge's reason for the best summary. Useful for what experts value: examples, storytelling, balance. | 每位评审选择最佳摘要的理由。可看出专家重视:例子、故事性、详略平衡。 |
| Figure 4 (p. 57) and Figure 11 (p. 65) | Road maps for qualitative and survey research. White blocks are human-only, dark grey LLM-only, light grey hybrid. | 定性研究和问卷研究的路线图。白色块为纯人类,深灰为纯 LLM,浅灰为混合。 |
| Table 15 (p. 59) | Demographics of the 605 human respondents, copied into the synthetic personas. | 605 名人类受访者的人口特征,用来建立合成人设。 |
| Figures 5–8 (pp. 59–62) | Human vs. synthetic means, spreads and distributions. Synthetic bars have shorter error bars and bunch at 4–5. | 人类与合成数据的均值、离散度和分布对比。合成数据误差线更短,集中在 4–5 分。 |
| Table 16 (p. 62) | Average absolute gaps from human data for LLM1–LLM3. Smaller is better; the correlation column shows the biggest gain. | LLM1–LLM3 与人类数据的平均绝对差。越小越好;相关性一列改善最大。 |
| Figures 9–10 (pp. 63–64) | Spread and correlation heat maps. LLM3's correlation map looks most like the human map. | 离散度与相关性热力图。LLM3 的相关图最像人类数据的相关图。 |
| Term 术语 | Meaning | 解释 |
|---|---|---|
| large language model (LLM) 大语言模型 | A very large text model, such as the GPT family, that predicts and generates language. | 像 GPT 系列这样的超大文本模型,能预测并生成语言。 |
| AI–human hybrid AI–人类混合模式 | A workflow where the LLM does repetitive steps and a human supervises and decides. | 由 LLM 做重复步骤、由人类监督和决策的工作方式。 |
| synthetic respondent 合成受访者 | An LLM answering questions while playing a given persona. | 按照指定人设扮演受访者来回答问题的 LLM。 |
| persona / system role 人设/系统角色 | Background traits given to the model that shape all its later answers. | 预先交给模型的背景特征,决定它之后所有的回答。 |
| zero-shot prompting 零样本提示 | Asking the model to do a task with no examples. | 不给任何示例,直接让模型完成任务。 |
| few-shot learning 少样本学习 | Putting a few examples, or earlier answers, into the prompt. | 在提示中放入少量示例或之前的回答。 |
| retrieval-augmented generation (RAG) 检索增强生成 | Fetching relevant passages from a knowledge base and adding them to the prompt. | 从知识库检索相关段落,加进提示后再生成。 |
| fine-tuning 微调 | Further training a model's parameters on task-specific data. | 用特定任务的数据继续训练模型参数。 |
| embedding 嵌入向量 | Text turned into a list of numbers that captures its meaning. | 把文本转成一串能代表其含义的数字。 |
| cosine similarity 余弦相似度 | A closeness score between two embeddings; higher means closer meaning. | 两个嵌入向量之间的接近程度;分数越高,含义越接近。 |
| theme recovery 主题找回率 | Share of the all-human themes that a hybrid process also found. | 混合流程也找到的全人类主题所占的比例。 |
| response heterogeneity 回答异质性 | How much answers vary across respondents; measured by standard deviation. | 不同受访者回答之间的差异程度,用标准差衡量。 |
| internal consistency 内部一致性 | Whether related survey items correlate as they should. | 相关的问卷题目之间是否有应有的相关性。 |
| mixed-effects model 混合效应模型 | A regression that allows for repeated ratings by the same evaluator or question. | 能处理同一评价者或同一题目多次评分的回归模型。 |
| Value 数值 | What it measures | 含义 | Page 页 |
|---|---|---|---|
| $84.3 billion | Size of the marketing research industry in 2023 | 2023 年营销研究行业规模(843 亿美元) | p. 43 |
| 75% | Agreement between human and LLM perceptual maps in prior work (over 75%) | 既有研究中人类与 LLM 感知图的一致率(超过 75%) | p. 46 |
| 10 | New personas the LLM generated in the recruitment condition | 招募条件下 LLM 生成的新人设数量(10 个) | p. 51 |
| 80 | Quality score threshold below which the LLM moderator probes | LLM 主持人追问的质量分门槛(80 分) | p. 51 |
| 250 | Prolific participants recruited to rate answers | Prolific 招募的评分参与者人数(250 人) | p. 52 |
| 1,000 | Total evaluations from the Prolific study | Prolific 研究的总评价次数(1,000 次) | p. 52 |
| 12 years | Schooling needed to read LLM-generated text | 读懂 LLM 生成文本所需的教育年限(12 年) | p. 52 |
| .680 | Increase in depth score for LLM answers | LLM 回答在深度得分上的提升(.680) | p. 53 |
| .498 | Increase in insightfulness score for LLM answers | LLM 回答在洞察力得分上的提升(.498) | p. 53 |
| 40–45 minutes | Time a human analyst needs for one question and its answers | 人类分析员处理一道题及其回答所需时间(40–45 分钟) | p. 54 |
| 20 | Summaries produced in the expert assessment (5 questions × 4 bins) | 专家评估中产生的摘要数量(20 份,5 题 × 4 组) | p. 54 |
| 35 | Sentences humans highlighted on average | 人类平均标出的句子数(35 句) | p. 55 |
| 19 | Sentences the LLM highlighted on average | LLM 平均标出的句子数(19 句) | p. 55 |
| .78 | Average cosine similarity of human and LLM highlighted sentences | 人类与 LLM 所标句子的平均余弦相似度(.78) | p. 55 |
| 96% | Theme recovery when the LLM analysed human data | LLM 分析人类数据时的主题找回率(96%) | p. 55 |
| 23% | New themes found when the LLM analysed human data | LLM 分析人类数据时发现的新主题比例(23%) | p. 55 |
| 86% | Theme recovery when humans analysed LLM data | 人类分析 LLM 数据时的主题找回率(86%) | p. 55 |
| 77% | Theme recovery in the all-LLM process | 全 LLM 流程的主题找回率(77%) | p. 55 |
| 605 | Respondents in the original 2019 survey | 2019 年原问卷的受访者人数(605 人) | p. 58 |
| 304 | Synthetic respondents who rated the resealable bag | 评价可重封袋装的合成受访者人数(304 人) | p. 59 |
| .40 | Correlation recovery gap for the zero-shot model (LLM1) | 零样本模型(LLM1)的相关性还原差距(.40) | p. 62 |
| .11 | Correlation recovery gap for few-shot plus RAG (LLM3) | 少样本加 RAG(LLM3)的相关性还原差距(.11) | p. 62 |
| 16 | Respondents whose interview transcripts formed the RAG knowledge base | 其访谈记录构成 RAG 知识库的受访者人数(16 人) | p. 64 |
| .7 | How much more extreme LLM answers were on a five-point scale | LLM 回答在五分量表上更极端的幅度(约 .7 分) | p. 65 |
Written by the group from the full text. Numbers and page references were checked against the source. Licensed cases and reports are summarised, not reproduced.
由小组根据全文整理,数字与页码已对照原文核对。授权案例与报告只作摘要,不转载原文。