Scientific Foundations of Chatbot-Assisted Evaluation Tasks
1. Idea Generation Task
This task is guided by the Geneplore framework (Finke et al., 1992), a cognitive model of creativity that distinguishes between two core processes:
- Generative processes: Creating mental representations or “preinventive structures.”
- Exploratory processes: Evaluating, transforming, and applying those representations in meaningful ways (Moreau & Dahl, 2005).
The chatbot evaluates the user across four dimensions that align with this generative–exploratory cycle, including collaboration style, critical thinking, novelty-feasibility balance, and decision ownership.
2. Problem-Finding Task
Based on the Augmented Problem-Finding Framework (Sabbah & Li, 2025), this task emphasises:
- Problem Discovery: Trend analysis, hypothesis generation, and weak signal detection.
- Problem Definition: Reframing problems, testing assumptions, and refining scopes.
ChatGPT is instructed to assess framework adherence and the depth of human–AI collaboration.
3. Tactical and Strategic Decision-Making Tasks
Adapted from Kline (1996), these tasks involve ranking decision actions. Deviation from expert rankings determines performance (lower score = better). The model analyses:
- Tactical reasoning patterns
- Confidence calibration
- AI reliance
- Suggestions for future improvement
4. Communication Skills Task
This task employs a 7-item scale adapted from de Vries et al. (2009), known as the PRESENT model:
- Preciseness: Clarity, conciseness, grammar
- Reflectiveness: Thoughtful, analytical expression
- Expressiveness: Eloquence and confidence
- Supportiveness: Relational and encouraging tone
- Emotionality: Emotional expressiveness or tension
- Niceness: Cheerfulness and friendliness
- Threateningness: Intimidation or manipulation (reverse scored)
5. Negotiation Task
Drawing from Ganesan (1993), this task focuses on four evaluative dimensions:
- Clarity of Position
- Negotiation Strategy: Competing, collaborating, compromising, avoiding, or accommodating
- Persuasiveness
- Active Listening
The chatbot is required to deliver detailed feedback with a focus on growth and learning.
6. Ethical Decision-Making Task
This task uses Rest’s (1986) Four-Component Model to assess moral reasoning:
- Moral Sensitivity: Awareness of stakeholders and consequences
- Moral Judgment: Balancing competing ethical principles
- Moral Motivation: Prioritising moral values over gain
- Moral Character: Integrity and action in ambiguity
The model guides feedback on justifications, confidence, AI influence, and ethical reasoning.
7. Risk-Based Decision-Making Task
Informed by Kahneman and Tversky’s (1979) certainty theory, this task evaluates how participants perceive and respond to uncertainty. The chatbot analyses:
- Risk preferences and consistency
- Confidence–risk alignment
- AI reliance in uncertain contexts
- Feedback for improving future decisions under uncertainty
Model and Parameters
The underlying chatbot model is OpenAI’s GPT-4o-mini, with a temperature set at 0.6 to balance coherence and creativity in dialogue.
References
- De Vries, R. E. et al. (2009). Communication Research, 36, 178–206.
- Finke, R. A., Ward, T. B., & Smith, S. M. (1992). Creative Cognition. MIT Press.
- Ganesan, S. (1993). Journal of Marketing Research, 30(2), 183–203.
- Kahneman, D., & Tversky, A. (1979). Econometrica, 47(2), 263–291.
- Kline, T. J. B. (1996). Personality and Individual Differences, 21(6), 1077–1079.
- Moreau, C. P., & Dahl, D. W. (2005). Journal of Consumer Research, 32(1), 13–22.
- Rest, J. R. (1986). Moral Development. Praeger.
- Sabbah, J., & Li, F. (2025). Innovation, 1–34.