AI In Education
Context
ChatGPT is widely used for educational purposes. According to Kevin Roose, a technology columnist for The New York Times, ChatGPT can “deepen students’ understanding” of content (Roose, 2023). As generative AI tools become increasingly integrated into education, our group is particularly interested in the role of understanding ChatGPT 3.5 in aiding student comprehension and understanding of study materials.
Objectives
Through observation and analysis of student interactions with AI-generated responses, this study aims to assess and improve the usability of ChatGPT 3.5 within an educational context. Specifically, we sought to:
Analyze how generative AI like ChatGPT 3.5 aids students in comprehending complex study materials
Identify challenges and pain points associated with ChatGPT 3.5 usage
Explore the strategies users adopt when interacting with ChatGPT 3.5
Compile a report of data-driven design recommendations
Research Questions
What methods and processes do participants use when utilizing ChatGPT 3.5 as a summarization tool?
Explored pain points in using ChatGPT for creating and studying material.
Assessed whether students achieved desired outputs from the tool.
Participant Profiles
To understand students’ academic use of ChatGPT 3.5, we recruited eight University of Washington students (18–24). Most participants studied STEM subjects and regularly used AI for tasks like coding, summarization, and editing papers.
Methodology
Recruitment
Inclusion Criteria | Exclusion Criteria |
|---|---|
Age 18+ | Non-College Students |
College Students |
Logistics
Environment
The study was conducted on May 14th and May 16th in neighboring study rooms at Odegaard Library, University of Washington. Each session included a facilitator guiding tasks and a notetaker documenting observations. Materials and technology were tested beforehand to ensure a smooth process.
Ethical Considerations
Participants signed consent forms granting permission to participate and be recorded. These forms were printed and signed prior to the study.
Technology
Tasks
Upon participants’ arrival, the coordinator greeted participants and directed them to the proper rooms. We had participants conduct three tasks throughout the usability study of ChatGPT 3.5.
Task 1: Summarization Analysis
Imagine you are doing research for a course assignment and you come across an article that is relevant to your work. Please choose an article from the ones provided, unless you have brought your own. Take as much time as you need, and please verbalize your thoughts out loud about your reading strategies and overall approach during the process.
Task 2: Coding Assistance
After you have nished reading the article, please write a summary of the article using ChatGPT, and please let us know when you think the summary has reached a good point. Please verbalize your thoughts and approaches as you go through this task, and feel free to ask any questions that come to mind.
Task 3: Coding Assistance
Let’s say you have a quiz on the topic in 3-4 days. Your task is to prepare for that quiz by using ChatGPT to create review material that you can use to study. Think of what you would want to know for a quiz based on the article you have read and how you can condense that down in a way you can recall during the quiz. To reiterate, please verbalize your thoughts and approaches as you go through this task, and feel free to ask any questions that come to mind, and please let us know when you think the review material has reached a good end point.
Data Collection
Analyzed qualitative and quantitative data to identify key patterns and insights.
Qualitative Methods | Quantitative Methods |
|---|---|
Written Prompting | Number of Prompts |
Verbalized Thoughts | Task Duration |
Observations |
Data Analysis
The data was analyzed by coding the transcripts from usability studies, interviews, and survey. After coding, we conducted a thematic analysis to cross-reference participant data in order to group common themes
Key Findings
Information Validity
100% of participants struggled to assess the accuracy of ChatGPT 3.5’s responses
Prompting Issues
75% of users were uncertain about starting chats and using prompts effectively
Regenerated Responses
64% of participants found regenerated responses unsatisfactory and had difficulty understanding the tool’s capabilities
Interface Navigation
25% of users found long chat logs challenging to navigate and the regenerate button inaccessible
Recommendations
Enhance Information Transparency
Introduce indicators or explanations to help users better assess the validity of ChatGPT’s responses
Improve Prompt Guidance
Provide guidance or examples for effective prompting to reduce user confusion and improve task completion
Refine Regenerate Feature
Make the regenerate button more accessible and provide clearer feedback when responses are regenerated
Improve Navigation of Chat History
Redesign chat log navigation to help users quickly locate specific parts of the conversation and improve overall usability
Conclusion
This usability study revealed key challenges in how ChatGPT 3.5 supports students in gathering and analyzing educational content, including issues with information accuracy, prompt uncertainty, redundant responses, and navigation inefficiencies. Addressing these usability gaps through improvements like better transparency, enhanced prompt guidance, varied responses, and streamlined navigation can significantly reduce user frustration and increase trust and engagement. By aligning AI capabilities with user-centered design principles, ChatGPT has the potential to become a more intuitive and reliable tool for academic use.
