Generative AI in the classroom: a controlled experiment to measure what it teaches — and what it erases
Ian Lucca Soares Mesquita and Christian Hirsch Santos presented at FCTE/UnB the TCC1 that plans a controlled laboratory experiment with Software Engineering students and alumni to measure, with objective and psychological metrics, the effect of mandatory use of generative AI on code quality and on learning.
Brasília, July 2026
If generative AI now writes much of the student’s code, the natural question is: what is left for the student to learn? The question is rhetorical in nearly every public debate, but rarely turns into an actual experimental design. It is such a design that Ian Lucca Soares Mesquita and Christian Hirsch Santos deliver in their TCC1, presented at the Faculty of Sciences and Technologies in Engineering (FCTE), advised by Prof. Sergio Antônio Andrade de Freitas .
The plan is a controlled laboratory study with students and alumni of the Software Engineering program at UnB. Participants are randomly split into two groups: one required to use generative AI tools, the other with no access to them. All perform the same task in React, which pins the technology context and frees attention for what actually matters — the effect of the treatment. The methodology is organized in three stages: recruitment, programming exercise, and final questionnaire.
Data collection combines two families of metrics. On the objective side, code quality measures: lines of code, code health indicators, lint errors, and execution time. On the subjective side, psychological dimensions captured in Likert scales — satisfaction, perceived usefulness, self-efficacy, positive emotions, and disposition toward autonomous learning. The plan is structured under the Goal-Question-Metric (GQM) model and aligned with the ACM SIGSOFT Empirical Standard for Experiments with Human Participants — a combination that anchors the TCC1 in both experimental rigor and replicability.
TCC1 closes the theoretical foundation and experimental design phase; it leaves for TCC2 the execution of the study, data analysis, and evidence write-up. This contribution talks directly to artificial intelligence applied to Software Engineering and to the active learning effort that teaching institutions need to calibrate carefully — so as not to mistake apparent productivity for actual education.
About CEDIS: The Center for Studies, Development, and Innovation in Software (CEDIS), linked to the University of Brasília, researches and develops innovative software solutions with strong presence in artificial intelligence and software quality .