Reading what students write

A framework for annotating free-text teaching evaluations

Published

July 19, 2026

Numeric ratings on a course evaluation form give you an average. They rarely tell you what to change. The open-ended comments do, but only if they are read in a consistent way and interpreted against what the research actually supports.

This document has two jobs. The first is to summarize what the peer-reviewed literature says about student evaluations of teaching (SET), sentiment analysis of text feedback, and the reliability of both human and machine coding. The second is to hand you a practical coding scheme, a codebook, that you can apply by hand or with a large-language model (LLM), together with the checks that keep the results trustworthy.

Note

Everything here concerns the qualitative comments, not the numeric items. The numeric items have their own long literature, and the short version of it shapes how we read the comments, so we start there.

What the evidence says about teaching evaluations

Numeric ratings measure satisfaction better than learning

The common assumption behind SET is that students learn more from instructors they rate highly. The strongest test of that assumption is the multisection study, where many sections of the same course are taught by different instructors, students are assigned in a way that approximates randomness, and a common final exam measures learning. Early meta-analyses of these studies reported small to moderate positive correlations between ratings and achievement. Re-analysis told a different story. Uttl et al. (2017) showed that those earlier correlations were largely an artifact of small samples and publication bias, and that in the larger, higher-quality studies the relationship between ratings and learning is close to zero. Their pooled estimates put the shared variance between ratings and learning at roughly one percent.

That finding sits inside a wider assessment of validity. Spooren et al. (2013), reviewing the field since 2000 in Review of Educational Research, concluded that SET remains a delicate instrument: it can be useful, but its validity depends heavily on questionnaire design, on whether students and instructors share a definition of good teaching, and on how the scores are later used. Stark & Freishtat (2014) make the practical version of the same argument. Averaging ordinal ratings, comparing an instructor to a department mean, and treating the difference as a fact about teaching quality all rest on assumptions that the data do not meet. Systematic context effects make the comparison worse still: instructors of quantitative and STEM courses tend to score lower than colleagues teaching qualitative material, independent of how well they teach, so a cross-department ranking partly measures the subject rather than the teacher (Uttl & Smibert, 2017).

None of this means the evaluations are worthless. It means a high or low average is a statement about student experience and satisfaction, not a direct readout of how much anyone learned (Clayson, 2009). The comments inherit the same limit, and reading them well starts from accepting it.

Ratings and comments carry measurable bias

If ratings only tracked teaching, two instructors delivering the identical course should receive similar scores. They do not, and the cleanest evidence comes from experiments that hold teaching constant and vary only what students believe about the instructor.

MacNell et al. (2015) ran an online course in which two instructors each taught two sections, one under their real identity and one under a swapped gender identity. Students rated the perceived-male identity higher than the perceived-female identity even when the same person was teaching. Larger observational and quasi-experimental studies point the same way. Boring (2017), using French data with near-random assignment, found that male instructors received higher ratings without any matching advantage in student performance on anonymous final exams. Mengel et al. (2019), with a sample of nearly twenty thousand evaluations, found that women received lower ratings, that the gap was larger among junior instructors and in quantitative subjects, and that it appeared in the language students used, not only in the numbers. Mitchell & Martin (2018) add that the vocabulary students reach for differs by perceived gender, so the bias shows up in the free text this framework is designed to code.

The pattern extends beyond gender. Black instructors tend to receive lower ratings than white colleagues, especially on items about competence and the value of a course (Smith & Hawkins, 2011), and white students more readily challenge the authority and expertise of faculty of color (Parker & Neville, 2019; Pittman, 2010). The penalties compound for women of color, whose ratings carry both racialized and gendered expectations (Croom, 2017). Instructors whose first language is not English are rated lower as well, and comments that fix on an accent track that gap (Fan et al., 2019). This matters for a framework built on the written comments, because the free text is where these disparities are sharpest: women and faculty of color draw more of the personal and derogatory remarks that a numeric average never shows (Wallace et al., 2019).

The lesson for annotation is direct. A comment about an instructor’s warmth, authority, or appearance may say more about student expectations than about instruction. Comments of that kind still need to be captured, because pretending they are absent hides the bias rather than measuring it. The framework therefore gives them their own category and treats them as an equity signal, not as feedback to act on.

The comments are the part worth analyzing

Given the limits above, why analyze the text at all? Because the open response is where students explain themselves. Brockx et al. (2012) found that written comments carry information the rating scales miss, including the reasons behind a score and concrete incidents that a number flattens. Wallace et al. (2019) reviewed the written-comment literature and showed both its value and its hazards, including that some groups of instructors receive systematically harsher or more personal commentary.

For a working instructor the payoff is formative. A three-point average does not tell you that the problem sets arrive too late to help before the exam. A comment does. The task is to read many such comments in a way that is repeatable, that separates the actionable from the merely emotional, and that does not quietly launder bias into a to-do list.

Methods for turning comments into data

There is no single correct method. There is a family of them, and they trade off effort, transparency, and scale.

Qualitative coding: content analysis and thematic analysis

The oldest and most transparent approach assigns comments to categories defined in a codebook. Hsieh & Shannon (2005) distinguish three flavors of qualitative content analysis. Conventional analysis lets categories emerge from the data, which suits a first pass on an unfamiliar course. Directed analysis starts from existing theory, which is what you do when you adopt a codebook like the one in this document. Summative analysis counts terms and then interprets them, which is where sentiment counting lives.

Thematic analysis, set out by Braun & Clarke (2006), is the most widely used version in education and the social sciences. Its six phases move from familiarization through initial codes to reviewed, named themes and a written account. The method is deliberately flexible, which is a strength for exploration and a risk for consistency, since two readers can name themes differently. That risk is exactly why the reliability step in Section 4 is not optional.

Sentiment analysis and its aspect-based form

Sentiment analysis assigns emotional orientation, usually positive, negative, or neutral, to a piece of text. Applied naively to a whole comment it fails often, because one comment frequently praises one thing and criticizes another. The response in the literature is aspect-based sentiment analysis (ABSA), which first identifies what the sentiment is about and then scores the sentiment for that aspect separately. Kastrati et al. (2021), in a systematic mapping study of NLP and deep learning on student feedback, document the shift toward aspect-level and transformer-based methods and the recurring problems: small labeled datasets, uneven category definitions, and results that do not transfer across institutions.

The practical implication is that sentiment should be coded per aspect, not per comment. A sentence such as “the lectures were clear but grading took forever” is positive about clarity and negative about assessment. Collapsing that to a single label throws away the part you can act on.

AI-assisted annotation and its limits

Large language models can apply a codebook to thousands of comments in minutes, and several studies report that they do it well. Gilardi et al. (2023) found that a zero-shot model matched or beat crowd workers on relevance, stance, topic, and frame tasks, at a fraction of the cost and with higher inter-coder agreement across repeated runs. Törnberg (2023) reported similar accuracy against expert labels on a political classification task.

The countervailing evidence is equally clear and belongs in any plan to automate. Reiss (2023) showed that model output is sensitive to prompt wording and can be unreliable on some tasks, and argued for caution. Pangakis et al. (2023) replicated 27 annotation tasks across 11 datasets and reached a conclusion this framework adopts as a rule: model performance varies by task and dataset in ways you cannot predict in advance, so every automated annotation run must be validated against human labels before its output is trusted. Separately, generative models can fabricate content that reads as confident and correct. The survey by Ji et al. (2023) catalogs this hallucination problem, which for annotation shows up as invented justifications, categories the codebook never defined, or a fluent rationale attached to a wrong label. A model that explains its reasoning is not therefore a model that reasoned correctly.

Two further cautions are specific to this data. Models can carry the same social biases the human literature documents, so an unaudited model may reproduce gendered readings of comments rather than flag them. And evaluation comments are sensitive personal data about identifiable instructors, which argues for care about what text is sent to any external system and under what agreement.

The annotation framework

This section defines the unit of analysis, a codebook of content categories, the cross-cutting attributes coded on each unit, and the sentiment scale. It is a directed content-analysis scheme in the sense of Hsieh & Shannon (2005), grounded in the SET dimensions synthesized by Marsh (1987) and extended with the equity and actionability concerns from Section 1.2.

Unit of analysis

Code at the level of the aspect segment, not the whole comment. A segment is a span of text expressing one evaluable idea about one aspect of the course. A single comment may contain several segments. Splitting first, then coding, is what makes aspect-based sentiment possible and keeps a mixed comment from being forced into one label. For short comments the segment and the comment coincide, which is fine.

Record, for each segment: the content category, the sentiment polarity, the specificity, the actionability, and the register. Keep a pointer back to the source comment so that a segment can be read in context during adjudication.

Content categories

The categories below are meant to be exhaustive enough that every segment lands somewhere and exclusive enough that most segments land in one place. Examples are illustrative and invented; do not use real student text in a shared codebook.

# Category Definition Include Exclude Illustrative example
1 Clarity of explanation How understandable the instruction was Explanations, examples, answering questions, clear or confusing delivery Pace or workload (see 4); materials themselves (see 5) “Explained hard ideas in a way that finally clicked.”
2 Organization and structure How the course and sessions were arranged Sequencing, syllabus, deadlines announced in advance, coherence between parts Room or scheduling logistics (see 9) “Lectures jumped around and I never knew what was due.”
3 Assessment, grading, and feedback Exams, assignments, rubrics, timeliness and fairness of grading Fairness, feedback quality, turnaround time, alignment of tests to content Overall difficulty (see 4) “Feedback came back too late to help on the next paper.”
4 Workload, pace, and difficulty Amount and speed of work relative to expectations Reading load, pace of lectures, perceived difficulty Whether grading was fair (see 3) “Reasonable workload, moved a little fast in week 3.”
5 Course content and materials The subject matter, textbook, slides, software, readings Relevance, quality, and currency of materials The instructor’s delivery of them (see 1) “The textbook was outdated and barely used.”
6 Availability and responsiveness Access to the instructor outside class Office hours, email replies, willingness to help In-class warmth (see 8) “Answered emails within a day, office hours were useful.”
7 Enthusiasm and engagement The instructor’s evident interest and ability to hold attention Energy, passion for the subject, engaging activities Rapport with students (see 8) “Clearly loves the material and it was contagious.”
8 Interaction, inclusion, and climate The relational and inclusive quality of the classroom Respect, fairness toward all students, sense of belonging, discussion climate Non-pedagogical remarks about the person (see 10) “Made sure quieter students got a chance to speak.”
9 Logistics and environment Physical and technical conditions outside the instructor’s control Room, scheduling, platform, class size Anything the instructor decided (see 2, 3) “The 8 a.m. slot and broken projector hurt.”
10 Non-pedagogical or identity-focused remarks Comments about the instructor’s appearance, personality, accent, gender, race, age, or perceived traits unrelated to teaching Personal remarks, praise or criticism of traits rather than actions Comments about teaching behavior, even if warm (see 7, 8) “She seemed too young to be teaching this.”
11 Off-topic or uninformative Segments that carry no evaluable content Jokes, single words, blanks, comments about other courses Anything that names a teaching aspect “N/A” or “it was a class.”

Category 10 is the equity flag. Its purpose is to make bias countable rather than to act on it. A cluster of category-10 comments, especially ones the bias literature would predict — gendered, racialized, or aimed at an accent (MacNell et al. (2015); Boring (2017); Mengel et al. (2019); Smith & Hawkins (2011); Croom (2017); Fan et al. (2019)) — is a signal about the evaluation process and the reviewers, not a set of instructions for the instructor.

One more distinction cuts across the whole table. Students are well positioned to report some of these things and poorly positioned to judge others. They can speak with authority about what they experienced directly: clarity, organization, the timeliness of feedback, the instructor’s availability, and the climate of the class. They are not positioned to certify the quality of the course content, the instructor’s command of the subject, the appropriateness of the course goals, or the soundness of the assessment design, all of which require disciplinary expertise students are still acquiring (Benton & Young, 2018; Clayson, 2020; Frey, 1974). The categories most exposed to this limit are course content and materials (5) and parts of assessment (3): a segment there records a real student experience, but it is evidence about how the course landed, not a verdict on whether the material or the exam was sound. Keep the segment; weight it accordingly, and do not read a run of “the textbook was useless” comments as an authoritative judgment on the textbook.

Cross-cutting attributes

Every segment also receives four attributes.

Sentiment polarity uses four values: positive, negative, mixed, and neutral. Mixed is reserved for a single segment that genuinely balances both, which should be rare once segmentation is done well; most mixed feelings resolve into separate positive and negative segments. Neutral covers descriptive statements with no evaluative charge.

Specificity is binary: specific or general. “Grading was unfair” is general. “The midterm tested topics we never covered” is specific. Specific negative feedback is the material an instructor can act on; general negative feedback is mostly a mood reading.

Actionability is binary: actionable or not. A segment is actionable if a reasonable instructor could change something in response to it. Praise can be actionable (“keep the weekly recap emails”) and criticism can be non-actionable (“the class was at a bad time”).

Register is a six-level scale defined in Section 3.4.

Register: what the comment is doing

Register captures the communicative stance of the segment, meaning what the student is doing with the words rather than whether the words are polite. It is a distinct axis from sentiment (polarity) and from civility (tone): a negative segment can be a helpful suggestion or an unproductive vent, and either can be phrased politely. The six levels run from most constructive to most destructive, and each segment gets exactly one.

# Register What the student is doing Include Typical example
1 Suggestion Proposes a specific change, addition, or action Forward-looking, prescriptive language, whether the polarity is positive or negative “I would post the slides the night before so we can review first.”
2 Constructive Diagnoses a grounded strength or problem in the teaching, without prescribing a fix Reasoned praise or criticism aimed at the course or its design, engaging the work and not the person “The exams tested material that was never covered in lecture.”
3 Experiential Recounts the student’s own experience, reaction, or what worked or did not work for them First-person framing (“for me,” “I struggled with,” “it helped me”) reporting a personal effect “The ranked-question format helped me manage exam time, which I’ve historically struggled with.”
4 Descriptive Plainly reports a fact or state with little evaluative or emotional charge Neutral statements of what happened “The class met at 8 a.m. and had weekly quizzes.”
5 Venting Releases frustration without offering specific, usable content Hyperbole, sarcasm, or exasperation “taking it out” on the instructor; mean-spirited but not a personal attack “This was the most ridiculous experience of my life.”
6 Hostile Attacks the instructor as a person Abusive, demeaning, contemptuous, or threatening language directed at the individual “He’s an arrogant idiot who shouldn’t be allowed near students.”

Two rules keep the levels exclusive.

First, hostility escalates above everything. If a segment contains abusive, demeaning, or threatening language aimed at the person, code it hostile even when it also contains a suggestion or a legitimate point, because the purpose of that level is to route the comment for wellbeing and, where relevant, institutional review rather than to treat it as ordinary feedback.

Second, below hostility, assign the register by what the segment actually carries. If it carries usable content, choose the most useful applicable level: a proposed fix is a suggestion; a grounded diagnosis without a fix is constructive; a personal account of an effect is experiential. If it carries no usable content, it is descriptive when the charge is low and venting when the charge is high. Venting is therefore the residual for emotionally loaded but empty segments; a heated comment that still names a concrete, gradeable problem belongs in constructive or suggestion, not venting.

Register is distinct from actionability even though the two often move together. Actionability asks whether the instructor could change something in response; register asks how the student framed it. A suggestion to move the class to a different time is a suggestion in register but not actionable, because the instructor does not control the schedule. A bare complaint that “grading was unfair” can be actionable while sitting in constructive or even venting, depending on whether it grounds the claim.

The two destructive levels, venting and hostile, serve the same protective purpose: they let harsh comments be set aside for wellbeing and, where relevant, institutional review rather than being read as routine feedback. Separating them matters because venting is common and usually harmless to route past an instructor, while genuine hostility is rarer and may warrant a different response.

The sentiment scale

Coders and models need the polarity definitions written down, because “positive” is not self-evident.

  • Positive: expresses satisfaction, praise, or a favorable judgment about the aspect.
  • Negative: expresses dissatisfaction, criticism, or an unfavorable judgment about the aspect.
  • Mixed: expresses both favorable and unfavorable judgment about the same aspect within one segment.
  • Neutral: describes the aspect without evaluative charge, or states a fact.

Sarcasm, negation (“not bad”), and faint praise are the usual failure points for both humans and models, so include two or three worked examples of each in the codebook you actually deploy.

Strengths and limitations of the framework

The framework’s strengths are that it is explicit, so two readers or a reader and a model can be compared; it separates topic from sentiment, which matches how comments actually behave (Kastrati et al., 2021); and it makes bias visible through category 10 rather than absorbing it silently.

The category boundaries, especially between clarity, organization, and content, will produce disagreement, which is why reliability must be measured rather than assumed. The scheme is tuned to conventional courses and will need new categories for labs, studios, clinical placements, or fully asynchronous formats. And no coding scheme repairs the underlying validity problem: coding satisfaction carefully still yields satisfaction, not learning (Uttl et al., 2017).

Coding it reliably

A codebook is a hypothesis about how comments should be read. Reliability testing is how you find out whether the hypothesis holds across readers.

Automating with an assistant

If you hand the coding to an LLM, you must manually validate.

Build a human-labeled gold standard first, using the pilot process in ?@sec-human-procedure. Then run the model on that same set and compare, following the validation-first workflow of Pangakis et al. (2023). A single overall accuracy number hides exactly the failures that matter.

Treat the following as standing requirements rather than nice-to-haves:

  • Keep a human in the loop. Audit a fresh random sample of the model’s labels on the full dataset, not only the calibration set, because performance on held-out data can differ (Pangakis et al., 2023).
  • Watch for fabrication. If you ask the model to justify a label, check that the justification quotes text that exists and uses categories that were defined; invented rationales are a documented failure mode (Ji et al., 2023).
  • Fix the prompt, then freeze it. Model output shifts with wording (Reiss, 2023), so iterate the prompt during calibration and record the exact version used for the production run.
  • Protect the data. Evaluation comments identify real instructors. Decide deliberately what leaves your environment and under what terms, and strip names where the analysis does not need them.
  • Re-validate after any model or prompt change. Agreement established for one configuration does not carry over to another.

A reasonable division of labor is to let the model do the first pass on the full corpus and have humans adjudicate the segments the model flags as low-confidence, as category 10, or as register hostile or venting, plus a random audit sample. The register flags matter here for the same reason category 10 does: a segment routed as hostile carries wellbeing and, where relevant, institutional-review consequences, so a person rather than a model should confirm that call. That keeps the scale benefit while holding a person responsible for the readings that carry the most weight.

Reading the results without over-reading them

The output of this process is a table of segments with categories, sentiment, and flags. What you may conclude from it is narrower than it looks.

Use it formatively. The clearest legitimate use is an instructor seeing that, say, most negative-actionable segments cluster in assessment and feedback, and changing grading turnaround next term. That is the use the comment data supports well. Register sharpens this: the segments worth reading first are the suggestions and constructive criticisms, which point at something to change, and the experiential ones, which explain how the course actually landed. The venting and hostile segments are best read separately, if at all, and treated as a wellbeing matter rather than a to-do list; a run of them says more about the emotional temperature of the class than about what to fix.

Be careful with counts. Who writes comments is not a random sample of who took the course: respondents skew toward students who are more engaged and who earn higher grades (Goos & Salomons, 2017), and extreme experiences at both ends are overrepresented, so a sentiment ratio describes the commenters, not the class. The problem is worst in small courses, where a handful of comments produce noisy numbers that should not be reported to a decimal place; empirically, ratings begin to stabilize only around twenty-five or more respondents, and some work puts the floor for a trustworthy picture nearer fifty (Feistauer & Richter, 2017; Fosnacht et al., 2017). Below that, report ranges and read the text, not the proportions.

Do not convert sentiment into a personnel metric. Given the near-zero link between ratings and learning (Uttl et al., 2017) and the documented bias in both scores and language (Boring, 2017; Fan et al., 2019; Mengel et al., 2019; Mitchell & Martin, 2018; Smith & Hawkins, 2011), an aggregated sentiment score is not a valid measure of teaching quality and should not be used on its own for hiring, promotion, tenure, or merit decisions. This is the point where the administrator’s interest and the evidence most sharply meet, and the evidence says to combine SET-derived signals with peer observation, teaching materials, and self-reflection rather than to rank people by comment sentiment. This is not a fringe view. The National Academies of Sciences, Engineering, and Medicine, among other bodies, have urged institutions to treat student surveys as one input among several rather than the decisive one (National Academies of Sciences, Engineering, and Medicine, 2020), and in a 2018 labor arbitration a Canadian university was barred from using survey scores to judge teaching effectiveness at all, on the ground that ranking instructors by them is fundamentally flawed (Ryerson University v Ryerson Faculty Association, 2018). The framework here inherits that logic: it produces a signal to be triangulated, not a score to rank by.

Remember that comments can be factually wrong. A student may misremember when feedback was returned or attribute a scheduling decision to the instructor that the department made. A smaller number are not mistaken but untruthful: some students knowingly skew their comments toward instructors they like or dislike, a tendency stronger among those who treat the course as a purchase (Clayson & Haley, 2011; Cox et al., 2021). The framework records what students reported, which is worth knowing, but a reported fact is a perception until it is checked against the record.

What this framework does not solve

The method makes reading comments consistent, transparent, and scalable. It measures satisfaction carefully and it makes bias countable. It does not turn satisfaction into learning, it does not remove the bias that the comments contain, and it does not license using comment sentiment as a verdict on a teacher. Held to those bounds, it gives an instructor a dependable way to hear what students are actually telling them, and gives everyone else an honest account of what that signal is and is not.

References

Benton, S. L., & Young, S. (2018). Best practices in the evaluation of teaching (IDEA Paper No. 69). IDEA Center.
Boring, A. (2017). Gender biases in student evaluations of teaching. Journal of Public Economics, 145, 27–41. https://doi.org/10.1016/j.jpubeco.2016.11.006
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
Brockx, B., Van Roy, K., & Mortelmans, D. (2012). The student as a commentator: Students’ comments in student evaluations of teaching. Procedia - Social and Behavioral Sciences, 69, 1122–1133. https://doi.org/10.1016/j.sbspro.2012.12.042
Clayson, D. E. (2009). Student evaluations of teaching: Are they related to what students learn? A meta-analysis and review of the literature. Journal of Marketing Education, 31(1), 16–30. https://doi.org/10.1177/0273475308324086
Clayson, D. E. (2020). A comprehensive critique of student evaluation of teaching: Critical perspectives on validity, reliability, and impartiality. Routledge (Taylor & Francis Group).
Clayson, D. E., & Haley, D. A. (2011). Are students telling us the truth? A critical look at the student evaluation of teaching. Marketing Education Review, 21(2), 101–112. https://doi.org/10.2753/MER1052-8008210201
Cox, S., Rickard, M. K., & Lowery, C. (2021). The student evaluation of teaching: Let’s be honest—who is telling the truth? Marketing Education Review. https://doi.org/10.1080/10528008.2021.1922924
Croom, N. N. (2017). Promotion beyond tenure: Unpacking racism and sexism in the experiences of black womyn professors. The Review of Higher Education, 40(4), 557–583.
Fan, Y. et al. (2019). Gender and cultural bias in student evaluations: Why representation matters. PLOS ONE, 14(2), e0209749. https://doi.org/10.1371/journal.pone.0209749
Feistauer, D., & Richter, T. (2017). How reliable are students’ evaluations of teaching quality? A variance components approach. Assessment and Evaluation in Higher Education, 42(8), 1263–1279. https://doi.org/10.1080/02602938.2016.1261083
Fosnacht, K., Sarraf, S., Howe, E., & Peck, L. K. (2017). How important are high response rates for college surveys? The Review of Higher Education, 40(2), 245–265. https://doi.org/10.1353/rhe.2017.0003
Frey, P. W. (1974). The ongoing debate: Student evaluation of teaching. Change, 6(1), 47–64. https://doi.org/The ongoing debate: Student evaluation of teaching
Gilardi, F., Alizadeh, M., & Kubli, M. (2023). ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30), e2305016120. https://doi.org/10.1073/pnas.2305016120
Goos, M., & Salomons, A. (2017). Measuring teaching quality in higher education: Assessing selection bias in course evaluations. Research in Higher Education, 58(4), 341–364. https://doi.org/10.1007/s11162-016-9429-8
Hsieh, H.-F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), 1277–1288. https://doi.org/10.1177/1049732305276687
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 1–38. https://doi.org/10.1145/3571730
Kastrati, Z., Dalipi, F., Imran, A. S., Nuci, K. P., & Wani, M. A. (2021). Sentiment analysis of students’ feedback with NLP and deep learning: A systematic mapping study. Applied Sciences, 11(9), 3986. https://doi.org/10.3390/app11093986
MacNell, L., Driscoll, A., & Hunt, A. N. (2015). What’s in a name: Exposing gender bias in student ratings of teaching. Innovative Higher Education, 40(4), 291–303. https://doi.org/10.1007/s10755-014-9313-4
Marsh, H. W. (1987). Students’ evaluations of university teaching: Research findings, methodological issues, and directions for future research. International Journal of Educational Research, 11(3), 253–388. https://doi.org/10.1016/0883-0355(87)90001-2
Mengel, F., Sauermann, J., & Zölitz, U. (2019). Gender bias in teaching evaluations. Journal of the European Economic Association, 17(2), 535–566. https://doi.org/10.1093/jeea/jvx057
Mitchell, K. M. W., & Martin, J. (2018). Gender bias in student evaluations. PS: Political Science & Politics, 51(3), 648–652. https://doi.org/10.1017/S104909651800001X
National Academies of Sciences, Engineering, and Medicine. (2020). Recognizing and evaluating teaching in higher education: Proceedings of a workshop—in brief. The National Academies Press. https://doi.org/10.17226/25685
Pangakis, N., Wolken, S., & Fasching, N. (2023). Automated annotation with generative AI requires validation. https://doi.org/10.48550/arXiv.2306.00176
Parker, T. L., & Neville, K. M. (2019). The influence of racial identity on white students’ perceptions of african american faculty. The Review of Higher Education, 42(3), 879–901.
Pittman, C. T. (2010). Race and gender oppression in the classroom: The experiences of women faculty of color with white male students. Teaching Sociology, 38(3), 183–196. https://doi.org/10.1177/0092055X10370120
Reiss, M. V. (2023). Testing the reliability of ChatGPT for text annotation and classification: A cautionary remark. https://doi.org/10.48550/arXiv.2304.11085
Ryerson University v Ryerson Faculty Association. (2018). Arbitration award, CanLII 58446 (ON LA).
Smith, B. P., & Hawkins, B. (2011). Examining student evaluations of black college faculty: Does race matter? The Journal of Negro Education, 80(2), 149–162. https://www.jstor.org/stable/41341117
Spooren, P., Brockx, B., & Mortelmans, D. (2013). On the validity of student evaluation of teaching: The state of the art. Review of Educational Research, 83(4), 598–642. https://doi.org/10.3102/0034654313496870
Stark, P. B., & Freishtat, R. (2014). An evaluation of course evaluations. ScienceOpen Research. https://doi.org/10.14293/S2199-1006.1.SOR-EDU.AOFRQA.v1
Törnberg, P. (2023). ChatGPT-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning. https://doi.org/10.48550/arXiv.2304.06588
Uttl, B., & Smibert, D. (2017). Student evaluations of teaching: Teaching quantitative courses can be hazardous to one’s career. PeerJ, 5, e3299. https://doi.org/10.7717/peerj.3299
Uttl, B., White, C. A., & Gonzalez, D. W. (2017). Meta-analysis of faculty’s teaching effectiveness: Student evaluation of teaching ratings and student learning are not related. Studies in Educational Evaluation, 54, 22–42. https://doi.org/10.1016/j.stueduc.2016.08.007
Wallace, S. L., Lewis, A. K., & Allen, M. D. (2019). The state of the literature on student evaluations of teaching and an exploratory analysis of written comments: Who benefits most? College Teaching, 67(1), 1–14. https://doi.org/10.1080/87567555.2018.1483317

A prompt template for AI-assisted coding

Adapt the bracketed parts. Freeze the wording once calibration is done, and record which model version produced the production run.

Theme discovery prompt (Stage 1)

Stage 1 reads the whole set of open-ended comments (with optional numeric survey context) and proposes a set of recurring themes. Its output is validated against theme-set.schema.json which you can find here. A human then curates that output (merge near-duplicates, rename, drop noise), adds the reserved unknown fallback, and saves the result as themes.json. The build step injects those theme codes into the Stage 2 schema, and Stage 2 (annotation-prompt.md) assigns one theme per segment from that fixed set plus unknown.

This is inductive coding: themes come from the data, not from a fixed list. Keep the set small enough to be usable (aim for 8 to 25 well-separated themes) and expect to curate it by hand before locking it.

You are performing inductive theme discovery on a set of student course-evaluation comments.
Read the entire set.
Identify the recurring themes that students actually raise.
Derive themes from the text only.
Do not import a preset list, and do not invent themes that the comments do not support.

Produce between 8 and 25 themes, unless the data clearly supports fewer.
Prefer a small set of distinct themes over many overlapping ones.
Merge near-duplicates rather than splitting hairs.

For each theme, provide:

- a stable code: snake_case, starts with a letter, at most 30 characters, unique.
- `label`: a short human-readable name.
- `definition`: one or two sentences stating what belongs under the theme.
- `recommended_category`: the single best fit among these eleven stable categories (advisory only; a theme may cross categories): "Clarity of explanation", "Organization and structure", "Assessment, grading, and feedback", "Workload, pace, and difficulty", "Course content and materials", "Availability and responsiveness", "Enthusiasm and engagement", "Interaction, inclusion, and climate", "Logistics and environment", "Non-pedagogical or identity-focused remarks", "Off-topic or uninformative".
- `example_evidence`: 2 to 4 SHORT verbatim snippets copied exactly from the comments.
    Each snippet must appear verbatim in the input.
    These ground the theme; do not paraphrase.
- `prevalence_estimate`: one of "rare", "occasional", "common", "pervasive".
- `inclusion` (optional): a clause on what to include or exclude when this theme is ambiguous.

Hard rules:

- Do NOT create a theme named "unknown" or "other".
    A reserved fallback is added later.
- Every `example_evidence` snippet must be a verbatim substring of some input comment.
- Codes must be unique and stable enough to reuse across terms.
- Use "notes" to flag overlaps, borderline pairs a human should consider merging, or themes you were unsure about.
- Output MUST be a single JSON object conforming to `theme-set.schema.json`.
    Return only the JSON object wrapped in a markdown fence.

Set `"theme_set_id"` from the course and term if given, otherwise `"candidate-themes"`.
If numeric survey context is provided, use it only to gauge prevalence and salience; the themes themselves come from the comments.

Segment annotation prompt (Stage 2)

This is Stage 2 of a two-stage pipeline. Stage 1 reads the whole corpus and proposes themes; a human curates them into themes.json. Stage 2 then applies the codebook to one comment at a time, assigning one theme per segment from that fixed set plus unknown.

Files this prompt pairs with: segment-annotation.schema.json (enforces the output shape), themes.json (the locked theme registry), and example-annotation.json (a worked instance that validates against the schema).

Two labeling axes run in parallel on every segment:

  • category: one of eleven stable, cross-course categories grounded in the SET literature.
  • theme: one code from the current themes.json, or unknown if none fits. This is the finer, course-specific layer you group by.

The two are independent on purpose. A theme can land under different categories in different segments, so do not force a theme to match its recommended category. Pick each axis on its own.

You are applying a FIXED codebook to a single student course-evaluation comment.
Annotate only what the text says.
Do not infer beyond the text.
Do not invent categories or themes.
If a segment fits no category, use "Off-topic or uninformative".
If a segment fits no theme, use "unknown".

**AXIS 1:** the eleven categories (stable), used verbatim as the "category" value.

1. "Clarity of explanation": how understandable the instruction was.
2. "Organization and structure": how the course and sessions were arranged.
3. "Assessment, grading, and feedback": exams, assignments, rubrics, fairness, and timeliness of grading and feedback.
4. "Workload, pace, and difficulty": amount and speed of work, perceived difficulty.
5. "Course content and materials": subject matter, textbook, slides, software, readings.
6. "Availability and responsiveness": office hours, email replies, willingness to help.
7. "Enthusiasm and engagement": the instructor's evident interest and ability to hold attention.
8. "Interaction, inclusion, and climate": respect, fairness toward all students, belonging, discussion climate.
9. "Logistics and environment": room, scheduling, platform, class size, and other conditions outside the instructor's control.
10. "Non-pedagogical or identity-focused remarks": remarks about appearance, personality, accent, gender, race, age, or perceived traits unrelated to teaching.
    This is an EQUITY FLAG, not feedback to act on.
11. "Off-topic or uninformative": segments with no evaluable content.

**AXIS 2:** the course themes (fine, course-specific), used as the "theme" value (the code, not the label).
Choose exactly one.

Value sets for the remaining fields:

- sentiment: "positive" | "negative" | "mixed" | "neutral"
    - positive  = favorable judgment about the aspect
    - negative  = unfavorable judgment about the aspect
    - mixed     = both favorable and unfavorable about the SAME aspect in one segment (rare; usually a sign the segment should be split further)
    - neutral   = describes the aspect with no evaluative charge, or states a fact
- specificity: "specific" | "general"
    - specific  = names a concrete instance, behavior, or incident
    - general   = a mood or blanket judgment with no concrete referent
- actionability: "actionable" | "not_actionable"
    actionable     = a reasonable instructor could change something in response (praise can be actionable, e.g. "keep the weekly recap emails")
    not_actionable = nothing the instructor controls, or too vague to act on
- register: "suggestion" | "constructive" | "experiential" | "descriptive" | "venting" | "hostile"
    This is the communicative stance: what the student is DOING, not how polite it is, and independent of sentiment.

    Assign exactly one, using this order:

    1. hostile ESCALATES above all: if the segment attacks the instructor as a person (abusive, demeaning, contemptuous, or threatening language), use "hostile" even if it also contains a suggestion or a valid point.
    2. Otherwise, if the segment carries usable content, pick the most useful that applies:

        - suggestion    = proposes a specific change, addition, or action (any polarity),e.g., "post the slides the night before"
        - constructive  = diagnoses a grounded strength or problem in the teaching without prescribing a fix, e.g., "the exams tested material never covered"
        - experiential  = reports the student's own experience or reaction, framed in the first person, e.g. "it helped me manage exam time"
    3. Otherwise (no usable content):

        - descriptive   = a flat factual statement with little charge
        - venting       = emotionally loaded but empty: hyperbole, sarcasm, or exasperation "taking it out" on the instructor, without a concrete gradeable point

    Note: register is NOT actionability.
    A suggestion to change the class time is register "suggestion" but "not_actionable"; a grounded complaint can be "actionable" while sitting in "constructive" or "venting".

**Procedure**

Step 1. Reproduce the comment VERBATIM in "comment_text".

Step 2. Split the comment into segments.
A segment is a span expressing ONE evaluable idea about ONE aspect.
Each "segment_text" must be an exact, verbatim substring of "comment_text" (no paraphrase, no merging across aspects).
For a short comment the segment and the comment coincide.
Even a fully uninformative comment yields exactly one segment, category "Off-topic or uninformative", theme "unknown".

Step 3. For each segment assign: category, theme, sentiment, specificity, actionability, register, a confidence in [0, 1], and an "evidence" quote.

**Hard rules**

- "evidence" MUST be a verbatim substring of that segment's "segment_text". If you cannot quote supporting text, set "evidence" to "", lower "confidence", and give a one-clause reason in "annotation_note".
- One category AND one theme per segment. If a span genuinely covers two aspects, split it.
- category and theme are independent. Do NOT change one to make it match the other.
- "theme" must be one of the codes listed above (from themes.json) or "unknown". Never invent a new theme code.
- Do NOT fill the "human_verified" block yourself.
    For every segment set it to exactly:
    `{ "status": "pending", "label": null, "reviewer": null, "reviewed_at": null, "notes": null }` A human reviewer sets these later.
- Do NOT add a "provenance" object.
    The pipeline adds run metadata.
- Output MUST be a single JSON object that conforms to segment-annotation.schema.json.

Return only the JSON object.
No prose, no markdown fences, no trailing text.