Recommendations presented to the Faculty Senate
February 7, 2002
Faculty Senate Advancement of Teaching (AOT) Committee
Teaching, or helping others learn, integrates a bit of art, philosophy, technique, and science along with content to achieve excellence. Assessment is similar. Measuring teaching excellence requires peer review, student assessment, personal reflection, and perhaps other bits of art, philosophy, technique, and science. Teaching assessment using multiple approaches and measures helps faculty and instructors learn and improve their teaching practice.
The Advancement of Teaching (AOT) Committee of the Faculty Senate was asked to add a question about "diversity" to student evaluation forms for classes and instructors, spring 1999. AOT and advisory faculty1 reviewed both our current assessment form, the science of teaching assessment, and diversity questions used by other universities. AOT members searched university assessment forms; none included a question about diversity. The UW Campus Climate Survey suggests that "diversity" requires several questions or a complete "climate" survey being considered at OSU. Faculty and students sometimes comment adversely about the current form (questions 1, 2, and 7 confuse two criteria; 3 and 10 are similar).
Criteria for making changes emphasized teaching improvement (formative), selection of questions based on the science of survey research and validation2 with proper reporting, and fair/accountable assessment. These recommendations complement the accreditation team's suggestions for "improved assessment" completed last spring. Extension and Distance Ed are considering similar questions with appropriate wording for the respective audiences. Therefore, AOT recommends:
V poor poor fair good v good excellent ... Doesn't apply
Back [space for 12 scan questions AND written comments on single page]
costs to develop computer program to scan both sides of form
$65008. Sample questions are available @ (see attached
College of Engineering will print ABET accreditation questions on
the back of OSU form. Distance Ed will beta-test on-line version of
assessment. Extension prefers similar form with questions worded for
appropriate learning situations.
inter-correlation factor analysis will confirm relationships among items
or provide evidence of little or no relationship. If lack of correlation
is found, changes will be made prior to printing the forms. Based on
comments from faculty and Robert Mason, retired Statistician, AOT invites
several faculty interested in trying the form winter quarter to contact
Ray William at email@example.com.
Clarify reporting policy. The science of educational measurement suggests that questions involving "overall" satisfaction with the course and instructor are valid for comparing faculty and courses between departments and colleges. Thus, questions 1 and 2 were selected for P&T, merit, and award dossiers with permission and assumed validation by UW Office of Educational Assessment. Additionally, 10 questions were selected and modified slightly from UW, KSU, or other universities to assess teaching with reports to faculty and supervisors. All questions selected by faculty should remain confidential (Hoyt and Pallett, 1999; Marsh and Roche, 1997; McKeachie, 1997).
It is moved that OSU develop a policy regarding distribution of assessment results such that the first two questions be used for P&T, merit, and awards; all twelve (12) questions (front side of form) be reported to faculty and supervisors to improve teaching within departments; and results from personalized assessment questions (back side of form) be reported to faculty only.
Continue search for the right action involving "diversity". AOT searched university evaluation forms and programs for questions about diversity. We discovered that Nana Lowell, UW/OEA Research Director, completed an extensive "campus climate" survey (http://www.washington.edu/oea/9919.htm) suggesting the need to integrate the community, campus, and classroom functions when assessing this complex issue. She has agreed to present a seminar identifying key questions and to facilitate a discussion beginning with a Faculty Forum, March 6 and 7, to consider appropriate actions or decisions at OSU.
College, Forestry, Distance Ed, and Extension evaluation specialists advised
2In the educational measurement literature, reliability covers consistency, stability, and generalizability, whereas validity confirms that the question asks what it was intended to measure. Reliability and validity testing for questions selected confirmed by University of Washington, Kansas State University, or #6 from U. of Minnesota. (see Cashin, 1988; Gillmore, 2000; Marsh and Roche, 1997; McKeachie, 1997).
3University of Washington, Kansas State University, Penn State, Virginia Tech, University of Minnesota, Northwestern, Indiana State University, and Washington State University were consulted.
4OSU faculty express either a strong preference for a 5-point or 6-point Likert scale. The 5-point scale contains a neutral mid-point and intended mean and results will be comparable between the old and proposed rating scores (e.g. similar to GPA with 0-1-2-3-4 scale). In contrast, a 6-point Likert scale spreads responses slightly and forces respondents to choose a mid-point (either fair or good). Since the January Senate meeting, AOT reviewed recent literature and asked 4 survey researchers for advice. The preponderance of opinion is that a 6- or 7-point scale is preferred to spread responses and encourage mid-point decisions rather than neutral responses. AOT members recognize and appreciate the reasons expressed by faculty for a 5-point scale; however, we conclude that erring on the side of slightly greater spread and decisions by respondents encourages our committee to recommend adoption of the 6-point scale.
5In survey research, the order of questions will influence subsequent responses. Assuming that most students want to provide valid information, both AOT and UW want to minimize the "conditioning response" for questions associated with P&T, merit, or awards. Again, AOT members respect the reasons for placing the questions as a summary at the bottom of the first page.
were selected with permission or modified as follows:
validated UW Form B; outcomes suggested as instruction adapts to outcomes
7Faculty suggested that narrative questions could be printed on the back to save paper and expense. However, confidentiality may be compromised in small classes, thereby requiring separate pages.
8Cost estimate includes $500 for printing a new master, $1000 to reconfigure OSU scanner for dual-side scanning, and $5000 for writing program to report both sides of OSU form; total $6500.
Cashin, W.E. 1988. Student Ratings of Teaching: A Summary of the Research. IDEA Paper No. 20. http://www.idea.ksu.edu/papers/pdf/Idea_Paper_20.pdf.
Cuseo, J. 2000. Evaluating new-student seminars & other first-year courses via course-evaluation surveys: Research-based recommendations regarding instrument construction & administration, data analysis, data summary, & reporting results. Marymount College. http://www.brevard.edu/fyc/fya/CuseoLink.htm.
England, J., P. Hutchings, and W.J. McKeachie. 1996. The professional evaluation of teaching. American Council of Learned Societies Occasional Paper No. 33. http://www.acls.org/op33.htm.
Hoyt, D.P. and W.H. Pallett. 1999. Appraising teaching effectiveness: Beyond student ratings. IDEA Paper #36. http://www.idea.ksu.edu/papers/pdf/Idea_Paper_36.pdf.
Gillmore, G.M. 2000. Drawing inferences about instructors: The inter-class reliability of student ratings of instruction. OEA Research Reports 00-02. http://www.washington.edu/oea/0002.htm.
Marsh, H.W. and L.A. Roche. 1997. Making students' evaluations of teaching effectiveness effective. American Psychologist. 52:1187-1197.
McKeachie, W.J. 1997. Student ratings: The validity of use. American Psychologist. 52:1218-1225.