Faculty Senate Back to Main Page
Recommendations presented to the Faculty Senate
February 7, 2002

Faculty Senate Advancement of Teaching (AOT) Committee

RECOMMENDS:

Teaching, or helping others learn, integrates a bit of art, philosophy, technique, and science along with content to achieve excellence. Assessment is similar. Measuring teaching excellence requires peer review, student assessment, personal reflection, and perhaps other bits of art, philosophy, technique, and science. Teaching assessment using multiple approaches and measures helps faculty and instructors learn and improve their teaching practice.

The Advancement of Teaching (AOT) Committee of the Faculty Senate was asked to add a question about "diversity" to student evaluation forms for classes and instructors, spring 1999. AOT and advisory faculty1 reviewed both our current assessment form, the science of teaching assessment, and diversity questions used by other universities. AOT members searched university assessment forms; none included a question about diversity. The UW Campus Climate Survey suggests that "diversity" requires several questions or a complete "climate" survey being considered at OSU. Faculty and students sometimes comment adversely about the current form (questions 1, 2, and 7 confuse two criteria; 3 and 10 are similar).

Criteria for making changes emphasized teaching improvement (formative), selection of questions based on the science of survey research and validation2 with proper reporting, and fair/accountable assessment. These recommendations complement the accreditation team's suggestions for "improved assessment" completed last spring. Extension and Distance Ed are considering similar questions with appropriate wording for the respective audiences. Therefore, AOT recommends:

  1. Revise OSU assessment form.
    The advisory faculty encouraged AOT to search for university programs3 with established expertise and assessment products. The following recommendations are an attempt to satisfy the many facets of teaching at OSU while maintaining a widely applicable yet flexible assessment instrument (Cuseo, 2000). Teaching, Extension, and Distance Education faculty will use similar forms and questions. An explanation of why AOT recommends a 6-point Likert scale is described in footnote4.

    • Two questions compare the tremendous diversity of courses and teaching approaches across campus for use in P&T, merit, and award documents; they are not meant to improve teaching. The questions and Likert scale have been validated by UW Education Assessment and used extensively by numerous universities. Reasons for placing these two questions at the top of the questionnaire are described in footnote5.

    V poor…poor…fair…good…v good…excellent……... Doesn't apply

      1. The course as a whole was….
      2. The instructor's contribution to the course was….

        • AOT, advisory, and review faculty selected 10 questions6 that assess fundamentals of teaching (front of form). The revised form includes space on the back to photocopy additional questions that interest faculty along with accreditation requirements and open-ended narrative questions for large classes7. A template to align photocopying will be developed.

         

      3. Clarity of course objectives or outcomes was….
      4. Clarity of student responsibilities and requirements was….
      5. Course organization was….
      6. Availability of extra help when needed was….
      7. Instructor's use of various instructional techniques to accommodate differences
        in learning styles among students was….
      8. Instructor's interest in my learning was….
      9. Instructor's ability to stimulate my thinking more deeply about the subject was..
      10. Instructor's timely feedback on tests and other work was….
      11. Instructor's respect for all students was….
      12. Instructor's evaluation of student performance in accordance with course objectives
        was….

      Back [space for 12 scan questions AND written comments on single page]

      Estimated costs to develop computer program to scan both sides of form $65008. Sample questions are available @ (see attached file: http://www.orst.edu/dept/senate/committee/aot/forms/request.html). College of Engineering will print ABET accreditation questions on the back of OSU form. Distance Ed will beta-test on-line version of assessment. Extension prefers similar form with questions worded for appropriate learning situations.

    An inter-correlation factor analysis will confirm relationships among items or provide evidence of little or no relationship. If lack of correlation is found, changes will be made prior to printing the forms. Based on comments from faculty and Robert Mason, retired Statistician, AOT invites several faculty interested in trying the form winter quarter to contact Ray William at williamr@bcc.orst.edu.

    It is moved that OSU adopt the modified student assessment form as proposed (Wording of questions may be improved with beta-testing winter quarter).

  2. Clarify reporting policy. The science of educational measurement suggests that questions involving "overall" satisfaction with the course and instructor are valid for comparing faculty and courses between departments and colleges. Thus, questions 1 and 2 were selected for P&T, merit, and award dossiers with permission and assumed validation by UW Office of Educational Assessment. Additionally, 10 questions were selected and modified slightly from UW, KSU, or other universities to assess teaching with reports to faculty and supervisors. All questions selected by faculty should remain confidential (Hoyt and Pallett, 1999; Marsh and Roche, 1997; McKeachie, 1997).

    It is moved that OSU develop a policy regarding distribution of assessment results such that the first two questions be used for P&T, merit, and awards; all twelve (12) questions (front side of form) be reported to faculty and supervisors to improve teaching within departments; and results from personalized assessment questions (back side of form) be reported to faculty only.

  3. Continue search for the right action involving "diversity". AOT searched university evaluation forms and programs for questions about diversity. We discovered that Nana Lowell, UW/OEA Research Director, completed an extensive "campus climate" survey (http://www.washington.edu/oea/9919.htm) suggesting the need to integrate the community, campus, and classroom functions when assessing this complex issue. She has agreed to present a seminar identifying key questions and to facilitate a discussion beginning with a Faculty Forum, March 6 and 7, to consider appropriate actions or decisions at OSU.


1Honors College, Forestry, Distance Ed, and Extension evaluation specialists advised AOT Committee.

2In the educational measurement literature, reliability covers consistency, stability, and generalizability, whereas validity confirms that the question asks what it was intended to measure. Reliability and validity testing for questions selected confirmed by University of Washington, Kansas State University, or #6 from U. of Minnesota. (see Cashin, 1988; Gillmore, 2000; Marsh and Roche, 1997; McKeachie, 1997).

3University of Washington, Kansas State University, Penn State, Virginia Tech, University of Minnesota, Northwestern, Indiana State University, and Washington State University were consulted.

4OSU faculty express either a strong preference for a 5-point or 6-point Likert scale. The 5-point scale contains a neutral mid-point and intended mean and results will be comparable between the old and proposed rating scores (e.g. similar to GPA with 0-1-2-3-4 scale). In contrast, a 6-point Likert scale spreads responses slightly and forces respondents to choose a mid-point (either fair or good). Since the January Senate meeting, AOT reviewed recent literature and asked 4 survey researchers for advice. The preponderance of opinion is that a 6- or 7-point scale is preferred to spread responses and encourage mid-point decisions rather than neutral responses. AOT members recognize and appreciate the reasons expressed by faculty for a 5-point scale; however, we conclude that erring on the side of slightly greater spread and decisions by respondents encourages our committee to recommend adoption of the 6-point scale.

5In survey research, the order of questions will influence subsequent responses. Assuming that most students want to provide valid information, both AOT and UW want to minimize the "conditioning response" for questions associated with P&T, merit, or awards. Again, AOT members respect the reasons for placing the questions as a summary at the bottom of the first page.

6Questions were selected with permission or modified as follows:

    #3. validated UW Form B; outcomes suggested as instruction adapts to outcomes based learning.
    #4. validated UW Form A.
    #5. validated UW Form A.
    #6. validated UW Form A.
    #7. UW and KSU ask several questions; AOT adapted this question from WSU student/teacher competencies form "using a variety of instructional techniques to accommodate differences in learning styles and abilities among students".
    #8. validated UW Form A.
    #9. adapted from KSU "Teval" form "stimulating you to think more deeply about the subject".
    #10. adapted UW Form X "Meaningful feedback on tests and other work was provided".
    #11. adapted from Univ of Minn; "instructor treated me with respect"; most discussions with faculty suggest that "all students" will encourage broader interpretation of question than "me".
    #12. adapted from UW Form X "evaluation of student performance was related to important course goals".

7Faculty suggested that narrative questions could be printed on the back to save paper and expense. However, confidentiality may be compromised in small classes, thereby requiring separate pages.

8Cost estimate includes $500 for printing a new master, $1000 to reconfigure OSU scanner for dual-side scanning, and $5000 for writing program to report both sides of OSU form; total $6500.


References

Cashin, W.E. 1988. Student Ratings of Teaching: A Summary of the Research. IDEA Paper No. 20. http://www.idea.ksu.edu/papers/pdf/Idea_Paper_20.pdf.

Cuseo, J. 2000. Evaluating new-student seminars & other first-year courses via course-evaluation surveys: Research-based recommendations regarding instrument construction & administration, data analysis, data summary, & reporting results. Marymount College. http://www.brevard.edu/fyc/fya/CuseoLink.htm.

England, J., P. Hutchings, and W.J. McKeachie. 1996. The professional evaluation of teaching. American Council of Learned Societies Occasional Paper No. 33. http://www.acls.org/op33.htm.

Hoyt, D.P. and W.H. Pallett. 1999. Appraising teaching effectiveness: Beyond student ratings. IDEA Paper #36. http://www.idea.ksu.edu/papers/pdf/Idea_Paper_36.pdf.

Gillmore, G.M. 2000. Drawing inferences about instructors: The inter-class reliability of student ratings of instruction. OEA Research Reports 00-02. http://www.washington.edu/oea/0002.htm.

Marsh, H.W. and L.A. Roche. 1997. Making students' evaluations of teaching effectiveness effective. American Psychologist. 52:1187-1197.

McKeachie, W.J. 1997. Student ratings: The validity of use. American Psychologist. 52:1218-1225.


| Agendas | Bylaws | Committees/Councils | Faculty Forum Papers | Handbook | Meetings/Locations | Membership | Minutes |