| Theme
1: Becoming a Teaching and Learning Community
Reflective Essay on Theme I, Issue No. 5: Student Learning
Outcomes Assessment and Program Effectiveness
Program Reviews and Measurement of General Achievement
and Satisfaction
The process of periodic program review is well established
in the academic culture at CSUSB. All degree programs are
required to undergo a review every five years, which includes
a prescribed self-study report, use of off-campus consultants
as reviewers, preparation of a summary and recommendations
by a university-wide committee, and development by the departments
of a action plan for follow up and improvement where needed.
The guidelines for the review were recently revised to call
for more focus on student learning outcomes and reporting
of the findings of annual assessments and evaluation of students,
as well as more emphasis on graduate programs, research, community
outreach and service, and external funding.
A key component of the self-study process is a survey of
current students and alumni, which seeks opinion and comments
on a wide variety of issues, including advising, satisfaction
with instruction, curriculum, and operations of the program.
These surveys are corroborated by use of similar questions
on surveys of graduating seniors and regular surveys done
during quarterly registration, which allows comparison across
programs and colleges as well as trend data. Students generally
praise the quality of instruction, the preparation, effectiveness,
enthusiasm, and level of assistance from faculty. The availability
amount and quality of advising, especially for career planning,
is the most consistent concern of students. A recent analysis
of the advising process in each department has led to increased
emphasis on making annual advising mandatory. The campus has
implemented a new Student Transition, Advising and Retention
Program (STARS http://thewasc.csusb.edu/repository/STARS.htm)
along with an on-line degree audit summary for easy analysis
and assessment of transfer credit, course planning, and reports
on progress toward degree. Each department now is being asked
to prepare and publicize a four-year schedule and rotation
of course offerings to encourage additional planning and earlier
graduation. (See also STARS report on retention efforts http://thewasc.csusb.edu/repository/RetentionReport2002.htm )
Our Graduating Student Surveys (http://ir.csusb.edu/commencement-2001.pdf ) have yielded a good deal of useful information about Campus
Environment. General Attitudes, Educational Expectations and
Achievement, CSUSB's Contribution to that Achievement, Experience
of Inappropriate Treatment, Satisfaction with the Educational
Experience, and future Professional Goals and Activities.
The results of these surveys have generally been gratifying
and our alumni express a great deal of satisfaction with their
experience at Cal State San Bernardino. Issues identified
in the commencement survey have been followed up on and probed
further in a number of other useful surveys, including the
Student Needs and Priorities Survey (SNAPS) http://ir.csusb.edu/SNAPS99.pdf , participation in pilot testing of the National Survey of
Student Engagement http://thewasc.csusb.edu/data/NSSE2002Slides.pdf,
the ETS Academic Profile exam of General Education Achievement,
and CSU Benchmarking studies. The SNAPS report for 1999 was
especially impressive because our campus was rated significantly
higher than the average for the CSU system in almost every
measure-and our scores had improved in most categories over
10% over the 1994 administration of the survey. Likewise,
the use of comparative data from other national instruments
and studies is beginning to focus attention on areas where
improvement and change is needed. (The findings and implications
of these very extensive surveys are discussed in detail in
the full report on this Issue and the raw data and analysis
for each survey can be found in the Appendix.)
With a gradual decline in the percentage responding to mail-in
surveys, the campus has turned to use of brief surveys taken
of all students as part of the quarterly telephone and web-based
registration process. By asking just 3-5 questions of a very
large number of students we have been able get better samples
and more significant results. These registration surveys also
allow us to do better breakdown of response major, age, ethnicity,
class rank, and other demographic characteristics. As more
students begin to use the Web for registration we will be
able to ask additional and more complexly structured questions
of each students.
Student Learning Outcomes in the Majors
CSUSB has moved forward steadily in its efforts to establish
a culture of evidence about our students' learning. On a scale
of 1 to 5, with "one," the lowest step, representing
reliance on traditional grading methods, step "two"
limited to discussion about the need for assessment, step
"three" characterized by individual initiatives
and experimentation, step "four" signifying organized
effort across the institution and step "five" full
implementation, CSUSB ranks at number four with a bullet.
CSUSB has a number of outcomes assessment structures and
processes in place. The outcomes assessment initiative at
CSUSB is organized around an infrastructure, institutional
funding, an array of planned and tracked assessment initiatives,
formative and summative assessment methodologies based on
agreed upon programmatic learning goals and objectives, and
developing baselines and norms. Overall, assessment practices
at CSUSB are being implemented in many areas of the campus
in a visible, systematic, and substantial way. (See http://gradstudies.csusb.edu/outcome/)
Although we have advanced steadily over the past decade in
the development of outcomes assessment campus wide, there
is still more we need to accomplish in order to reach the
top of the scale and become an institution where outcomes
assessment has been fully implemented across the institution,
creating a fully embedded culture of quality characterized
by continuous improvement and shared organizational learning.
Central to both the university's mission and strategic plan
is the improvement of student learning through assessment
of programs. The assessment of student learning is a central
strategy in CSUSB's primary goal to become "a learning
community that excels in creating, applying, and exchanging
knowledge." (See http://www.csusb.edu/president/vm.html and http://www.csusb.edu/president/strategicplan.html)
For nearly a decade, CSUSB has worked systematically to organize
itself to attain its educational objectives through outcomes
assessment. The campus led the CSU system movement in student
learning outcomes in the early 1990's, and our assessment
plans and methods were used by the CSU Chancellor's office
as a model for other campuses to emulate and build on. Assessment
plans have been developed for most of the majors and a significant
number of departments have already evaluated their programs
under self-designed and institution-approved instruments,
reported their results, and begun to use the findings for
program improvement.
Since 1994, CSUSB has been working to ensure that programs
have clear academic standards for student achievement in place.
As of Spring 2003, 83% of the undergraduate and graduate programs
at CSUSB (excluding Interdisciplinary programs) have submitted
outcomes assessment plans to the University Outcomes Assessment
Committee. Of those submitted, 69% have been approved for
implementation by the committee. Overall, 58% of the undergraduate
and graduate programs at CSUSB (excluding Interdisciplinary
programs) have university-approved outcomes assessment plans.
In addition to the assessment plans, programs have been required
to submit annual assessment status reports each year since
1998. In their reports, programs outline their assessment
activities, findings, and how they have used the results for
program improvement. Forming an integral component of CSUSB's
commitment to continuous program improvement and student learning,
the annual status reports have begun to provide us with a
systematic method to gather specific information on student
learning and how departments are using what they have learned
for program improvement. CSUSB makes this information public
through its annual Accountability reports to the CSU Board
of Trustees. (See http://gradstudies.csusb.edu/outcome/bycollege.html and http://gradstudies.csusb.edu/outcome/statusrpt.html)
Evidence contained in the assessment plans and status reports,
suggests that CSUSB is gradually becoming an assessment-oriented
institution where direct evidence of student learning is being
used by programs to ascertain whether they are achieving their
key educational goals. One result of outcomes assessment has
been an increase in the spirit of inquiry and self-study within
programs. Status reports show that the majority of programs
use assessment information to facilitate curriculum discussions
at faculty meetings, curriculum committee meetings, and faculty
retreats. Programs have begun also to employ their assessment
findings to shape curriculum. They have used their assessment
findings to add and delete courses, change course sequences,
revise course content, guide changes in degree programs, develop
new degree program options, change degree requirements, and
justify past curriculum changes to show program improvement
resulting from those changes.
For other programs, assessment information has led to a greater
emphasis on advising, and other institutional functions designed
to enhance student development, resulting in some departments'
development of new avenues of career exploration and career
services for their students. For other programs, the results
from their outcomes assessment activities have led to changes
in instructional emphasis for current faculty and changes
in emphasis for new and vacant faculty positions. Assessment
information has enabled some programs to strengthen ties with
the community by sharing assessment information with alumni,
review boards, and advisory boards composed of community leaders.
Signaling a growing willingness to engage in a continuous
process of self-examination, more than half of the departments
with an approved assessment plan report that besides aiding
them in curriculum and staffing decisions, assessment information
is useful in helping them to further refine their assessment
methods and implement new ones.
(See http://gradstudies.csusb.edu/outcome/improvements.html.)
Although the campus has made significant progress in organizing
itself to become a learning community committed to the use
of outcomes assessment to improve student learning, there
are still a number of items we need to accomplish to achieve
full implementation. First, in order to demonstrate in a more
comprehensive manner that learning is occurring and that our
key educational goals are being achieved, all of our programs,
including the undergraduate and graduate degree programs must
be involved in the assessment process. In addition, we need
to create a comprehensive student database by combining the
information on student learning provided by the academic programs
with the many other university activities that yield a wealth
of information about our students. These measures include
our Graduating Student Surveys, TRACS surveys, benchmarking
and customer satisfaction surveys, and such system-wide and
national surveys as the NSSE and SNAPS. Such a step would
provide a common tool for all members of the University to
engage in a continuous process of self-examination whose outcome
is the enhancement of student learning at CSUSB. Furthermore,
university-wide engagement in student learning would contribute
greatly to our efforts to create learning communities comprised
of groups of students and faculty, sharing common interests
and goals. The implementation of assessment university-wide
and the creation of a common database would assist the CSUSB
Learning Research Institute (LRI) whose charge is to promote
interdisciplinary scholarly focus on teaching and learning
and how it may be improved.
The University needs to align more closely its outcomes assessment
structures and processes with other core university functions
involved in helping CSUSB meet its institutional objectives
and standards of academic achievement so that they are mutually
supporting. We need to coordinate our outcomes assessment
efforts with our efforts in the area of faculty development.
A natural point of linkage is with the Teaching Resource Center
(TRC). The TRC's substantial work in the area of faculty development,
especially its support of teaching, is one of the institutional
strategies that helps improve learning through effective teaching,
thus contributing to educational effectiveness. The bridging
of outcomes assessment and the initiatives of the TRC can
contribute to our educational effectiveness by what leading
to classroom assessment, classroom research, and the use of
classroom assignments for assessment. We also need to coordinate
our outcomes assessment efforts with the work of the University
General Education and Curriculum committees. We would like
to establish a linkage between outcomes assessment and the
processes of advising, and mentoring, especially the activities
of the Office of Academic Services and Advising, so that familiarizing
students with the requirements for program completion goes
hand-in-hand with their understanding of the learning goals
and objectives for the major.
We have recently begun to connect assessment with the broader
process of program review, so that these formal self-studies
reflect a greater emphasis on student learning outcomes. Program
reviews cover many different dimensions of a department's
functioning, including average class size, number of majors,
and faculty resources.
Student learning and development is another of the many aspects
of the unit that need to be examined as part of the departmental
self-study. By incorporating assessment more integrally into
the review process, the campus will become more in line with
state, regional, and national accrediting agencies that are
now heavily focused on student learning outcomes. Not only
does this revision in the self-study document lead to increased
efficiency and decrease the duplication of efforts, more important,
it will highlight the integral link between inputs (number
of faculty, of resources, a library holdings, etc) and outcomes,
the measurement of student learning according to agreed upon
academic goals.
As our outcomes assessment efforts become more established
and mature we anticipate creating a closer link between outcomes
assessment and the program planning and budgeting process.
Our goal is to link outcomes assessment and the program approval
and revision process. Substantial revisions in existing programs
and decisions about future ones would be based on information
gained from student learning outcomes.
Finally, the results of assessment must be used to improve
assessment itself. We need to establish a mechanism by which
faculty can critically examine the assessment process to see
if it is providing the information they need for themselves
and their students.
General Education Outcomes Assessment
Although CSUSB ranks squarely at number "four"
based on its development of organized and systematic outcomes
assessment measures for the majors, we are probably closer
to being at step "three" in our development of outcomes
assessment for general education. While our assessment for
the majors is organized around an infrastructure, institutional
funding, an array of planned and tracked assessment initiatives,
formative and summative assessment methodologies based on
agreed upon programmatic learning goals and objectives, and
developing baselines and norms, our assessment of general
education is not yet developed to the same degree. Currently,
general educational outcomes assessment is characterized by
individual institutional initiatives and experimentation as
a way to formulate a sustainable university-wide assessment
infrastructure.
The total GE program is reviewed every five years with a
focus each year on one of the categories (Basic Skills, and
breath areas of Humanities, Social Sciences, Natural Sciences,
Multi-cultural and Gender Perspectives and Lifelong Learning).
This review by the University GE Committee has focused primarily
on analysis of course syllabi, assignments, and the consistency
and rigor of the course content as well as whether each course
continues to meet the criteria established for the particular
category. Very minor modifications and changes in the courses
and curriculum have resulted from this overall review of the
GE program.
Since 2000, we have made a number of major steps in our development
of assessment for the General Education Program. Faculty members
serving on the University General Education Outcomes Assessment
Committee and the General Education Breadth Area Assessment
Committees have developed learning goals and objectives for
the four General Education Basic Skills areas (Written Communication,
Oral Communication, Mathematics, and Critical Thinking) and
the three General Education Breadth areas (Natural Sciences,
Humanities and Social and Behavioral Sciences). (See http://gradstudies.csusb.edu/outcome/BasicSkills.html and http://gradstudies.csusb.edu/outcome/BreadthArea.html.)
They based the goals on university-wide input from faculty,
department chairs, and university, college and department
GE, curriculum and outcomes assessment representatives and
carefully aligned them with the University General Education
Policy. (See http://gradstudies.csusb.edu/outcome/Alignment.html.)
During the 2000-2001 AY, we began our assessment of the basic
skills areas by administering the Academic Profile (ETS).
Several issues emerged from our use of this instrument. Because
it did not focus on our curriculum, specifically our outcomes
goals and objectives, the information we received from the
Academic Profile was too general and less useful than we had
hoped as an indicator of the effectiveness of our GE program.
This is a problem inherent in such surveys. In several classes,
students completed the exam as an "extra credit"
option, resulting in little incentive to perform well. In
addition, our campus' scores were below the mean.
One major outgrowth of our experience with the Academic Profile
was our realization that we need to develop local instruments
that will more readily assess our particular constellation
of courses. Therefore, in Fall 2002, the GE assessment committees
developed and began piloting campus-based embedded assessment
of student learning in the breadth areas. To date they have
piloted questions in fifteen general education breadth area
courses, involving nearly 2,000 students.
In spring 2003 the breadth area committees began their analysis
of the results from the pilot project with particular focus
on what student responses tell us about (1) the alignment
between the goals and objectives and types of embedded exam
questions, (2) how students are meeting the goals and objectives,
(3) the usefulness of the data in closing the assessment loop
and shaping curriculum, and (4) the feasibility of developing
a faculty-supported sustainable assessment infrastructure
for CSUSB's General Education program. We have learned that
although our exam questions and learning goals parallel one
another closely, we need to develop mechanisms to ensure consistent
alignment, especially as we assess more and more courses.
Our preliminary data suggests that our students' performance
is adequate, ranging from 70 to 76 percent in correct responses
to the embedded exam questions, and over time, additional
findings will be valuable in helping us shape and improve
our GE curriculum. The pilot has shown us that our biggest
challenge lies in the development of a sustainable assessment
infrastructure for CSUSB's General Education program. This
challenge stems from two characteristics of our GE program:
its size, students must complete 82 quarter units, and, even
more problematic, the range and diversity of the program's
more than 150 courses from which students may choose. The
results of our pilot project suggest that a "one-size-fits-all"
structure and the same methodology are not practicable for
our program.
Besides our efforts to develop embedded assessment for the
GE program, CSUSB engages in a number of other GE assessment
activities, especially in the area of basic skills. We screen
the math and writing skills of all our incoming freshmen (EPT
and ELM exams), and require them to take remedial classes
if their skills are below a set standard. Our upper-division
writing requirement requires students to achieve proficiency
in critical thinking, organizational skills, and the ability
to use evidence. The upper-division writing requirement is
also a means by which we assess the writing skills of all
students entering our master's programs either by exam or
by course enrollment.
(See the Full Report on this topic of review and assessment
of General Education at: http://thewasc.csusb.edu/edueffrev/thm1/thm1iss5/thm1iss5(15).htm )
|
