Requirements Analysis

The OVCAA Program Approval and Review site contains a number of documents that provide insight into the kinds of improvement goals found important by various academic units at the University of Hawaii.

We analyzed documentation from the reviews of seven academic units (ICS, SOEST, Social Work, TIM, Nursing, Social Sciences, and Pacific/Asian Studies) to generate proposals for measures and assessment mechanisms that appear amenable to visualization and tracking using a dashboard.

The following sections describe findings from each of the seven reviews.

Measures of "health" and "performance"

This section presents several measures that appear in multiple review documents and that provide a rough indication of unit "health". Much like a medical thermometer, a "normal" reading for any of the following measures does not guarantee the absence of problems, and an "abnormal" reading does not provide insight into the underlying causal factors. That said, just like a temperature reading, the following measures are relatively easy to collect and, in combination with other information, can be combined together to provide useful insight. Other measures in this section provide some insight into unit "performance" (though performance must be balanced with "quality", as discussed below).

Student Semester Hours (SSH). The total SSH delivered each AY by an academic unit is one measure of the "reach" of a program to UH students. Units strive to keep SSH level or to increase it. Decreasing SSH is not seen as a positive trend. SSH categories include undergraduate and graduate.

(Faculty) Full Time Equivalents (FTE). Most reviews also discuss issues around faculty staffing per AY, often in the context of upcoming faculty retirements and the worry that the unit will not be able to hire replacements. Decreasing FTE is not seen as a positive trend. Note that not all FTEs are the same: some units distinguish tenure track hires from non-tenure track. So, the dashboard should provide measures that count total FTEs available to a unit, as well as breakdowns by type (tenure track vs. non-tenure track).

Extramural Funding. All units attach importance to extramural funding per AY, though some units appear to focus on improving the number of proposals submitted as opposed to the actual amount of extramural funding received in a given year. This indicates that the dashboard should provide a few different measures related to extramural funding: the number of proposals submitted, the dollar value of proposals submitted, as well as the total number and dollar value of proposals funded.

Graduation rate. Most units view the number of graduates per AY, at both undergraduate and graduate levels, as a key measure of its impact.

Retention. Several units view retention as an important issue. Retention is the percentage of students beginning an academic program who go on to complete it. A low retention rate could be due to any number of factors, but in general, a low retention rate indicates an opportunity to improve the degree program in some manner.

Diversity. Units in STEM view diversity (both in terms of women and underrepresented groups) as a challenge. To better understand the issue, the dashboard can measure the total numbers as well as percentage of women and underrepresented groups at various points in the educational program (i.e. admission, 100-level, 200-level, 300-level, 400-level, undergraduate degrees, and graduate admissions and degree).

Publications. Several of the reviews stress the importance of faculty productivity with respect to publications. Some units also focus on the quality of the publication venue as an important feature. So, both "quantity" and "quality" of publications per AY appear important to track, though "quality" in particular will have a unit-specific definition. In the case of Nursing, the progress report included a recommendation that all Ph.D. dissertations "include a minimum of three papers that reflect various aspects of the students’ doctoral work that have been accepted for publication in referred journals." For this program, it can be useful to track the number of publications per Ph.D. student.

Graduate Student Support. Many of the reviews indicate the importance of maintaining or increasing graduate student support, either through intramural funding or through extramural funding. So, an important measure is one that tracks the total number of graduate FTEs supported per AY, as well as whether the source of that support was intramural or extramural. SOEST extended that to the numbers of Post-Docs, and to the livability of their compensation package.

Ranking. A few reviews discussed their ranking by third parties (US News and World Report, World University Rankings). This indicates that ranking can be a useful measure to track for certain academic units.

Measures of "quality"

The above measures are not sufficient to answer a simple but important question: "Is the academic unit doing a good job?" Obviously, what constitutes a "good job" is defined differently by different units. In addition, different units confront different barriers to achieving what they consider to be a good job. But all of the reviews indicate that units have goals beyond "health" and "performance": they want to (at least) maintain if not improve the "quality" of their research, education, and/or service to the community.

Quality is problematic to define, let alone measure. Upon analysis of the reviews, it appears that the most straightforward and practical way to gain insight into many of the "quality" issues raised by units is through periodic assessment of impacted stakeholder groups.

The example assessment questions are designed to be answered on a five point scale (from strongly agree to strongly disagree). This provides a way to generate an aggregate score that can track changes over time.

Department-wide Course Evaluation Questions. Several reviews indicate goals regarding the curriculum that could be assessed through the incorporation of department-wide questions into the end-of-semester course evaluation process. For example, "As a professional student, I found this course attractive and compatible with my work life" (SPAS), "This course provided opportunities for me to improve my software development skills" (ICS), "This course provided me with useful interdisciplinary experiences." (Nursing)

Exit Surveys. An exit survey for graduating students can provide answers to several questions found important in reviews. For example: "I feel I am well-qualified to move on to a Post-Doc position." (Nursing), and "I was well prepared to take part in an internship during my degree program." (Social Sciences, ICS), "I received useful mentoring regarding my degree program and my professional development?" (SOEST), "My workplace provided a healthy, safe, equitable and productive climate in which all employees and students treated each other with dignity and respect." (SOEST, Communications).

Stakeholder Surveys. The above assessments provide insight into the quality of education from the student perspective. There are other stakeholders beyond students, such as local employers or state government. Such stakeholders might wish to benefit not simply from the students graduated by the academic unit, but also by the research performed by the unit, or through consulting relationships with faculty, or joint projects, and so forth. One way to gain insight into the impact of an academic unit beyond its students is through periodic surveys of various stakeholder groups who have the potential to benefit from the department. Because this is a relatively costly and time-consuming endeavor, it could occur only every two years. Sample questions include: "I am satisfied with the quality of graduates from (academic unit)", "I have ample opportunities to collaborate with members of (academic unit)."

Local Impact. Several reviews make it clear that many departments feel an obligation to make a positive impact on the State of Hawaii beyond the education of local students. This is through creation of research products directly related to Hawaii, through the creation of community programs, or other activities.

Finally, it is very important to note that quality must be measured in order to provide a counterbalance to the various measures of "performance" described in the previous section. For example, focusing only on improved "performance" could lead a unit to increase SSH per faculty member to levels that make a severe negative impact on the student experience. If the dashboard attempts to measure "performance", it must also attempt to measure "quality" in order to prevent these forms of measurement dysfunction.

Teaching, research, and service

The canonical responsibilities of a faculty member, and by extension an academic unit, are teaching, research and service. The above measures seem to adequately cover the "health", "performance", and "quality" of teaching. The "health" and "performance" of research appear to be covered through measures of publications, extramural funding, and graduate student exit interviews (which can assess the ability of the graduate program to prepare students for research careers).

It is interesting to note that in the context of these reviews, "service" is entirely defined in terms of impact on the local community. No review considered (for example) "membership on conference program committees" to be an important indicator or activity in need of support or improvement. At present, the most appropriate assessment of service appears to occur through stakeholder survey responses.