ICS External Review Report (2019)

The 2019 External Review of Information and Computer Science reveals a number of potential measures, visualizations, and applications for helping a department to improve through a dashboard:

On peer institutions

The external review report included:

In an attempt to quantify [ICS performance as a department], we compared the data we were provided for ICS with the data reported in the Computing Research Association’s 2018 Taulbee Survey for 29 Computer Science programs at public universities with 15-25 faculty. (Note that many of these programs are not at R1 universities such as UH Mānoa, where one would expect a higher level of performance.)"

This provides one approach to defining "peer institutions" for dashboard visualizations that require them.

On useful measures to track in a dashboard

The external review report included the following bullet points:

• The number of Bachelors degrees granted annually per tenure-track faculty member in ICS (including the BS in CS and the BA in ICS) is 72% of the median of these other programs.
• Annual Masters degrees granted per tenure-track faculty member (including the MS in CS but not the MLISc) are only 1/3 of the median.
• Annual PhD degrees granted per tenure-track faculty member (including the PhD in CS and an appropriate proportion of the PhD in C&IS) also are roughly 1/3 of the median.
• Annual research expenditures per faculty member are 84% of the median, but this is largely due to only 3 faculty members. Half of the ICS faculty has not had any external research support in the past 5 years, and several others have had non-zero but very small levels of external support.

The above comments indicate measures (such as "B.S. degrees granted annually") as well as ways to normalize the values for comparison to peer institutions (such as "per tenure-track faculty member").

ICS currently falls below the median for all of these measures. A dashboard could recalculate these measures each year to see if changes in the department are producing positive movement.

On student preparation

The external review report included:

We heard a consistent message from students, from employers, and from members of the university community who employ ICS students that the ICS Bachelors programs – BS and BA – produce too many students who are not prepared to practice: they are lacking the skills that employers expect of Computer Science graduates. Many courses are based on “book learning” and lack appropriate programming (skill-building) assignments/projects; the programming component tends to be delegated to optional senior-level courses where it might be too late for students to garner internships and compete in the job market. Courses in key areas such as Artificial Intelligence and Cybersecurity are sometimes outdated or not focused appropriately.

This comment suggests at least the following measures:

• A department-wide course evaluation question such as "Do you feel this course included useful skill-building assignments and projects?" This would provide insight from the student community about each course in the curriculum.

• A yearly or bi-annual survey of employers and members of the university community that includes a question such as, "Do you feel recent ICS graduates are prepared to practice?" This would provide insight about the net effect of the ICS curriculum on professional preparation.

On community engagement

The external review report included:

We heard from external constituencies – from elsewhere in the university, and from the outside community – that engagement is limited. These external constituencies want greater engagement, and are willing to invest in achieving it.

This suggests another question for a yearly or bi-annual survey of employers and members of the university community: "Do you feel that you have sufficient engagement with the ICS Department?"

On graduate students

The external review report included:

At the graduate level, students feel they have few advisors to choose from. Because of the low level of research funding, new PhD students are supported as Graduate Teaching Assistants (GTAs), so don’t have the opportunity to get involved in research early, delaying their maturity as researchers. GTAs feel that they are over-worked, leaving them little time for more creative/constructive activities.

On the computation of attrition

The external review report included:

We noted in the main body of the report that steps have been taken recently that have significantly reduced the attrition rate of undergraduates at all levels. To determine this, we viewed attrition in a way that differed from the charts we were provided in two ways:

• First, for any particular cohort (i.e., for students who arrived in a particular year), we computed the attrition in a particular year (e.g., the attrition in the third year) as a percentage of the students who entered that year, rather than as a percentage of the students who entered as freshmen.
• Second, for each calendar year, we looked at the attrition in students’ first year, in students’ second year, in students’ third year, etc. This allowed us to look for changes in overall retention in a particular year."

This provides an interesting way to compute and visualize attrition for display in the dashboard. Tracking attrition over time provides a way for the department to see if interventions are working.

On the need for accurate data

The external review report included:

Preparation for the review would have been facilitated by more accurate data. We encountered many inconsistencies in the tallies of research awards – probably due to the fact that faculty are associated with multiple units, and awards are attributed in different ways at different times. But we even had to dig in order to determine the active faculty in ICS.

Hopefully, a dashboard can address this problem.

Their recommendations

The external review report included the following recommendations:

1. Increase the number of Computer Science graduates from UH Mānoa.
2. Increase the employability of UH Mānoa Computer Science graduates, with emphasis on local private sector and public sector employers.
3. Increase interdisciplinary research collaboration by Computer Science faculty.
4. Increase engagement with external public sector and private sector entities.
5. Increase the faculty productivity in terms of PhD production, research expenditures, etc.
6. Increase entrepreneurial activities of CS faculty and students."

Each of these appears amenable to measurement and tracking in a department dashboard. And their presence in a dashboard would provide concrete evidence regarding the impact of changes made by ICS in response to the external review report.