An established order for a higher purpose

On August 31, the Office of the Higher Education Commission (Ohec) made history when it publicly announced Thailand's top 50 universities, pursuant to the findings of the 2005 University Ranking Project.

Unlike university rankings conducted by foreign media, the response to Ohec's rankings was negative overall.

Out of nearly 1,200 university students surveyed by the Abac Poll on September 1-2, about 24 percent expressed confidence in the ranking results, but 39 percent expressed no confidence, while 37 percent had no comment. The majority of students also believe that they will have a more difficult time securing jobs, if their universities are listed toward the bottom.

Many universities publicly criticised the official rankings, citing the unreliability of data and possible biases of Ohec as the main reasons. Having disagreed with the rankings methodology from the outset, more than half of the universities nationwide, including some faculties at universities that did participate, never submitted all of the data required for Ohec to properly establish the rankings. Among the universities that submitted incomplete data is Kasetsart University.

``The rankings process is different from that of a poll. Rankings are based on a set of data -reliable data - sound data collection protocols, calculations and verifications,'' says Dr Surapol Patharakorn, dean of Kasetsart University's faculty of science, during a recent seminar on university rankings at the thirty-second Congress of Science and Technology.

According to Dr Surapol, his faculty submitted the required data, but somehow Ohec may not have received them.

To some university officials, the August 31 announcement was incomplete because it focused primarily on the rankings results and the grouping of universities based on their total scores. Missing was the methodologies used to develop the rankings, which normally are disclosed along with the results, as is done by many foreign publications that publish university rankings. Without a candid disclosure of the methodologies used, the results are merely meaningless numbers, and may be justifiably questioned. In the beginning

The 2005 University Ranking Project, nicknamed Ranking 2005, is a descendant of a previous Ohec project to evaluate the status of research in state universities. Initially, the scale of the project involved a few university faculties, such as math, medicine, science, and agriculture.

``The objective at that time was to determine the factors that influence research, such as staff and budget,'' says Prof Dr Supot Hannogbua from Chulalongkorn University, a member of the working committee for the 2005 University Ranking Project.

With Ohec's support, the project expanded to include additional faculties, which later were grouped into seven academic disciplines: science, technology, biomedicine, humanities and arts, social science, agriculture, and education. When Ohec announced the Ranking 2005 project in a meeting with the Council of University Presidents on December 17, 2005, 137 universities that were under Ohec's supervision became the targets of the project.

Following the expansion, there was a shift in the project's objective to evaluate the efficiency of universities in their teaching and research functions through an online database system.

``Ohec plans to use this database for the benefit of managing the allocation of funds to universities, which are presently scattered among schools. The funding will be used to help universities retain more researchers and lecturers who have PhD's,'' Dr Supot says.

Ohec methods revealed

Under the online database system, all data collections and communications between the working committees and universities are via the project's website. ``When we designed the programme, we assumed that most of the required data would be supplied by each university faculty,'' says Dr Supot. The data input process, he adds, requires that data be collected at both the faculty and university levels.

By consulting with fellow academics and reviewing the methodologies adopted by foreign publications and the external quality assessment documents of the Office for National Education Standards and Quality Assessment (Onesqa), the working committee generated their own list of 31 key performance indicators (KPIs), comprising nine rankings criteria.

To determine a university's research rank, 17 research ranking indicators (RRIs) are grouped under four criteria, namely funding (20 percent), personnel (20 percent), research output (45 percent), and post-graduate student composition (15 percent).

Similarly, methodology to establish a university's teaching rank is divided into 16 teaching ranking indicators (TRIs) and six criteria, including student selectivity (20 percent), student-lecturer ratio (20 percent), faculty resources (20 percent), financial resources (20 percent), internationality (10 percent), and quality of education (10 percent). Soon after, the committee scrapped the student selectivity criteria because it cannot be measured or verified.

``In the first year, we emphasised the selection of indicators that can be verified,'' says Dr Supot. ``Many of these indicators are very good ones, but the data cannot be collected. So, we tried to select the ones that can be represented numerically.''

Initially, the working committee allocated one month, January, for data collection and input; instead, it required eight months, until it ended in August. Also, the numbers collected by Ohec and those submitted by the universities didn't coincide. ``We found up to 20-30 percent discrepancies between the data submitted by universities and those that Ohec has in hand. Among the problematic ones are the number of lecturers, academic programmes, and faculties,'' Dr Supot explains, adding that in the end, the committee decided to trust the numbers submitted by the universities.

``Whether or not the final results are reliable depends on the two sets of RRIs and TRIs that we use. We felt uneasy that some data were missing, so we designed the programme to lower the overall scores of each university that submitted incomplete sets of data,'' he says.

Question of reliability

These early warning signs, coupled with the selection of RRIs and TRIs, became major points of concern among university officials after the announcement of the rankings results in August.

``We all know that working behaviours change according to the indicators used to measure the performance,'' says Assoc Prof Dhirayos Wititsuwannakul, dean of Walailuk University's School of Science.

``What we don't know is, based on these indicators, where is Ohec steering the universities? Right now, the set of ranking criteria adopted by Ohec suggests that there can be only one type of university. In fact, there are many university types in this country,'' he adds. On a similar note, Assoc Prof Chayan Boonyarak, dean of Naresuan University's faculty of science, points out that basing the university rankings project on the teaching and research functions alone is insufficient.

According to the university act, a university has four major functions: teaching (35 percent), research (35 percent), educational services (20 percent), and maintenance of cultural and artistic values (10 percent). ``Universities in Bangkok provide very little educational services and maintenance functions to communities, but those in the provinces perform these two functions a lot,'' he says.

Ohec's Ranking 2005 project also overlooked an important methodology in the science of ranking: peer review. Rather than relying heavily on data submitted by the universities themselves, the rankings of foreign publications usually include data obtained by opinions of a university's peers, people in the education sector.

The Thes, for example, clearly states in its disclosed methodology of its 2005 World University Rankings that peer review was the core of its analysis and was given 40 percent weighting.

In addition to peer reviews, 10 percent was derived from opinions of major international employers and of graduates; 20 percent from the number of citations for academic papers generated by each staff member; 20 percent from the staff-to-student ratio; and five percent each was derived from the percentage of international staff and students.

The Thes rankings took each type of university into account and excluded institutions that did not teach at the undergraduate level. This, however, is not the case for Ohec's rankings.

Similar to The Thes, the 2000 Asiaweek's rankings of top Asian universities designated 20 percent weighting to peer review, in which a factor representing the academic reputation is derived. The rest of the criteria are distributed among student selectivity (25 percent), faculty resources (25 percent), research output (20 percent), and financial resources (10 percent).

Judging from the poll results and the universities' responses to the final rankings, Dr Supot and his team acknowledge that more work is required to improve the project's reliability. And despite the negative responses by the interviewees of the poll, most academics agree that the rankings present a good way to encourage universities to improve their own performance. In the future, perhaps Ohec will consider disclosing the rankings methodologies it uses, together with the published ranking results,. This way, the public can determine for themselves whether or not to trust the numbers they see.

At this point, however, the future of Ohec's university rankings project remains unclear, as the September 19 coup interrupted the process of instaling Ohec's new secretary general to replace the retired Pavich Thongroach. So far, the ministry of education, under the leadership of Dr Wijit Srisa-arn, has not decided on Pavich's replacement. Had the project been discontinued, university officials say, those universities that appear at the bottom of the list would have no way to redeem themselves to the public. Whether or not Ohec continues to rank universities remains to be seen.

For more information on the 2005 University Ranking project, visit www.ranking.mua.go.th . For the latest in university world rankings visit http://www.thes.co.uk/worldrankings/ .