University rankings: The web ranking

The publication in 2003 of the Ranking of Universities by Jiao Tong University of Shanghai has revolutionized not only academic studies on Higher Education, but has also had an important impact on the national policies and the individual strategies of the sector. The work gathers the main characteristics of this and other global university rankings, paying special attention to their potential benefits and limitations. The Web Ranking is analyzed in depth, presenting the model on which its compound indicator is based and analyzing its different variables. ------- Rankings de universidades: El ranking web Resumen La publicacion en 2003 del Ranking de Universidades de la Universidad Jiao Tong de Shanghai ha revolucionado no solo los estudios academicos sobre la Educacion Superior, sino que tambien ha tenido un importante impacto sobre las politicas nacionales y las estrategias individuales del sector. El trabajo recoge las principales caracteristicas de este y otros rankings mundiales de universidades, prestando especial atencion a sus potencialidades y limitaciones. Se analiza en profundidad el Ranking Web, presentando el modelo en el que se basa su indicador compuesto y analizando sus diferentes variables y principales resultados. DOI : 10.18870/hlrc.v2i1.56 PDF document contains both the original in Spanish and an English translation.


Rankings Mundiales de Universidades
Tras la publicación del Ranking de Shanghai, a la vista de su impacto y aprovechando ciertas carencias, otros rankings con pretendida cobertura mundial han ido apareciendo. Se pueden dividir en dos grandes grupos, rechazando de esta lista aquellos que son anónimos, no tienen base metodológica clara, o presentan fuertes sesgos en sus resultados: Rankings basados en encuestas de opinión. Se trata de dos rankings muy similares, derivados de un trabajo realizado por la empresa británica QS para la revista Times Higher Education desde 2004 y que a partir de 2010 dio lugar a dos productos distintos producidos por cada una de estas organizaciones. El núcleo central de los resultados se obtiene a partir de encuestas de opinión, complementados con datos bibliométricos. QS utiliza la base de datos Scopus/Elsevier, mientras THE se basa en la información proporcionada en Web of Knowledge (WoK) de Thomson/Reuters.
El Ranking Web utiliza uno de los catálogos de universidades en la Web más completos y actualizados, ya que estudia unas 20.000 instituciones de educación superior de todo el mundo. La figura 1 muestra la cobertura relativa por regiones del Ranking. Figura 1. Cobertura mundial y regional del Ranking Web de Universidades, con indicación de las instituciones situadas entre las mil primeras (Datos de Julio de 2011).
Entre las ventajas de este método destacaremos la posibilidad de incluir todas las misiones de la universidad, siempre que estas queden reflejadas en la web. Sin embargo, la simplicidad del sistema de rankings dificulta conocer con exactitud la contribución o importancia relativa de cada una de las misiones, por lo que siempre se ofrece una visión integrada en la que es necesario trabajos adicionales para identificar fortalezas o debilidades.
La Web es el canal futuro y presente para la enseñanza no presencial, el escaparate más económico y el más atractivo para reclutar los mejores estudiantes y profesores internacionales y es el punto de encuentro entre la universidad, la sociedad, sus actores económicos y los líderes políticos.
La brecha digital entre las universidades de EE.UU. y las del resto del mundo es preocupante pues supone un colonialismo científico y cultural sobrevenido de profundas consecuencias para el futuro.

Introduction
Before 2003, studies on Higher Education had a limited scope, with study populations that were either small or geographically or culturally biased. In most cases, it was an analysis of the situation in North America or Western Europe, with special attention paid to the model defined by North American universities intensely focused on research. The universities from the rest of the world were the subject matter merely of specific studies and those from developing countries rarely deserved any attention.
On the other hand, the classifications or rankings available were national in nature, frequently made by large-circulation newspapers or magazines, which purpose was to inform or guide future students when choosing a university on where to start or continue their studies. The criteria used merely described academic offerings and focused on the costs of education, the quality of life on campus or in the city, security and the offering of services of universities (Eccles, 2002). These criteria based, generally, on subjective perceptions, vague sources or very biased data.
Basically, there were no world rankings before the publication of what was known as the Shanghai Ranking (www.shanghairanking.com), the "Academic Ranking of World Universities" (ARWU), made by scientists from Jiao-Tong University of Shanghai, in the People's Republic of China (Liu & Cheng, 2005). In 2003, the first edition of this annual ranking was published. Its main novelties are:  A list of the top 500 universities in the world, with more than 40 countries represented.  A compound index combining variables related to the excellence of the institution, mainly regarding research results.  The ranking is made by researchers from a public university based on openly available data and sources.  Open publication on the Web, with a very simple consultation interface. It uses English, the lingua franca of the scientific community, as the language of the texts.
The Shanghai Ranking impact was immediate and profound. The reactions were diverse, but after some criticism (van Raan, 2005), different countries started to take political measures to adapt their national university systems in order to improve their results in the mentioned Ranking. Individual institutions also took similar measures and it can be said that the ranking has attained a level of transcendence and relevance that has a truly global reach (Marginson & van der Wende, 2007).
That impact seems to fly in the face of the rigorous and verified criticism in scientific articles that the Ranking received and continues to receive (Billaut, Bouyssou, & Vinke, 2009). Actually, in order to maintain the methodological stability and comparability among editions, the Shanghai Ranking has not corrected practically any of the problems that were found in it.
Among the most prominent:  The lack of specific indicators for technologies, social sciences and humanities, which forces specific treatment of certain very prestigious institutions (for example, the London School of Economics).  The bibliometric biases in favor of biomedical institutions, some of which do not have a clear university status.  The use of very old series for certain variables. Nobel Prizes were considered from their inception in 1900, although with decreasing importance according to the decade.  The use of the alma mater of the winners (Nobel or Field medals) as a measure of teaching quality.  The lack of very relevant awards, besides the mentioned Nobel and Field, especially in disciplines not included in them.  The exclusion of the universities of Berlin due to an alleged difficulty in assigning values from before World War II.  The acceptance of secondary institutions among the highly quoted scientists' affiliations (visiting professors in Saudi universities).  The combination of absolute and relative indices.

Global Ranking of Universities
After the publication of the Shanghai Ranking, in light of its impact and taking advantage of certain omissions, several other rankings have been published, with the goal of having global coverage. They can be divided into two large groups, excluding from the list those that are anonymous, have no clear methodological basis or show clear biases in their results: Rankings based on opinion surveys. They are two very similar rankings, derived from a worked performed by the British company QS for the magazine Times Higher Education from 2004 and that as of 2010 generated two different products produced by each of these organizations. The core of the results is obtained from opinion surveys and complemented with bibliometric data. QS uses the Scopus/Elsevier database, while THE is based on the information provided by Thomson/Reuter's Web of Knowledge (WoK).
Rankings based on Bibliometrics/Cybermetrics. There are four rankings that use, almost exclusively, quantitative data derived from their research results (scientific articles and bibliographic citations) or their Web presence (Web pages, links and mentions). The HEEACT ranking from Taiwan and the CWTS (University of Leiden, Holland) rely on the WoK data, while the ranking of the Scimago group (Spain) uses its competitor, Scopus. The Web Ranking or Webometrics share methodology but not sources and it uses, apart from data from search engines, information extracted from Google Scholar, a bibliographic database of citations similar to Scopus and Wok.
Just as with the Shanghai Ranking, these classifications have also been the object of criticism, although after more than 8 years of experience, most of them have managed to become part of political agendas and higher education strategies at a global level. As a matter of fact, today it is not uncommon for them to be used practically to validate degrees, grant student loans or even at parliamentary debates.
However, not all of them are the same and it is necessary to highlight both their strengths and weaknesses:  QS Ranking (www.topuniversities.com). This is one of the most commonly used rankings.
It provides information on more than 700 universities and offers rankings by region (Asia, Latin America) and by discipline. Strongly biased in favor of universities in countries where English is the main or principal language, it also over represents Asian countries. This was due to methodological problems with the surveys used and it appears to have been solved in the latest editions. As in other rankings, the universities occupying the last positions, beyond position 400, are not trustworthy.
 THE Ranking (www.timeshighereducation.co.uk/world-university-rankings). Presented as an evolution and improvement of the former, it includes several extra variables, whose relative contribution is quite subjective. The first editions have suffered from significant problems in their bibliometric calculations and from the lack of transparency about the universities actually evaluated. The latest version is still inadequate.
 Leiden Ranking (socialsciences.leiden.edu/cwts/products-services/leiden-ranking-2010cwts.html). This classification is produced by what is possibly the group that is the global leader in bibliometric research. The quality of the data is very good. However, unfortunately, five different rankings are presented that use exactly the same source information. The authors do not explicitly favor any of their own proposals and are also developing the main indicator, called the "Crown indicator", into a new one which is in fact quite different.  Taiwan Raking (ranking.heeact.edu.tw). Another exclusively bibliometric ranking, developed from a combination of variables that are strongly correlated and that does not seem to respond to a mixed model.
 Scimago Ranking (www.scimagoir.com). The only one that includes both universities and research centers, with a total of 3,000 institutions analyzed. Although it provides values for six different variables, some of which are only available here, the order of presentation is derived solely from the number of scientific publications according to the Scopus database. This lack of a true "ranking", as well as the biases in Scopus' coverage, render the comparability of its results with other rankings difficult.
 Webometrics Ranking (www.webometrics.info). This ranking will be discussed in detail below, but we can indicate, among its advantages, its broad coverage, both in number of universities (20,000) and in academic missions (teaching, research, commitment to the community, transfer of technology, internationalization). However, since it is a webbased ranking, it is highly dependent on the application of proper politics by the universities and discrepancies due to poor practices in matters of Web presence are not uncommon.

Results Comparison
It is difficult to perform an in-depth analysis of all these rankings without using sophisticated statistical techniques or political-economic considerations (Aguillo et al., 2010;Dill & Soo, 2005). However, by choosing a wide comparison level, in this case the countries represented among the 500 best institutions according to the different classifications, it is possible to illustrate the differences among them and at the same time offer a candidate for a global vision of the worldwide university scenario. Table 1 shows the data corresponding to the last available editions (2011, except for the Leiden Ranking, for which the 2010 data were used) with the following specifications: the selected version of the Leiden Ranking (CWTS) has been the so-called "brute force" (total publications multiplied by the crown indicator). The Scimago data have been combined in a proportion of 1:1 (50%:50%) between publications and citations, after being standardized. For the QS Ranking, only the 498 universities appearing on their Website were used. The THE ranking has not been included in the analysis. For comparability purposes, the three bibliometric rankings are shown consecutively.
At the country level, we can study the relative coverage. A total of 59 countries (including Hong King and Taiwan) appear on the combined list. QS is the ranking with the widest coverage (50), while Taiwan (HEEACT), which only includes 39, is the ranking with the least diversity. Thirteen countries appear in only one of the rankings, mostly presented by QS (8), while Webometrics has two and Leiden, Scimago and Shanghai one each. From these data, it can be deduced that QS is the ranking that deviates the most from the average behavior, although that possibly has a fundamental influence on the last positions of the countries included.  In order to study the behavior of QS in greater depth, it is necessary to rely on the individual countries, especially those occupying the first positions. The ranking that differs the most from the median for 24 countries where this value is greater than two has been identified. QS is the ranking that deviates the most in 11 cases (including the U.S.A., United Kingdom, Australia, Ireland, India or New Zealand, all countries where English is the official or principal language), followed by Webometrics, whose 7 countries include, precisely, examples where English is not the main language (Japan, France, Spain, South Korea or Brazil). Bibliometric rankings differ, as it can be expected, quite less and the overrepresentation of China in Scimago or Turkey in Leiden are noteworthy, as is the poor presence of Germany in the Shanghai Rankings.
Some explanations are possible. The bias of the QS towards universities in countries with English as the official or principal language is perhaps the consequence of the uneven coverage of the survey. As for Webometrics, the discrimination of English is penalized by global audiences of the Web. Scopus, the database used by Scimago, may have a wider coverage of the Chinese periodical publications, while Leiden has been able to perform a better job identifying Turkish contributions. Finally, the unjustifiable exclusion of the universities of Berlin in Shanghai reflects on the cited result. Cybermetrics and the Web Ranking The Web Ranking of Universities (Aguillo, Ortega & Fernandez, 2008) has been published since 2004, and is directly inspired by the compound indicator model of the Shanghai Ranking, but using cybernetic data taken from the university websites. Cybermetrics or Webometrics is an emerging discipline developed in the mid-90s that aims to describe in a quantitative manner the scientific communication processes and the structure of academic and research units from the current information or the information exchanged via the Internet (Aguillo et al., 2006). The Web is explored using web crawlers. The data can be obtained directly or through commercial search engines that, today, are the most powerful and efficient tools to describe global scenarios.
The Web Ranking uses one of the most complete and updated university catalogs on the Web, as it studies more than 20,000 institutions of higher education from around the world. Figure 1 shows the Ranking's relative coverage by region. The Web Ranking uses a compound indicator model that keeps a 1:1 proportion between measures of activity or Web presence and those of visibility or Web impact (Aguillo, 2009). This means that each one of them contributes 50% to the weight of the indicator total. For practical purposes, four variables, defined as follows, are used.
 Size. Number of Web pages of a university domain, obtained from search engines.  Rich files. Number of rich format documents (pdf, doc, ppt, ps) related to the activity of teaching, research or administration. Also obtained through search engines.  Scholar. Number of research articles or papers included in the Google Scholar and Scimago databases.
 Visibility. Number of external links and mentions to the university Web domain.
The first three values are combined almost in equal proportion (20%, 15% and 15% plus a round up to improve legibility) to complete the 50% corresponding to the activity, while the other 50% corresponds to the virtual referendum that the visibility measure implies.
Among the advantages of this method, we highlight the possibility of including all the missions of the university, provided that they are reflected on the Web. However, the simplicity of the rankings system makes it difficult to know with precision the contribution or relative importance of each of the missions; hence, an integrated perspective, in which additional works are necessary to identify strengths and weaknesses, is always offered.
Among the main inconveniences of the Web Ranking, we point out their great vulnerability to poor practices in the design of Web presence strategies. Several hundred universities have more than one central Web domain, which divides the impact of their presence, apart from being a confusing and unpractical method. In other cases, the domains are changed with no contingency plan to keep the visibility of the original and, frequently, with no apparent or logical reason for those changes. Some universities make their scientific archives available to other organizations for study under a domain other than that of the university. Design problems, poorly organized content, a highly centralized structure, and lack of international information (in English) are some of the other causes for having lower positions in the Ranking.
The results obtained closely correlate with those obtained through other rankings, especially if the mentioned discrepancies are excluded. However, there is a characteristic pattern that is only revealed by the Web Ranking: the existence of an academic digital gap between North American (United States and Canada) universities and their European counterparts. As shown in Table 2, there are many North American institutions among the top 100 or 200, even twice or three times as much as from the rest of the world. By considering the top 500, those inequalities disappear, which indicates that it is the leading institutions that, apart from a strong commitment with Web publishing, create a network that includes universities highly active in research.

Conclusions
In summary, we may recommend that global rankings should not be ignored, since they fulfill an important mission and are an excellent tool in the design of strategies and university policies. A global critique is absurd, but the Rankings whose methodologies are not very academic, those with unjustified biases and those with commercial interests must be identified.
The Web is the most important scientific communication tool and, therefore, Web presence is an indicator that reflects the university's overall performance, not only its online activities. The Web significantly increases the audience, the visibility and the impact of the activities and achievements of a university's professors, researchers and students.