World university rankings and their methodology

INTERNATIONAL university rankings are growing in popularity. Their publishing is anticipated. The results are argued. People use rankings in order to pursue their personal goals: select an educational institution, build a professional career, or even develop a national education strategy. Who creates the ranking lists of the best educational institutions in the world and what criteria are used? Is anything special required from an educational institution to get to the top? How are Russian educational institutions ranked in the world rankings?

In 1983 the U.S. News & World Report published the first ever university ranking. Thirty years later the university rankings are on the boom: there is at least a hundred different rankings published every year. The following global rankings are considered the most influential:

• Academic Ranking of World Universities (ARWU) also known as Shanghai Ranking;
• QS World University Rankings (QS WUR);
• Times Higher Education World University Ranking-Thomson Reuters (THE).

The Academic Ranking of World Universities (ARWU) was originally compiled and issued by Shanghai Jiao Tong University in 2003 in order to compare the level of education in Chinese educational institutions and the best educational institutions in the world. ARWU annually selects around 1,000 educational institutions out of 17,000 educational institutions in the world and ranges the first 500 in the league table (published on www.shanghairanking.com). Shanghai Ranking mainly focuses on academic research. Educational institutions are only eligible for this ranking provided their alumni or staff are Nobel laureates and Fields Medalists (“weighing” 30% of the total evaluation) or highly cited researchers (“weighing” 20% of the total evaluation). The remaining “weight” is distributed between other compulsory criteria: papers published in the Nature and Science, papers indexed in the Science Citation Index and the Social Science Citation Index as well as Per capita academic performance of an institution. In addition to the main table, there are additional lists compiled in a variety of broad fields and specific subjects. The information about the university activity is taken exclusively from the open sources. By the way, over the entire history of ARWU existence, its top ranking leader remains the same— for the past fourteen years the list has been headed by Harvard University.

Another global ranking—QS World University Rankings—was developed by the British consultancy company Quacquarelli Symonds (QS) in cooperation with the Times Higher Education magazine (THE) in 2004. In its first version the ranking was called the World University Rankings, however in 2010 The Times decided to modernize it and made a partnership with Thomson Reuters. Nowadays the Times Higher Education magazine publishes the World University Rankings, and Quacquarelli Symonds continues to produce rankings based on its own methodology, now called the QS World University Rankings.

Every year QS reviews around 2,500 educational institutions all over the world. In order to be included into the QS World University Rankings an educational institution should have a sufficient number of publications indexed in international bibliometric databases and offer bachelor’s, master’s and postgraduate programs in at least two out of five fields of knowledge: humanities, natural science, engineering, social studies, life sciences and medicine. Rating compilers take into consideration the quality of education and research achievements of the academic staff and alumni (indicators: academic peer review, citations per faculty, international staff ratio and student-to-faculty ratio), practical application of the obtained knowledge and skills mastered (indicators: employer reputation, international student ratio). The ranking includes statistical data on an educational institution, however, at least 50% of the total evaluation comes from the results of a survey of an academic community and employers. The poll includes the leading researchers in the field, rectors of over 500 educational institutions and representatives of several thousand companies from 90 countries. The subjective complexity of QS rating is the popularity and recognizability of an educational institution which makes a large share of the expert evaluation.

In addition to its World University Rankings, QS annually publishes ten other rankings and tables in the field of education (for comparison, The Times publishes six other rankings). Thus, the authors select top higher education institutions in BRICS countries, the best universities in Asia, the Arab Region, Latin America, rank the best student cities, and award QS stars like in the tourism industry. The rankings of the leading educational institutions according to QS are available from the project website www.topuniversities.com.

The youngest of the global “Big Three” is the Times Higher Education World University Ranking (ТHE). In different time periods this ranking included different number of higher education institutions: in 2015 the total ranking included 400 educational institutions while in 2016—over 900. THE doesn’t review the universities involved in specific fields of research and publishing below 200 research papers per year. On average the methodology contains 13 different indicators grouped under 5 categories: teaching, research, citations, international mix, industry income. All the indicators have different “weighing,” citations being the “heaviest” (worth 32.5%). The level of  institution is derived from the expert survey; however, the subjective data are worth much less than in the QS Ranking (academic reputation—15%, research reputation in certain areas— 19.5%). Depending on their reputation The Times ranges the educational institutions in a separate ranking—the World Reputation Rankings employing an academic opinion survey only. Besides, the Times Higher Education annually published the ranking of universities which are under 50 years old, as well as subject specific and regional rankings: Asia and BRICS & Emerging Economies.

The top 200 universities and specialized rankings are available from www.timeshighereducation.com.

All the tree classic global ranking methodologies are of integral nature, i.e. they are evaluated based on a variety of criteria and their “weighing,” and every educational institution obtains an overall result expressed in nominal units. However, there are other options for leader definition. Thus, the U-multirank ranking developed by the Center for Higher Education (CHE) in Germany and the Center for Higher Education Policy Studies (CHEPS) in the Netherlands do not bring data to a single indicator. On the project website www.umultirank.org four separate ratings are available: research, teaching and learning, international orientation, knowledge transfer and regional engagement. U-multirank rates them by grouping into 5 broad categories, from A to E, where A stands for very good, while E stands for weak. Thus, the ranking developers let the users define which aspect of a university activity is more important.

There is also a methodology for evaluating the national system of higher education in general, rather than indicators demonstrated by certain universities. This may be referred to the U21 Rankings of National Higher Education Systems developed by the methodology of Melbourne Institute of Applied Economic and Social Research, University of Melbourne, Australia. U21 evaluates the national systems of higher education based on 24 measures grouped into 4 categories: resources (including state and private investments “weighing” 25% of the overall score); output (research, publications, meeting the requirements of the national labor market, “weighing” 40%); connectivity (the level of international cooperation, “weighing” 10%); and environment (state politics and regulations, possibilities for getting education, “weighing” 25%). The ranking includes 50 countries. In 2016 using this approach, the top countries with the best systems of education are the following: the United States, Switzerland, Denmark, the United Kingdom and Sweden. Indonesia scored last. Russia was ranked 34th.

About ten-fifteen years ago the Russian higher education institutions were seldom represented in the world rankings. Even so, they were included into the ranking list by default, initiated by the compilers. For example, Lomonosov Moscow State University was included into the THE-QS ranking due to the references of foreign experts. However, many national higher education institutions were kept beyond these rankings. This fact may be explained by the following reasons: orientation of global rankings to Anglo-Saxon higher school (it is not by chance that 40% of higher education institutions represented in the rankings are from the English speaking countries), and certain self-sufficiency of Russian higher education institutions.

According to Alevtina Chernikova, Rector of the National University of Science and Technology, the Russian system of education is one of the best educational systems in the world and our researchers, particularly in exact sciences and engineering are second to none in expertise as compared to their colleagues abroad. It is enough to mention, that our physicists, engineers, programmers are well-demanded abroad. Thus, for some reason the Russian science and education were not integrated into the world academic community for a long period of time. In the long run we remained shrank onto ourselves, our scientists were reluctant to publish the results of their research work in foreign journals, national higher education institutions had no incentives to go beyond the borders of the former Soviet Union.

Yet, this situation is no longer true. Which was stated on the highest level. In 2012 President of the Russian Federation Vladimir Putin issued an Order, to enable at least 5 Russian higher education institutions to be ranked among the first hundred world’s best universities by 2020. This goal presumed grasping “the rules of the game” along with enormous work on modernization of the national higher education institutions, intensifying their research and academic capacity. All this was to be done within a very short period of time.

In order to fulfill the President’s Order Project 5-100 was launched within which the state support was provided to higher education institutions. The project is supposed to result in a group of contemporary Russian leading universities with effective management structure and international academic reputation. These universities are meant to represent Russia on the global level and compete for the leading positions in the global rankings. State-funded higher education institutions were eligible to participate in the national contest for state support, excluding Lomonosov Moscow State University and Saint Petersburg State University due to their special status and financing. There are currently 21 educational institutions participating in this program. A specially appointed council on enhancing the competitiveness of the leading national higher education institutions reviews their “road maps,” evaluates achievements (including the places in global rankings) and makes recommendations to the Federal Ministry of Education and Science whether to subsidize them.

The state granted funds may be used for the following purposes. First and foremost, it is the development of international cooperation: joint educational projects with foreign educational institutions, internships, student and staff mobility programs, new world famous scientists. In general the project presumes that by 2020 the international student share in every higher education institution participating in Project 5-100 is expected to be at least 15% and the share of foreign faculty is supposed to be at least 10%. Besides, the above mentioned state support enables a higher education institution to promote research projects, construct academic buildings, promote new formats of student education (online lectures and courses).

In 2017 the participating institutions will receive 10.3 billion rubles (in 2018 the prospective amount of state support is 10.2 billion rubles, in 2019—10 billion rubles and in 2020—14.5 billion rubles). Beginning from this year the model of payment distribution changes. Before this year the money could be spent on the priority development fields, but now the financial flow is divided into two parts: systematic transformational efforts and strategic academic units which are bound to promote the educational institution to the leadership positions.

It is presumed that the strategic academic units would generate research competencies as well as result in specific product highly demanded by the society. Upon the project completion these strategic academic units should boost the further development of educational institutions.

The past 2016 was the midyear for Project 5-100. Remarkably, according to experts’ opinion this year was the most successful in terms of achievements of the participating institutions since the beginning of the program. “The Russian educational institutions stopped working in isolation from the rest of the world,” as was announced by the international experts. The results of ARWU, THE and QS rankings revealed a significant increase of the Russian universities in the lists. Thus, the conservative Shanghai Ranking included three Russian universities in 2016. Novosibirsk State University was included in the 401-500 group for the first time. Saint Petersburg State University is in the 301-400 group, just like last year, Moscow State University was ranked 87th (down from 86th in 2015).
The global QS Ranking features 22 Russian educational institutions, 18 of which improved their position as compared with the previous year. The highest position is held by Lomonosov Moscow State University. It is in the 108th place, just like in the previous year. The second position is held by Saint Petersburg State University: in comparison with the previous year it went down to the 258th place. Novosibirsk State University is 26 positions up, resulting in the 291st place and the third result in the list.

The top ten national universities included Bauman Moscow State Technical University, Moscow Institute of Physics and Technology (MIPT), Moscow State University of Foreign Affairs (MGIMO), Tomsk State University, Tomsk Polytechnic University, Moscow Engineering Physics Institute (MEPhI) and Higher School of Economics (HSE). Four of these higher education institutions moved up over 100 positions. Thus, Tomsk State University jumped over 104 places up from the 481-490 group to the 377th place. Moscow Engineering Physics Institute jumped 100 up from 501-550 to 401-410. Higher School of Economics improved its position 90 up from 550-501 to 411-420.

Nearly 77% of higher education institutions improved their indicators in Proportion of international students; 45% of the Russian higher education institutions enhanced their position in Proportion of international faculty, as it was mentioned by Zoya Zaitseva, Regional Director for Eastern Europe and Central Asia, QS.

Russian higher education institutions demonstrated exceptional results in the Times Higher Education Ranking: by 2016 the number of presented Russian universities almost doubled. Eleven higher education institutions joined the thirteen ones included into the ranking a year before. Moscow State University has become the best national university ranked 188th, 27 positions down. The second best result is held by Moscow Institute of Physics and Technology (MIPT) ranked 301-350. Saint Petersburg National Research University of Information Technology, Mechanics and Optics (ITMO) is ranked 351-400, which makes it hold the third position on the national level. Tomsk State University moved 100 places up and was included into the 501-600 group, unlike Tomsk Polytechnic University which is ranked in the same group, but moved 250 positions down in comparison with the previous year.

Saint Petersburg National Research University of Information Technology, Mechanics and Optics (ITMO) (ranked 351-400) and Higher School of Economics (ranked 401-500) were included into the ranking for the first time. The following higher education institutions were also included into the ranking for the first time: Lobachevsky University, Peoples’ Friendship University of Russia, I.M. Sechenov First Moscow State Medical University (MSMU), Siberian Federal University, Samara National Research University. All of them were ranked in 801+. This group also includes the Ural Federal University and the National University of Science and Technology (MISiS) which were originally ranked 601-800.
According to Petr Glybochko, Rector of the National University of Science and Technology, the result was achieved by means of enhancing the academic research and intensifying publication activity by 80% for the past five years.

The Russian higher education institutions move even faster in subject-specific and regional rankings as compared to the global ones. Thus, seventeen Russian higher education institutions were ranked in the QS World University Rankings by Subject 2016, including eight ranked in the top 100 (for reference, only ten Russian higher education institutions were included into the ranking in 2015). The best result was demonstrated by Lomonosov Moscow State University which was ranked 17th in Linguistics (for reference, in 2015 it was down in the 35th place). Besides, MSU was ranked 27th in Physics and Astronomy and 33rd in Mathematics. The University was ranked in the top 100 in nine more subjects (including Chemistry, Engineering, Philosophy, Politics, Archeology). Saint Petersburg State University also strengthened its positions. It moved up to the 26th place in Extraction of Minerals Research and got in the top 100 in Linguistics. Moscow Engineering Physics Institute (MEPhI) and Lobachevsky State University of Nizhny Novgorod were ranked in the top 100 in Physics and Astronomy.

Russia gained substantial advantage due to inclusion of six additional subjects in 2016. As a result, Tchaikovsky Moscow State Conservatory ranked 27th and Rimsky-Korsakov Saint Petersburg Conservatory ranked 47th were included in the top 100 in Performing Arts. Two more higher education institutions made their debut appearance in Engineering—Mineral & Mining: the National Mineral Resources University and Gubkin Russian State University of Oil and Gas which were both ranked 51-100.

In accordance with the Times Higher Education Ranking by Subject, four Russian universities are included into the top 100 best world’s universities: Lomonosov Moscow State University ranked 43rd in Computer Science and 66th in Art and Humanities; Moscow Institute of Physics and Technology (MIPT) was ranked 78th in Physical Sciences; Higher School of Economics was ranked 83rd in Business and Economics; Saint Petersburg National Research University of Information Technology, Mechanics and Optics (ITMO) made a breakthrough by taking the 56th place in the Computer Science table. Alexander Bukhanovsky, Director of the School of Translational Information Technologies, comments on the success of his university, “We shoot upwards due to the synergy of a range of factors. It is a brilliant school of information technology and programming which dates back to 1930s. It is the university strategy of research diversification and its transmission to socially important areas such as medicine, civil construction, safety, etc. It actively collaborates with the leading national and international companies on Research and Advanced Development. And of course, it is the support from Project 5-100, which in many ways accelerated the driving force of the first three positions.”

As for regional rankings, QS BRICS ranked 20 higher education institutions in the top 100, thus, 13 places were taken by higher education institutions— participants in Project 5-100. Nine Russian higher education institutions were included in the top 100 of the THE’s BRICS & Emerging Economies Rankings in 2016 (for reference, in 2015 there were four higher education institutions in the list, and in 2014 only two—Lomonosov Moscow State University and Saint Petersburg State University).

The position of Russian higher education institutions in the Times Higher Education World Reputation Rankings also present some interest. In 2015 our country was represented by Lomonosov Moscow State University (ranked 25th) and Saint Petersburg State University (ranked 71-80) only. At the beginning of 2016 Peter the Great Saint Petersburg Polytechnic University, Moscow Institute of Physics and Technology (MIPT), Tomsk Polytechnic University and Kazan Federal University joined the second hundred. Our country enhanced its position in the QS Best Student Cities ranking. According to research published by QS in February 2017 Moscow (ranked 39th), Saint Petersburg (ranked 78th), Tomsk (ranked 91st) and Novosibirsk (ranked 93rd) were included in the top 100 best student cities. Twelve higher education institutions, participants in Project 5-100, are located in these cities. Apropos, only Moscow was listed in this ranking in 2016.

One of the criteria for evaluation of academic activity of higher education institutions in Russia is their upward movement in the global ranking systems. However, there is every likelihood that a new national ranking system is going to be acknowledged on the global scale. The new Three Missions of Universities ranking of international universities established by the Russian Union of Rectors and the Russian Academy of Sciences was announced at the end of 2016.

“Many countries have learned from our educational system. We were an example even for the developed countries, which are now leaders in various rankings. Our education system deserves a deeper look, to be understood and used for our development,” Viktor Sadovnichy, Rector of Lomonosov Moscow State University, said at a press conference for the launch of the ranking.

Three classic missions of universities are connected with science, education and society. A new ranking system incorporates them all: it will take into consideration the achievements and quality of education, evaluate the impact of an educational institution on culture, define its value for people. The ranking includes 34 criteria joined in 5 groups: Education (“weighing” 40% in the overall score), Science (“weighing” 30% in the overall score), Internationalization, Sustainability and Development Capacity, Distance Learning (“weighing” in the overall score 10% each).

“We wanted to have a look at the universities through the ’camera with very high resolution,’ which makes a good and attractive university profile. All the aspects of a university activity have to be reviewed,” summarized V. Sadovnichy, Rector of Lomonosov Moscow State University.

All scientific data bases including the Russian Science Citation Index will be taken into consideration when compiling a ranking.
“We understand that the global research community speaks English and our scientists are actively publishing results of their research in the leading research journals,” as the new ranking system was commented by Irina Karelina, Director of Strategic Planning of the Higher School of Economics, “This makes our universities visible in the academic community, and the reputation of our higher education institutions is on the increase. Thus, the national ranking may show the specific character of Russian education in a more understandable and clear cut way, therefore, it is worthwhile taking part in Russian rankings with a well-developed methodology.”

The Three Missions of Universities Ranking will embrace three levels: Russian, regional (for instance, BRICS countries), and international. Dozens of countries expressed their willingness to participate in the ranking, including China, India, Iran, Turkey, Armenia, Azerbaijan. Higher education institutions of Japan, Great Britain, France and the USA are also invited to participate. The first Three Missions of Universities ranking of world universities will be introduced in the fall of 2017, the Russian part will be ready by March-April, 2017.
Taking the leading positions in the world ranking systems is a complicated process with a lot of ins and outs. On the one hand, the methodology changes and is improved every year, new trustworthy rankings emerge aspiring to more objective results. On the other hand, the universities compete for the top positions and it is extremely difficult for the newcomers to join the elite of the world education.

According to Duncan Ross, Data and Analytics Director at the Times Higher Education, the universities taking the leading positions in the ranking are contributing even more efforts in order to become better. Therefore, it is not easy for the Russian universities to move up in the ranking. It is not because they do not improve, it is because it concerns everyone. However, ranking is not a goal in itself, it is a tool to understand the position of the university, the dynamics for its development and how far it is from its leadership position.