Are University Rankings Statistically Significant? A Comparison among Chinese Universities and with the USA

doi:10.2478/jdis-2021-0014

Abstract

Abstract:

Purpose: Building on Leydesdorff, Bornmann, and Mingers (2019), we elaborate the differences between Tsinghua and Zhejiang University as an empirical example. We address the question of whether differences are statistically significant in the rankings of Chinese universities. We propose methods for measuring statistical significance among different universities within or among countries.
Design/methodology/approach: Based on z-testing and overlapping confidence intervals, and using data about 205 Chinese universities included in the Leiden Rankings 2020, we argue that three main groups of Chinese research universities can be distinguished (low, middle, and high).
Findings: When the sample of 205 Chinese universities is merged with the 197 US universities included in Leiden Rankings 2020, the results similarly indicate three main groups: low, middle, and high. Using this data (Leiden Rankings and Web of Science), the z-scores of the Chinese universities are significantly below those of the US universities albeit with some overlap.
Research limitations: We show empirically that differences in ranking may be due to changes in the data, the models, or the modeling effects on the data. The scientometric groupings are not always stable when we use different methods.
Practical implications: Differences among universities can be tested for their statistical significance. The statistics relativize the values of decimals in the rankings. One can operate with a scheme of low/middle/high in policy debates and leave the more fine-grained rankings of individual universities to operational management and local settings.
Originality/value: In the discussion about the rankings of universities, the question of whether differences are statistically significant, has, in our opinion, insufficiently been addressed in research evaluations.

Key words: Ranking, University, China, Significance of differences, Indicators

Leydesdorff, Loet, Caroline S. Wagner, and Lin Zhang. "Are University Rankings Statistically Significant? A Comparison among Chinese Universities and with the USA." Journal of Data and Information Science, vol.6, no.2, 2021, pp.67-95. DOI: 10.2478/jdis-2021-0014

Figures/Tables 21

Table 1

Table 2a

Table 2b

Table 3

Table 4

Figure 1.

Figure 2.

Table 5

Figure 3.

Table 6

Top-20 universities in each of the three clusters. (See for a full list in the Annexes.)

	top group (top 20)	z	middle group (top 20)	z	bottom group (top 20)	z
1	Tsinghua Univ.	11.005	Zhejiang Univ.	3.800	Hubei Univ.	-0.498
2	Hunan Univ.	10.193	Harbin Institute of Technology	3.226	Northwest Univ.	-1.609
3	Hong Kong Univ. of Science and Technology	6.566	Huazhong Univ. of Science and Technology	2.907	South China Agricultural Univ.	-1.631
4	Univ. of Science and Technology of China	6.482	Peking Univ.	2.689	Zhejiang Univ. of Technology	-1.676
5	City Univ. of Hong Kong	6.454	Nanjing Univ.	1.973	China Univ. of Mining and Technology	-1.678
6	Shandong Univ. of Science and Technology	6.444	Xiamen Univ.	1.954	Nanjing Univ. of Aeronautics and Astronautics	-1.816
7	Hong Kong Polytechnic Univ.	6.406	Wuhan Univ.	1.765	Second Military Medical Univ.	-1.844
8	South China Univ. of Technology	6.049	Tianjin Univ.	1.738	Zhejiang Sci-Tech Univ.	-1.886
9	Chinese Univ. of Hong Kong	5.993	Dalian Univ. of Technology	1.553	Taiyuan Univ. of Technology	-1.934
10	Nankai Univ.	4.860	East China Normal Univ.	1.333	Shanghai Univ. of Traditional Chinese Medicine	-1.950
11	Univ. of Hong Kong	4.418	Soochow Univ.	1.310	Qingdao Univ. of Science and Technology	-1.961
12	Shenzhen Univ.	4.089	Univ. of Electronic Science and Technology of China	1.154	Nanjing Univ. of Chinese Medicine	-1.981
13	Qufu Normal Univ.	3.724	Nanjing Univ. of Science and Technology	1.140	Harbin Engineering Univ.	-2.033
14	Fuzhou Univ.	3.487	Southwest Jiaotong Univ.	1.088	Lanzhou Univ.	-2.110
15	Beihang Univ.	3.432	China Univ. of Geosciences	1.035	Northeast Normal Univ.	-2.128
16	Univ. of the Chinese Academy of Sciences	3.354	Huazhong Agricultural Univ.	1.030	China Jiliang Univ.	-2.130
17	Central China Normal Univ.	3.090	Tongji Univ.	0.954	Jiangnan Univ.	-2.163
18	Univ. of Macau	3.046	Univ. of Science and Technology Beijing	0.932	Jinan Univ.	-2.227
19	Northwestern Polytechnical Univ.	2.915	China Agricultural Univ.	0.704	Hangzhou Normal Univ.	-2.245
20	Wuhan Univ. of Technology	2.914	Beijing Institute of Technology	0.587	Southern Medical Univ.	-2.263

Table 6

Figure 4.

Figure 5.

Figure 6.

Table 7

Table 8

Figure 7.

Figure 8.

Table 9

Table 10

Middle group of 69 Chinese universities.

University	z	overall rank	within-group
Zhejiang University	3.800	13	1
Harbin Institute of Technology	3.226	18	2
Huazhong University of Science and Technology	2.907	23	3
Peking University	2.689	24	4
Nanjing University	1.973	29	5
Xiamen University	1.954	30	6
Wuhan University	1.765	31	7
Tianjin University	1.738	32	8
Dalian University of Technology	1.553	33	9
East China Normal University	1.333	35	10
Soochow University	1.310	37	11
University of Electronic Science and Technology of China	1.154	40	12
Nanjing University of Science and Technology	1.140	41	13
Southwest Jiaotong University	1.088	43	14
China University of Geosciences	1.035	44	15
Huazhong Agricultural University	1.030	45	16
Tongji University	0.954	46	17
University of Science and Technology Beijing	0.932	47	18
China Agricultural University	0.704	48	19
Beijing Institute of Technology	0.587	49	20
Sun Yat-sen University	0.572	50	21
North China Electric Power University	0.414	52	22
China University of Petroleum Beijing	0.389	53	23
Guangdong University of Technology	0.332	54	24
Shanghai University	0.332	55	25
Fourth Military Medical University	0.257	56	26
Beijing Normal University	0.178	57	27
Chongqing University	0.000	58	28
Hefei University of Technology	0.000	59	29
Xi’an Jiaotong University	0.000	60	30
Nanjing University of Posts and Telecommunications	-0.110	61	31
Nanjing Agricultural University	-0.171	63	32
China Pharmaceutical University	-0.259	64	33
Central South University	-0.280	65	34
China University of Petroleum East China	-0.407	66	35
Beijing Jiaotong University	-0.448	68	36
Xidian University	-0.529	71	37
Dalian Medical University	-0.643	72	38
Chongqing University of Posts and Telecommunications	-0.656	73	39
Huaqiao University	-0.795	74	40
Anhui Agricultural University	-0.862	75	41
Wuhan University of Science and Technology	-0.899	76	42
Nanjing Tech University	-0.919	77	43
Donghua University	-0.923	78	44
East China University of Science and Technology	-0.943	79	45
Army Medical University	-0.952	80	46
Anhui University	-1.004	81	47
Tianjin University of Technology	-1.047	82	48
Renmin University of China	-1.064	83	49
Nanjing Normal University	-1.189	84	50
Southeast University	-1.213	85	51
Fujian Agriculture and Forestry University	-1.218	86	52
Northwest Agriculture and Forestry University	-1.226	87	53
University of Shanghai for Science and Technology	-1.232	88	54
North China University of Science and Technology	-1.249	89	55
Wenzhou University	-1.306	90	56
Southwest Petroleum University	-1.311	91	57
Shaanxi University of Science and Technology	-1.314	92	58
China Three Gorges University	-1.333	93	59
University of South China	-1.345	94	60
Shanghai Normal University	-1.369	95	61
Dalian Maritime University	-1.380	96	62
Yangzhou University	-1.392	97	63
Changzhou University	-1.544	98	64
Nanjing Forestry University	-1.581	99	65
Guangzhou University of Chinese Medicine	-1.685	104	66
Fujian Normal University	-1.781	105	67
China Academy of Chinese Medical Sciences	-1.840	107	68
Shanghai Jiao Tong University	-1.876	109	69

Table 10

Table 11

Bottom group of 102 Chinese universities.

University	z	rank	within-group
Hubei University	-0.498	70	1
Northwest University	-1.609	100	2
South China Agricultural University	-1.631	101	3
Zhejiang University of Technology	-1.676	102	4
China University of Mining and Technology	-1.678	103	5
Nanjing University of Aeronautics and Astronautics	-1.816	106	6
Second Military Medical University	-1.844	108	7
Zhejiang Sci-Tech University	-1.886	110	8
Taiyuan University of Technology	-1.934	111	9
Shanghai University of Traditional Chinese Medicine	-1.950	112	10
Qingdao University of Science and Technology	-1.961	113	11
Nanjing University of Chinese Medicine	-1.981	114	12
Harbin Engineering University	-2.033	115	13
Lanzhou University	-2.110	117	14
Northeast Normal University	-2.128	118	15
China Jiliang University	-2.130	119	16
Jiangnan University	-2.163	120	17
Jinan University	-2.227	121	18
Hangzhou Normal University	-2.245	122	19
Southern Medical University	-2.263	123	20
Hunan Normal University	-2.282	124	21
Henan University	-2.328	125	22
Beijing University of Chinese Medicine	-2.346	126	23
Tianjin Medical University	-2.348	127	24
Shantou University	-2.383	128	25
Xi’an University of Architecture and Technology	-2.384	129	26
Hainan University	-2.417	130	27
Hohai University	-2.453	131	28
Beijing University of Posts and Telecommunications	-2.458	132	29
Qingdao University	-2.465	133	30
Jiangsu University	-2.467	134	31
Jiangxi Normal University	-2.486	135	32
Shaanxi Normal University	-2.505	136	33
Shandong University of Technology	-2.506	137	34
Nanchang University	-2.528	138	35
Henan Agricultural University	-2.584	139	36
Guizhou University	-2.675	140	37
Guangxi Normal University	-2.709	141	38
Northeast Forestry University	-2.751	142	39
Northeastern University	-2.761	143	40
Henan Normal University	-2.775	144	41
Shandong Agricultural University	-2.783	145	42
Anhui Normal University	-2.834	146	43
Lanzhou University of Technology	-2.835	147	44
Ningbo University	-2.838	148	45
Southwest University	-2.855	149	46
Yunnan University	-2.940	150	47
Shanghai Ocean University	-2.954	151	48
Northwest Normal University	-2.962	152	49
Inner Mongolia University	-2.968	153	50
Beijing University of Technology	-2.969	154	51
Fudan University	-3.065	155	52
Yanshan University	-3.100	156	53
Tianjin University of Science and Technology	-3.150	157	54
Beijing Forestry University	-3.151	158	55
People’s Liberation Army University of Science and Technology	-3.152	159	56
Capital Normal University	-3.188	160	57
Southwest University of Science and Technology	-3.433	161	58
Hebei University of Technology	-3.558	162	59
Shenyang Pharmaceutical University	-3.713	163	60
Xinjiang University	-3.741	164	61
Kunming University of Science and Technology	-3.832	165	62
Nantong University	-3.935	166	63
Xiangtan University	-4.012	167	64
Northeast Agricultural University	-4.025	168	65
Wenzhou Medical University	-4.062	169	66
Chongqing Medical University	-4.106	170	67
Guangxi University	-4.119	171	68
Xi’An University of Technology	-4.203	172	69
Zhengzhou University	-4.244	173	70
Ocean University of China	-4.246	174	71
Henan Polytechnic University	-4.348	175	72
Shihezi University	-4.400	176	73
Kunming Medical University	-4.506	177	74
Xuzhou Medical College	-4.604	178	75
Tianjin Polytechnic University	-4.775	179	76
South China Normal University	-4.827	180	77
Nanjing Medical University	-4.941	181	78
Chengdu University of Technology	-4.958	182	79
Xinxiang Medical University	-4.962	183	80
Harbin University of Science and Technology	-5.001	184	81
North University of China	-5.068	185	82
Yangtze University	-5.093	186	83
Guangzhou Medical University	-5.200	187	84
Sichuan Agricultural University	-5.212	188	85
Sichuan University	-5.247	189	86
National University of Defense Technology	-5.257	190	87
Harbin Medical University	-5.331	191	88
Tianjin Normal University	-5.358	192	89
Shanxi Medical University	-5.378	193	90
Henan University of Science and Technology	-5.416	194	91
Shandong University	-5.619	195	92
Anhui Medical University	-5.951	196	93
Hebei University	-6.307	197	94
Guangxi Medical University	-7.267	198	95
Peking Union Medical College	-7.621	199	96
Xinjiang Medical University	-7.703	200	97
China Medical University	-8.064	201	98
Fujian Medical University	-8.169	202	99
Hebei Medical University	-8.966	203	100
Jilin University	-9.607	204	101
Capital Medical University	-12.944	205	102

Table 11

Table 12

75 top universities with overlapping confidence values.

University	Z	rank	within group
Tsinghua University	11.005	1	1
Hunan University	10.193	2	2
Hong Kong University of Science and Technology	6.566	3	3
University of Science and Technology of China	6.482	4	4
City University of Hong Kong	6.454	5	5
Shandong University of Science and Technology	6.444	6	6
Hong Kong Polytechnic University	6.406	7	7
South China University of Technology	6.049	8	8
Chinese University of Hong Kong	5.993	9	9
Nankai University	4.860	10	10
University of Hong Kong	4.418	11	11
Shenzhen University	4.089	12	12
Zhejiang University	3.800	13	13
Qufu Normal University	3.724	14	14
Fuzhou University	3.487	15	15
Beihang University	3.432	16	16
University of the Chinese Academy of Sciences	3.354	17	17
Harbin Institute of Technology	3.226	18	18
Central China Normal University	3.090	19	19
University of Macau	3.046	20	20
Northwestern Polytechnical University	2.915	21	21
Wuhan University of Technology	2.914	22	22
Huazhong University of Science and Technology	2.907	23	23
Peking University	2.689	24	24
Southern University of Science and Technology	2.332	25	25
Beijing University of Chemical Technology	2.273	26	26
GuangZhou University	2.120	27	27
Hong Kong Baptist University	2.094	28	28
Nanjing University	1.973	29	29
Xiamen University	1.954	30	30
Wuhan University	1.765	31	31
Tianjin University	1.738	32	32
Dalian University of Technology	1.553	33	33
Shandong Normal University	1.478	34	34
East China Normal University	1.333	35	35
Jiangsu Normal University	1.313	36	36
Soochow University	1.310	37	37
Zhejiang Normal University	1.238	38	38
University of Jinan	1.195	39	39
University of Electronic Science and Technology of China	1.154	40	40
Nanjing University of Science and Technology	1.140	41	41
Nanjing University of Information Science and Technology	1.093	42	42
Southwest Jiaotong University	1.088	43	43
China University of Geosciences	1.035	44	44
Huazhong Agricultural University	1.030	45	45
Tongji University	0.954	46	46
University of Science and Technology Beijing	0.932	47	47
China Agricultural University	0.704	48	48
Beijing Institute of Technology	0.587	49	49
Sun Yat-sen University	0.572	50	50
Hangzhou Dianzi University	0.449	51	51
North China Electric Power University	0.414	52	52
China University of Petroleum Beijing	0.389	53	53
Guangdong University of Technology	0.332	54	54
Shanghai University	0.332	55	55
Fourth Military Medical University	0.257	56	56
Beijing Normal University	0.178	57	57
Chongqing University	0.000	58	58
Hefei University of Technology	0.000	59	59
Xi’an Jiaotong University	0.000	60	60
Nanjing University of Posts and Telecommunications	-0.110	61	61
Heilongjiang University	-0.158	62	62
Nanjing Agricultural University	-0.171	63	63
China Pharmaceutical University	-0.259	64	64
Central South University	-0.280	65	65
China University of Petroleum East China	-0.407	66	66
Qingdao Agricultural University	-0.422	67	67
Beijing Jiaotong University	-0.448	68	68
Chang’an University	-0.465	69	69
Hubei University	-0.498	70	70
Dalian Medical University	-0.643	72	71
Chongqing University of Posts and Telecommunications	-0.656	73	72
Huaqiao University	-0.795	74	73
Anhui Agricultural University	-0.862	75	74
Army Medical University	-0.952	80	75

Table 12

References 33

[1]	Alsmadi, I., & Taylor, Z. (2018). Examining university ranking metrics: Articulating issues of size and web dependency. In Proceedings of the 2018 International Conference on Computing and Big Data (ICCBD '18). Association for Computing Machinery, New York, NY, USA, 73-77. doi: https://doi.org/10.1145/3277104.3277111
[2]	Anderson, J., Collins, P.M.D., Irvine, J., Isard, P.A., Martin, B.R., Narin, F., & Stevens, K. (1988). On-line approaches to measuring national scientific output: A cautionary tale. Science and Public Policy 15(3), 153-161.
[3]	Bornmann, L., & Leydesdorff, L. (2013). The validation of (advanced) bibliometric indicators through peer assessments: A comparative study using data from InCites and F1000. Journal of Informetrics, 7(2), 286-291.
[4]	Bornmann, L., & Mutz, R. (2011). Further steps towards an ideal method of measuring citation performance: The avoidance of citation (ratio) averages in field-normalization. Journal of Informetrics, 5(1), 228-230.
[5]	Bourdieu, P. (1998). The state nobility: Elite schools in the field of power. Stanford University Press.
[6]	Bowman, N.A., & Bastedo, M.N. (2011). Anchoring effects in world university rankings: Exploring biases in reputation scores. Higher Education, 61(4), 431-444.
[7]	Brewer, D.J., Gates, S.M., & Goldman, C.A. (2001). In Pursuit of Prestige: Strategy and Competition in U.S. Higher Education. Piscataway, NJ: Transaction Publishers, Rutgers University.
[8]	Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
[9]	Colliander, C., & Ahlgren, P. (2011). The effects and their stability of field normalization baseline on relative performance with respect to citation impact: A case study of 20 natural science departments. Journal of Informetrics, 5(1), 101-113.
[10]	Cumming, G. (2013). Understanding the New Statistics: Effect sizes, confidence intervals, and meta-analysis. New York, NY/ Hove, East Sussex, UK: Routledge.
[11]	DiMaggio, P.J., & Powell, W.W. (1983). The Iron Cage Revisited: Institutional isomorphism and collective rationality in organizational fields. American Sociological Review, 48, 147-160.
[12]	Garfield, E. (1971). The mystery of the transposed journal lists—wherein Bradford's Law of Scattering is generalized according to Garfield's Law of Concentration. Current Contents, 3(33), 5-6.
[13]	Gingras, Y. (2016). Bibliometrics and research evaluation: An overview. Cambridge MA: MIT Press.
[14]	Halffman, W., & Leydesdorff, L. (2010). Is Inequality Among Universities Increasing? Gini Coefficients and the Elusive Rise of Elite Universities. Minerva, 48(1), 55-72. pmid: 20401157
[15]	Heilbron, J., Boncourt, T., & Sorá, G. (2018). Introduction: The Social and Human Sciences in Global Power Relations. In The Social and Human Sciences in Global Power Relations (pp. 1-25). Pal. grave Macmillan, Cham.
[16]	Leydesdorff, L., & Wagner, C. (2009). Macro-level indicators of the relations between research funding and research output. Journal of Informetrics, 3(4), 353-362. doi: 10.1016/j.joi.2009.05.005
[17]	Leydesdorff, L., Bornmann, L., & Mingers, J. (2019). Statistical significance and effect sizes of differences among research universities at the level of nations and worldwide based on the leiden rankings. Journal of the Association for Information Science and Technology, 70(5), 509-525.
[18]	Leydesdorff, L., Bornmann, L., Mutz, R., & Opthof, T. (2011). Turning the Tables on Citation Analysis One More Time: Principles for Comparing Sets of Documents. Journal of the American Society for Information Science and Technology, 62(7), 1370-1381.
[19]	Leydesdorff, L., Wouters, P., & Bornmann, L. (2016). Professional and citizen bibliometrics: Complementarities and ambivalences in the development and use of indicators-a state-of-the-art report. Scientometrics, 109(3), 2129-2150. pmid: 27942086
[20]	McAllister, P.R., Narin, F., & Corrigan, J.G. (1983). Programmatic Evaluation and Comparison Based on Standardized Citation Scores. IEEE Transactions on Engineering Management, 30(4), 205-211.
[21]	Peters, M.A. (2017). Global university rankings: Metrics, performance, governance. Educational Philosophy & Theory, 51(1), 1-9.
[22]	Schneider, J. (2015). Null hypothesis significance tests. A mix-up of two different theories: The basis for widespread confusion and numerous misinterpretations. Scientometrics, 102(1), 411-432.
[23]	Sheskin, D.J. (2011). Handbook of Parametric and Nonparametric Statistical Procedures (5th Edition). Boca Raton, FL: Chapman & Hall/CRC.
[24]	Shin, J.C., Toutkoushian, R.K., & Teichler, U. (2011). University Rankings: Theoretical Basis, Methodology and Impacts on Global Higher Education. Dordrecht: Springer.
[25]	Sivertsen, G. (2016). Patterns of internationalization and criteria for research assessment in the social sciences and humanities. Scientometrics, 107(2), 357-368.
[26]	Sivertsen, G., Rousseau, R., & Zhang, L. (2019). Measuring scientific contributions with modified fractional counting. Journal of Informetrics, 13(2), 679-694.
[27]	Tijssen, R.J.W., Visser, M.S., & van Leeuwen, T.N. (2002). Benchmarking international scientific excellence: Are highly cited research papers an appropriate frame of reference? Scientometrics, 54(3), 381-397.
[28]	van Raan, A.F.J., van Leeuwen, T.N., Visser, M.S., van Eck, N.J., & Waltman, L. (2010). Rivals for the crown: Reply to Opthof and Leydesdorff. Journal of Informetrics, 4(3), 431-435.
[29]	Waltman, L. (2016). Conceptual difficulties in the use of statistical inference in citation analysis. Journal of Informetrics, 4(10), 1249-1252.
[30]	Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E., Tijssen, R.J., Eck, N.J., .. Wouters, P. (2012). The Leiden Ranking 2011/2012: Data collection, indicators, and interpretation. Journal of the American Society for Information Science and Technology, 63(12), 2419-2432.
[31]	Waltman, L., Van Eck, N.J., Van Leeuwen, T.N., Visser, M.S., & Van Raan, A.F.J. (2011). Towards a New Crown Indicator: Some Theoretical Considerations. Journal of Informetrics, 5(1), 37-47.
[32]	Wasserstein, R.L., & Lazar, N.A. (2016). The ASA's statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129-133.
[33]	Whitley, R.D. (1984). The Intellectual and Social Organization of the Sciences. Oxford: Oxford University Press.

Country (top 10 of 65)	N of universities
China	205
United States	198
United Kingdom	58
Germany	54
Japan	53
South Korea	44
Italy	41
Spain	41
India	36
Iran	36	%
sum	766	65.1%
<55 other countries>	= (1,176 - 766) = 410	34.9%
65 countries	1,176	100%

Observed values	top 10%	non-top	total p
Tsinghua	2,738	17,164	19,902
Zhejiang	2,604	20,906	23,510
	5,342	38,070	43,412

Expected values	top 10%	non-top	total p
Tsinghua	2,449.01	17,452.99	19,902.00
Zhejiang	2,892.99	20,617.01	23,510.00
	5,356.09	38,055.91	43,412.00

Chi-square	top 10%	non-top
Tsinghua	34.10	4.79	38.89
Zhejiang	28.87	4.05	32.92
		χ² =	71.80

	Top 10%	Non-top
Tsinghua	5.84	-2.19
Zhejiang	-5.37	2.01