Practical operation and theoretical basis of difference-in-difference regression in science of science: The comparative trial on the scientific performance of Nobel laureates versus their coauthors

doi:10.2478/jdis-2023-0003

Abstract

Abstract:

Purpose: In recent decades, with the availability of large-scale scientific corpus datasets, difference-in-difference (DID) is increasingly used in the science of science and bibliometrics studies. DID method outputs the unbiased estimation on condition that several hypotheses hold, especially the common trend assumption. In this paper, we gave a systematic demonstration of DID in the science of science, and the potential ways to improve the accuracy of DID method.

Design/methodology/approach: At first, we reviewed the statistical assumptions, the model specification, and the application procedures of DID method. Second, to improve the necessary assumptions before conducting DID regression and the accuracy of estimation, we introduced some matching techniques serving as the pre-selecting step for DID design by matching control individuals who are equivalent to those treated ones on observational variables before the intervention. Lastly, we performed a case study to estimate the effects of prizewinning on the scientific performance of Nobel laureates, by comparing the yearly citation impact after the prizewinning year between Nobel laureates and their prizewinning-work coauthors.

Findings: We introduced the procedures to conduct a DID estimation and demonstrated the effectiveness to use matching method to improve the results. As a case study, we found that there are no significant increases in citations for Nobel laureates compared to their prizewinning coauthors.

Research limitations: This study ignored the rigorous mathematical deduction parts of DID, while focused on the practical parts.

Practical implications: This work gives experimental practice and potential guidelines to use DID method in science of science and bibliometrics studies.

Originality/value: This study gains insights into the usage of econometric tools in science of science.

Key words: Science of Science, Bibliometrics, Difference-in-Difference, CEM, PSM, Nobel Prize

Yurui Huang , Chaolin Tian , and Yifang Ma .(2023). Practical operation and theoretical basis of difference-in-difference regression in science of science: The comparative trial on the scientific performance of Nobel laureates versus their coauthors. Journal of Data and Information Science, 8(1), 29-46. DOI: 10.2478/jdis-2023-0003

0
/ / Recommend

Add to citation manager EndNote|Ris|BibTeX

URL: http://manu47.magtech.com.cn/Jwk3_jdis/EN/10.2478/jdis-2023-0003

http://manu47.magtech.com.cn/Jwk3_jdis/EN/Y2023/V8/I1/29

This work is licensed under the Creative Commons Attribution 4.0 International License.

Figures/Tables 5

Figure 1.

Table 1.

Table 2.

Table 3.

Table 4.

References 45

[1]	Abadie A., & Imbens G. W. (2006). Large sample properties of matching estimators for average treatment effects. Econometrica, 74(1), 235-267. doi: 10.1111/j.1468-0262.2006.00655.x
[2]	Austin P. C. (2009). Some methods of propensity-score matching had superior performance to others: results of an empirical investigation and Monte Carlo simulations. Biometrical Journal: Journal of Mathematical Methods in Biosciences, 51(1), 171-184.
[3]	Azoulay P., Graff Zivin J. S., & Wang J. (2010). Superstar extinction. The Quarterly Journal of Economics, 125(2), 549-589. doi: 10.1162/qjec.2010.125.2.549
[4]	Becker S. O., & Ichino A. (2002). Estimation of average treatment effects based on propensity scores. The Stata Journal, 2(4), 358-377. doi: 10.1177/1536867X0200200403
[5]	Blackwell M., Iacus S., King G., & Porro G. (2009). cem: Coarsened exact matching in Stata. The Stata Journal, 9(4), 524-546. doi: 10.1177/1536867X0900900402
[6]	Borjas G. J., & Doran K. B. (2015). Prizes and productivity how winning the fields medal affects scientific output. Journal of human resources, 50(3), 728-758.
[7]	Bourdieu P. (2004). Science of science and reflexivity. Polity.
[8]	Broadus R. N. (1987). Toward a definition of “bibliometrics”. Scientometrics, 12(5), 373-379. doi: 10.1007/BF02016680
[9]	Chan H. F., Frey B. S., Gallus J., & Torgler B. (2014). Academic honors and performance. Labour Economics, 31, 188-204. doi: 10.1016/j.labeco.2014.05.005
[10]	Coca-Perraillon M. (2007). Local and global optimal propensity score matching. SAS Global Forum, https://support.sas.com/resources/papers/proceedings/proceedings/forum2007/185-2007.pdf
[11]	Cover T., & Hart P. (1967). Nearest neighbor pattern classification. IEEE transactions on information theory, 13(1), 21-27. doi: 10.1109/TIT.1967.1053964
[12]	Donald S. G., & Lang K. (2007). Inference with difference-in-differences and other panel data. The review of Economics and Statistics, 89(2), 221-233. doi: 10.1162/rest.89.2.221
[13]	Fortunato S., Bergstrom C. T., Börner K., Evans J. A., Helbing D., Milojević S., Petersen A. M., Radicchi F., Sinatra R., & Uzzi B. (2018). Science of science. Science, 359(6379), eaao0185. doi: 10.1126/science.aao0185
[14]	Garrido M. M., Kelley A. S., Paris J., Roza K., Meier D. E., Morrison R. S., & Aldridge M. D. (2014). Methods for constructing and assessing propensity scores. Health services research, 49(5), 1701-1720. doi: 10.1111/1475-6773.12182 pmid: 24779867
[15]	Heckman J. J., LaLonde R. J., & Smith J. A. (1999). The economics and econometrics of active labor market programs. In Handbook of labor economics (Vol. 3, pp. 1865-2097). Elsevier.
[16]	Hill J., & Reiter J. P. (2006). Interval estimation for treatment effects using propensity score matching. Statistics in medicine, 25(13), 2230-2256. doi: 10.1002/sim.2277 pmid: 16220488
[17]	Hirsch J. E. (2005). An index to quantify an individual's scientific research output. Proc Natl Acad Sci U S A, 102(46), 16569-16572. https://doi.org/10.1073/pnas.0507655102 doi: 10.1073/pnas.0507655102
[18]	Huang J. M., Gates A. J., Sinatra R., & Barabási A. L. (2020). Historical comparison of gender inequality in scientific careers across countries and disciplines. Proceedings of the National Academy of Sciences, 117(9), 4609-4616. doi: 10.1073/pnas.1914221117
[19]	Huber M., Lechner M., & Steinmayr A. (2015). Radius matching on the propensity score with bias adjustment: tuning parameters and finite sample behaviour. Empirical Economics, 49(1), 1-31. doi: 10.1007/s00181-014-0847-1
[20]	Iacus S. M., King G., & Porro G. (2012). Causal inference without balance checking: Coarsened exact matching. Political analysis, 20(1), 1-24.
[21]	Jin C., Ma Y. F., & Uzzi B. (2021). Scientific prizes and the extraordinary growth of scientific topics. Nature Communications, 12(1), 1-11. doi: 10.1038/s41467-020-20314-w
[22]	King G., & Nielsen R. (2019). Why propensity scores should not be used for matching. Political analysis, 27(4), 435-454. doi: 10.1017/pan.2019.11
[23]	King G., Nielsen R., Coberley C., Pope J. E., & Wells A. (2011). Comparative effectiveness of matching methods for causal inference. Unpublished manuscript, Institute for Quantitative Social Science, Harvard University, Cambridge, MA.
[24]	Leuven E., & Sianesi B. (2003). PSMATCH2: Stata module to perform full Mahalanobis and propensity score matching, common support graphing, and covariate imbalance testing.
[25]	Li J. C., Yin Y. A., Fortunato S., & Wang D. S. (2019). A dataset of publication records for Nobel laureates. Scientific data, 6(1), 1-10. doi: 10.1038/s41597-018-0005-2
[26]	Li W. H., Aste T., Caccioli F., & Livan G. (2019). Early coauthorship with top scientists predicts success in academic careers. Nat Commun, 10(1), 5170. https://doi.org/10.1038/s41467-019-13130-4 doi: 10.1038/s41467-019-13130-4 pmid: 31729362
[27]	Liu M. J., & Hu X. (2022). Movers' advantages: The effect of mobility on scientists' productivity and collaboration. Journal of Informetrics, 16(3), 101311. doi: 10.1016/j.joi.2022.101311
[28]	Ma Y. F., Mukherjee S., & Uzzi B. (2020). Mentorship and protege success in STEM fields. Proc Natl Acad Sci U S A, 117(25), 14077-14083. https://doi.org/10.1073/pnas.1915516117 doi: 10.1073/pnas.1915516117
[29]	Ma Y. F., & Uzzi B. (2018). Scientific prize network predicts who pushes the boundaries of science. Proc Natl Acad Sci U S A, 115(50), 12608-12615. https://doi.org/10.1073/pnas.1800485115 doi: 10.1073/pnas.1800485115
[30]	McCaffrey D. F., & Bell R. M. (2003). Bias reduction in standard errors for linear regression with multi-stage samples. Quality control and applied statistics, 48(6), 677-682.
[31]	Merton R. K. (1968). The Matthew effect in science: The reward and communication systems of science are considered. Science, 159(3810), 56-63. pmid: 5634379
[32]	Petersen A. M. (2018). Multiscale impact of researcher mobility. Journal of The Royal Society Interface, 15(146), 20180580. doi: 10.1098/rsif.2018.0580
[33]	Petersen A. M., Fortunato S., Pan R. K., Kaski K., Penner O., Rungi A., Riccaboni M., Stanley H. E., & Pammolli F. (2014). Reputation and impact in academic careers. Proc Natl Acad Sci U S A, 111(43), 15316-15321. https://doi.org/10.1073/pnas.1323111111 doi: 10.1073/pnas.1323111111
[34]	Price D. J. (1986). Little science, big science... and beyond (Vol. 480). Columbia University Press New York.
[35]	Radicchi F., Fortunato S., & Castellano C. (2008). Universality of citation distributions: Toward an objective measure of scientific impact. Proceedings of the National Academy of Sciences, 105(45), 17268-17272. doi: 10.1073/pnas.0806977105
[36]	Rosenbaum P. R., & Rubin D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41-55. doi: 10.1093/biomet/70.1.41
[37]	Roth J., Sant'Anna P. H., Bilinski A., & Poe J. (2022). What's Trending in Difference-in-Differences? A Synthesis of the Recent Econometrics Literature. arXiv preprint arXiv:2201.01194.
[38]	Rubin D. B. (1977). Assignment to treatment group on the basis of a covariate. Journal of educational Statistics, 2(1), 1-26. doi: 10.3102/10769986002001001
[39]	Rubin D. B. (2005). Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100(469), 322-331. doi: 10.1198/016214504000001880
[40]	Sinatra R., Wang D. S., Deville P., Song C. M., & Barabasi A. L. (2016). Quantifying the evolution of individual scientific impact. Science, 354(6312). https://doi.org/10.1126/science.aaf5239
[41]	Stuart E. A. (2010). Matching methods for causal inference: A review and a look forward. Statistical science: a review journal of the Institute of Mathematical Statistics, 25(1), 1.
[42]	Uzzi B., Mukherjee S., Stringer M., & Jones B. (2013). Atypical combinations and scientific impact. Science, 342(6157), 468-472. doi: 10.1126/science.1240474 pmid: 24159044
[43]	Wang D. S., Song C. M., & Barabási A. L. (2013). Quantifying long-term scientific impact. Science, 342(6154), 127-132. doi: 10.1126/science.1237825 pmid: 24092745
[44]	Wang J. X. (2021). To use or not to use propensity score matching? Pharmaceutical Statistics, 20(1), 15-24. doi: 10.1002/pst.2051
[45]	Yang Y., Chawla N. V., & Uzzi B. (2019). A network's gender composition and communication pattern predict women's leadership success. Proceedings of the National Academy of Sciences, 116(6), 2033-2038. doi: 10.1073/pnas.1721438116

	Treatment	Control	Treatment-Control
Prior	E(Y⁰¹)	E(Y⁰⁰)	E(Y⁰¹)-E(Y⁰⁰)
Posterior	E(Y¹¹)	E(Y¹⁰)	E(Y¹¹)-E(Y¹⁰)
Posterior-Prior	E(Y¹¹)-E(Y⁰¹)	E(Y¹⁰)-E(Y⁰⁰)	[E(Y¹¹)-E(Y⁰¹)]-[E(Y¹⁰)-E(Y⁰⁰)]

	Prize year	Total #Publication	Total #Citation
Minimum	1934	10	443
Maximum	2011	1,627	194,896
Mean	1987	245.9	25,662.6
Median	1991	157	13,061
Standard deviation	18.6	266.1	31,571.4
Count	335	335	335

	Average citation index before prizewinning
Matching Techniques	-		CEM		PSM
Group	Control	Treated	Control	Treated	Control	Treated
Observations	227	108	80	80	84	108
Mean	4.7083	5.2350	5.0701	5.2869	5.1353	5.2350
Std. dev.	1.4886	1.5034	1.7627	1.6595	1.5821	1.5034
Minimum	0.3519	0.1997	0	0	1.3754	0.1997
Maximum	8.4902	8.2194	8.3668	8.4933	8.4902	8.2194

	DID regression Model Results
Model	Model I	Model II	Model III
Matching Techniques	-	CEM	PSM
ATT	-0.1488	-0.0780	-0.1481
SE	0.0998	0.1307	0.1317
Common trend assumption	Fail	Pass	Pass
p-value	0.0999	0.6836	0.1582
Fixed Effect Controls
Individual	Yes	Yes	Yes
Year	Yes	Yes	Yes