Please wait a minute...
Journal of Data and Information Science  2018, Vol. 3 Issue (1): 19-39    DOI: 10.2478/jdis-2018-000219
Research Paper     
Does Monetary Support Increase the Number of Scientific Papers? An Interrupted Time Series Analysis
Tonta Yaşar()
Department of Information Management, Faculty of Letters, Hacettepe University, 06800 Beytepe, Ankara, Turkey
Download: PDF (3853 KB)      HTML  
Export: BibTeX | EndNote (RIS)      

Abstract  

Purpose: One of the main indicators of scientific production is the number of papers published in scholarly journals. Turkey ranks 18th place in the world based on the number of scholarly publications. The objective of this paper is to find out if the monetary support program initiated in 1993 by the Turkish Scientific and Technological Research Council (TüB?TAK) to incentivize researchers and increase the number, impact, and quality of international publications has been effective in doing so.

Design/methodology/approach: We analyzed some 390,000 publications with Turkish affiliations listed in the Web of Science (WoS) database between 1976 and 2015 along with about 157,000 supported ones between 1997 and 2015. We used the interrupted time series (ITS) analysis technique (also known as “quasi-experimental time series analysis” or “intervention analysis”) to test if TüB?TAK’s support program helped increase the number of publications. We defined ARIMA (1,1,0) model for ITS data and observed the impact of TüB?TAK’s support program in 1994, 1997, and 2003 (after one, four and 10 years of its start, respectively). The majority of publications (93%) were full papers (articles), which were used as the experimental group while other types of contributions functioned as the control group. We also carried out a multiple regression analysis.

Findings: TüB?TAK’s support program has had negligible effect on the increase of the number of papers with Turkish affiliations. Yet, the number of other types of contributions continued to increase even though they were not well supported, suggesting that TüB?TAK’s support program is probably not the main factor causing the increase in the number of papers with Turkish affiliations.

Research limitations: Interrupted time series analysis shows if the “intervention” has had any significant effect on the dependent variable but it does not explain what caused the increase in the number of papers if it was not the intervention. Moreover, except the “intervention”, other “event(s)” that might affect the time series data (e.g., increase in the number of research personnel over the years) should not occur during the period of analysis, a prerequisite that is beyond the control of the researcher.

Practical implications: TüB?TAK’s “cash-for-publication” program did not seem to have direct impact on the increase of the number of papers published by Turkish authors, suggesting that small amounts of payments are not much of an incentive for authors to publish more. It might perhaps be a better strategy to concentrate limited resources on a few high impact projects rather than to disperse them to thousands of authors as “micropayments.”

Originality/value: Based on 25 years’ worth of payments data, this is perhaps one of the first large-scale studies showing that “cash-for-publication” policies or “piece rates” paid to researchers tend to have little or no effect on the increase of researchers’ productivity. The main finding of this paper has some implications for countries wherein publication subsidies are used as an incentive to increase the number and quality of papers published in international journals. They should be prepared to consider reviewing their existing support programs (based usually on bibliometric measures such as journal impact factors) and revising their reward policies.



Key wordsPerformance-based research funding systems      Publication subsidies      Publication support programs      Interrupted time series analysis     
Published: 19 March 2018
Corresponding Authors: Tonta Ya?ar     E-mail: yasartonta@gmail.com
Cite this article:

Tonta Yaşar. Does Monetary Support Increase the Number of Scientific Papers? An Interrupted Time Series Analysis. Journal of Data and Information Science, 2018, 3(1): 19-39.

URL:

http://manu47.magtech.com.cn/Jwk3_jdis/10.2478/jdis-2018-000219     OR     http://manu47.magtech.com.cn/Jwk3_jdis/Y2018/V3/I1/19

Year Papers Other Total Year Papers Other Total
N % N % N N % N % N
1976 216 80 53 20 269 1996 3,359 84 623 16 3,982
1977 229 72 91 28 320 1997 3,844 83 796 17 4,640
1978 272 72 108 28 380 1998 4,460 82 1,001 18 5,461
1979 256 71 106 29 362 1999 5,201 83 1,078 17 6,279
1980 343 74 123 26 466 2000 5,462 84 1,059 16 6,521
1981 299 73 110 27 409 2001 6,684 84 1,271 16 7,955
1982 315 70 132 30 447 2002 8,985 86 1,434 14 10,419
1983 354 72 141 28 495 2003 10,662 84 1,978 16 12,640
1984 420 77 129 23 549 2004 13,199 84 2,488 16 15,687
1985 447 76 145 24 592 2005 14,194 83 2,877 17 17,071
1986 506 77 151 23 657 2006 15,070 79 4,099 21 19,169
1987 588 77 174 23 762 2007 17,853 80 4,414 20 22,267
1988 672 75 227 25 899 2008 19,327 82 4,379 18 23,706
1989 829 80 209 20 1,038 2009 21,655 82 4,627 18 26,282
1990 912 78 261 22 1,173 2010 22,833 83 4,760 17 27,593
1991 1,134 80 290 20 1,424 2011 23,588 82 5,325 18 28,913
1992 1,351 77 406 23 1,757 2012 25,254 82 5,607 18 30,861
1993 1,519 76 482 24 2,001 2013 26,526 79 7,200 21 33,726
1994 1,754 73 643 27 2,397 2014 27,242 79 7,315 21 34,557
1995 2,233 72 885 28 3,118 2015 28,662 79 7,530 21 36,192
Total / Avg. 318,709 81 74,727 19 393,436
Table 1 Number of publications with Turkish affiliations (1976-2015).
Figure 1. Number of papers and total number of publications with Turkish affiliations (1976-2015).
Year # of papers supported by TüB?TAK # of papers with Turkish affiliations (WoS) Percentage supported (%)
1997 2,247 3,844 58
1998 2,657 4,460 60
1999 3,088 5,201 59
2000 3,298 5,462 60
2001 4,216 6,684 63
2002 5,888 8,985 66
2003 7,517 10,662 71
2004 9,511 13,199 72
2005 7,036 14,194 50
2006 8,122 15,070 54
2007 10,551 17,853 59
2008 10,411 19,327 54
2009 11,554 21,655 53
2010 11,592 22,833 51
2011 9,574 23,588 41
2012 10,641 25,254 42
2013 10,203 26,526 38
2014 10,257 27,242 38
2015 8,014 28,662 28
Total 146,377 300,701 49
Table 2 Number of papers supported by TüB?TAK (1997-2015).
Figure 2. Number of papers listed in WoS with Turkish affiliations and supported by TüB?TAK (1997-2015).
Figure 3. Time path of papers with Turkish affiliations (1976-2015).
Figure 4. Correlograms of autocorrelation (ACF) and partial autocorrelations (PACF) functions.
Model Statistics
Model Number of Predictors Model Fit statistics Ljung-Box Q(18) Number of Outliers
Stationary R-squared Statistics DF Sig.
Makale say?s?-Model_1 3 .607 23.531 17 .133 0
Table 3 Test statistic (Ljung Box).
Estimate SE t Sig.
# of papers Model 1 # of papers No transformation Constant AR Lag 1 -57.138
.153
334.811
.170
-.171
.899
.866
.375
Difference 1
Time series No transformation Numerator Lag 0 14.051 29.910 .470 .642
Before/after Support
Program
No transformation Numerator Lag 0 11.258 708.202 .016 .987
Effect No transformation Numerator Lag 0 29.091 36.715 .792 .434
Table 4 ARIMA Model Parameters.
Year Predicted increase SE t-value p-value
1994 563.633 390.084 1.446 .157
1997 651.241 431.129 1.510 .140
2003 825.784 571.279 1.446 .157
2015 1,174.941 947.761 1.240 .224
Table 5 Values showing the delayed effect of TüB?TAK’s support program.
Figure 5. Rate of increase of papers and non-papers.
Note: Scales for y axes on the left and right are different. y axis on the left represents the number of papers while the one on the right respresents the number of non-papers.
Time series # of pubs # of papers Stage Impact Pre-int 1 Post-int 1 Pre-int 4 Post-int 4 Pre-int 10 Post-int 10 Pre-int 21 Post-int 21
1 269 216 0 0 1 0 1 0 1 0 1 0
2 320 229 0 0 2 0 2 0 2 0 2 0
3 380 272 0 0 3 0 3 0 3 0 3 0
4 362 256 0 0 4 0 4 0 4 0 4 0
5 466 343 0 0 5 0 5 0 5 0 5 0
6 409 299 0 0 6 0 6 0 6 0 6 0
7 447 315 0 0 7 0 7 0 7 0 7 0
8 495 354 0 0 8 0 8 0 8 0 8 0
9 549 420 0 0 9 0 9 0 9 0 9 0
10 592 447 0 0 10 0 10 0 10 0 10 0
11 657 506 0 0 11 0 11 0 11 0 11 0
12 762 588 0 0 12 0 12 0 12 0 12 0
13 899 672 0 0 13 0 13 0 13 0 13 0
14 1,038 829 0 0 14 0 14 0 14 0 14 0
15 1,173 912 0 0 15 0 15 0 15 0 15 0
16 1,424 1,134 0 0 16 0 16 0 16 0 16 0
17 1,757 1,351 0 0 17 0 17 0 17 0 17 0
18 2,001 1,519 0 0 18 0 18 0 18 0 18 0
19 2,397 1,754 1 19 19 0 22 -3 28 -9 40 -21
20 3,118 2,233 1 20 19 1 22 -2 28 -8 40 -20
21 3,982 3,359 1 21 19 2 22 -1 28 -7 40 -19
22 4,640 3,844 1 22 19 3 22 0 28 -6 40 -18
23 5,461 4,460 1 23 19 4 22 1 28 -5 40 -17
24 6,279 5,201 1 24 19 5 22 2 28 -4 40 -16
25 6,521 5,462 1 25 19 6 22 3 28 -3 40 -15
26 7,955 6,684 1 26 19 7 22 4 28 -2 40 -14
27 10,419 8,985 1 27 19 8 22 5 28 -1 40 -13
28 12,640 10,662 1 28 19 9 22 6 28 0 40 -12
29 15,687 13,199 1 29 19 10 22 7 28 1 40 -11
30 17,071 14,194 1 30 19 11 22 8 28 2 40 -10
31 19,169 15,070 1 31 19 12 22 9 28 3 40 -9
32 22,267 17,853 1 32 19 13 22 10 28 4 40 -8
33 23,706 19,327 1 33 19 14 22 11 28 5 40 -7
34 26,282 21,655 1 34 19 15 22 12 28 6 40 -6
35 27,593 22,833 1 35 19 16 22 13 28 7 40 -5
36 28,913 23,588 1 36 19 17 22 14 28 8 40 -4
37 30,861 25,254 1 37 19 18 22 15 28 9 40 -3
38 33,726 26,526 1 38 19 19 22 16 28 10 40 -2
39 34,557 27,242 1 39 19 20 22 17 28 11 40 -1
40 36,192 28,662 1 40 19 21 22 18 28 12 40 0
 Appendix A. Time series data prepared for interrupted time series analysis (1976-2015)
[1]   Abramo G.,& D’Angelo .+? (2011). National-scale research performance assessment at the individual level. Scientometrics, 86(2), 347-364.
doi: 10.1007/s11192-010-0297-2
[2]   Abramo G.,& D’Angelo .+? (2016). Refrain from adopting the combination of citation and journal metrics to grade publications, as used in the Italian national research assessment exercise (VQR 2011-2014). Scientometrics, 109(3), 1-13.
doi: 10.1007/s11192-016-2008-0
[3]   Abramo G., D’Angelo C.A., & Di Costa F. (2011). National research assessment exercises: A comparison of peer review and bibliometrics rankings. Scientometrics, 89: 929. https://doi.org/10.1007/s11192-011-0459-x.
doi: 10.1007/s11192-011-0459-x
[4]   Adam D. (2002). Citation analysis: The counting house. Nature, 415(415), 726-729.
doi: 10.1038/415726a
[5]   Akademik Te?vik ?dene?i Y?netmeli?i (By-law of Payment of Academic Incentive). (2015).Resm? Gazete. Retrieved from .
[6]   Albarrán P., Crespo J.A., Ortu?o I., & Ruiz-Castillo J. (2011). The skewness of science in 219 subfields and a number of aggregates. Scientometrics, 88(2), 385-397.
doi: 10.1007/s11192-011-0407-9
[7]   Auranen O.,& Nieminen ,M. (2010). University research funding and publication performance—An international comparison. Research Policy, 39(6), 822-834.
doi: 10.1016/j.respol.2010.03.003
[8]   Butler L. (2003). Explaining Australia’s increased share of ISI publications—the effects of a funding formula based on publication counts. Research Policy, 32(1), 143-155.
doi: 10.1016/S0048-7333(02)00007-0
[9]   Butler L. (2004). What happens when funding is linked to publication counts? In H.F. Moed et al.,(Ed.), Handbook of Quantitative Science and Technology Research: The Use of Publication and Patent Statistics in Studies of S&T Systems (pp. 389-405). Dordrecht: Kluwer.
[10]   Butler L. (2017). Response to van den Besselaar et al.: What happens when the Australian context is misunderstood. Journal of Informetrics, 11(3), 919-922.
doi: 10.1016/j.joi.2017.05.017
[11]   Casadevall A.,& Fang ,F.C. (2012. Causes for the persistence of impact factor mania. mBio, 5(2). Retrieved on April 28, 2017, from 2012). Causes for the persistence of impact factor mania. mBio, 5(2). Retrieved on April 28, 2017, from .
[12]   Cochrane Effective Practice and Organisation of Care Review Group. Data Collection Checklist.(2002). Retrieved from .
[13]   De Boer H., Jongbloed B.W.A., Benneworth S., Cremonini, L. Kolster R., Kottmann A., . . & Vossensteyn, J.J. (2015). Performance-based Funding and Performance Agreements in Fourteen Higher Education Systems. Enschede: University of Twente. Retrieved from
[14]   European Commission (2010). Assessing Europe’s University-Based Research. Retrieved from .
[15]   Franzoni C., Scellato G., & Stephan P. (2011). Changing incentives to publish. Science, 33(6043), 702-703.
[16]   Garfield E. (1972). Citation analysis as a tool in journal evaluation. Science, 178(4060), 471-479.
doi: 10.1126/science.178.4060.471
[17]   Geuna A.,& Martin ,B. (2003). University research evaluation and funding: An international comparison. Minerva, 41(4), 277-304.
doi: 10.1023/B:MINE.0000005155.70870.bd
[18]   Good B., Vermeulen N., Tiefenthaler B., & Arnold E. (2015). Counting quality? The Czech performance-based research funding system. Research Evaluation, 24(2), 91-105.
doi: 10.1093/reseval/rvu035
[19]   Gl?nzel W.,& Moed, H.F.(2002). Journal impact measures in bibliometric research. Scientometrics, 53(2), 171-193.
doi: 10.1023/A:1014848323806
[20]   Herbst M. (2007). Financing public universities: The case of performance funding. Dordrecht: Springer.
doi: 10.1007/978-1-4020-5560-7
[21]   Heywood J.S., Wei X., & Ye G. (2011). Piece rates for professors. Economics Letters, 113(3), 285-287.
doi: 10.1016/j.econlet.2011.08.005
[22]   Hicks D. (2012). Performance-based university research funding systems. Research Policy, 41(2), 251-261.
doi: 10.1016/j.respol.2011.09.007
[23]   Hicks D. (2017). What year? Difficulties in identifying the effect of policy on university output. Journal of Informetrics, 11(3), 933-936.
doi: 10.1016/j.joi.2017.05.020
[24]   Interrupted time series analysis. (2013. Retrieved on April 28, 2017, from 2013). Retrieved on April 28, 2017, from .
[25]   Kamalski J.et al. (2017).World of Research 20152017). World of Research 2015: Revealing Patterns and Archetypes in Scientific Research. Elsevier Analytic Services. Retrieved from .
[26]   Larivière V., Kiermer V., MacCallum C.,. . . & Curry S. (2016). A simple proposal for the publication of journal citation distributions. Retrieved from .
[27]   McDowall D., McCleary R., Meidinger E.E., & Hay R.A. (1980). Interrupted Time Series Analysis. Newbury Park: Sage.
[28]   Osuna C., Cruz-Castro L., & Sanz-Menéndez L. (2011). Overturning some assumptions about the effects of evaluation systems on publication performance. Scientometrics, 86(3), 575-592.
doi: 10.1007/s11192-010-0312-7
[29]   ?nder C., ?evkli M., Alt?nok T., & Tavuk?uo?lu C. (2008). Institutional change and scientific research: A preliminary bibliometric analysis of institutional influences on Turkey’s recent social science publications. Scientometrics, 76(3), 543-560.
doi: 10.1007/s11192-007-1878-6
[30]   Schneider J.W. (2009). An outline of the bibliometric indicator used for performance-based funding of research institutions in Norway. European Political Science, 8(3), 364-378.
doi: 10.1057/eps.2009.19
[31]   Seglen P.O. (1997 February 5). Why the impact factor of journals should not be used for evaluating research . British Medical Journal 314 (7079), 498-502. Retrieved from .
[32]   Shao J.,& Shen H. (2012). Research assessment: The overemphasized impact factor in China. Research Evaluation, 21(3), 199-203.
doi: 10.1093/reseval/rvs011
[33]   TüB?TAK Türkiye Adresli Uluslararas? Bilimsel Yay?nlar? Te?vik (UBYT) Program? Uygulama Usul ve Esaslar?.(2016). Retrieved from .
[34]   Teixeira da Silva,J.A. (2017). Does China need to rethink its metrics- and citations-based research reward policies? Scientometrics, 112(3), 1853-1857.
doi: 10.1007/s11192-017-2430-y
[35]   Tonta Y. (2014. Use and misuse of bibliometric measures for assessment of academic performance, tenure and publication support. In the 77th Annual Meeting of the Association for Information Science and Technology, October 31 - November 5, 2014, Seattle, WA. 2014). Use and misuse of bibliometric measures for assessment of academic performance, tenure and publication support. In the 77th Annual Meeting of the Association for Information Science and Technology, October 31 - November 5, 2014, Seattle, WA. .
[36]   Tonta Y. (2015). Support programs to increase the number of scientific publications using bibliometric measures: The Turkish case.In A.A. Salah et al.(Eds.). Proceedings of ISSI 2015 Istanbul: 15th International Society of Scientometrics and Informetrics Conference, Istanbul, Turkey, 29 June to 4 July, 2015 (pp. 767-777). ?stanbul: Bo?azi?i University.
[37]   Tonta Y. (2017a. Does monetary support increase the number of scientific papers? An interrupted time series analysis. Paper presented at ISSI 2017: 16th International Scientometrics and Informetrics Conference, 16-20 October 2017, Wuhan University, Wuhan, China. Retrieved from 2017a). Does monetary support increase the number of scientific papers? An interrupted time series analysis. Paper presented at ISSI 2017: 16th International Scientometrics and Informetrics Conference, 16-20 October 2017, Wuhan University, Wuhan, China. Retrieved from .
[38]   Tonta Y. (2017b). TüB?TAK Türkiye Adresli Uluslararas? Bilimsel Yay?nlar? Te?vik (UBYT) Program?n?n De?erlendirilmesi. Ankara: TüB?TAK ULAKB?M . Retrieved from.
[39]   research funding? Butler’s Australian case revisited. Journal of Informetrics, 11(3), 905-918. DOI: .
[40]   van Raan,A.F.J. (2005). Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics, 62(1), 133-143.
doi: 10.1007/s11192-005-0008-6
[41]   Wouters P., Thelwall M., Kousha K., Waltman L., de Rijcke S.,Rushforth, A. & Franssen, T. (2015). The metric tide literature review: Supplementary report I to the independent review of the role of metrics in research assessment and management. Retrieved from .
[42]   Yuret T. (2017). Do researchers pay attention to publication subsidies? Journal of Informetrics, 11(2), 423-434.
doi: 10.1016/j.joi.2017.02.010
[43]   Yurtsever E., Gülg?z S., Yedek?io?lu ?.A., & Tonta M. (2001). Sosyal Bilimler At?f Dizini’nde (SSCI) Türkiye 1970-1999 (Turkey in Social Sciences Citation Index (SSCI): 1970-1999). Ankara: Türkiye Bilimler Akademisi.
[44]   Yurtsever E., Gülg?z S., Yedek?io?lu ?.A., & Tonta M. (2002). Sa?l?k Bilimleri,Mühendislik ve Temel Bilimlerde Türkiye’nin Uluslararas? At?f Dizinindeki Yeri 1973-1999 (Turkey’s Place in Health Sciences, Engineering and Basic Sciences in International Citation Index). Ankara: Türkiye Bilimler Akademisi.
No related articles found!