Measuring and Visualizing Research Collaboration and Productivity

doi:10.2478/jdis-2018-0004

Abstract

Abstract:

Purpose: This paper presents findings of a quasi-experimental assessment to gauge the research productivity and degree of interdisciplinarity of research center outputs. Of special interest, we share an enriched visualization of research co-authoring patterns.

Design/methodology/approach: We compile publications by 45 researchers in each of 1) the iUTAH project, which we consider here to be analogous to a “research center,” 2) CG1— a comparison group of participants in two other Utah environmental research centers, and 3) CG2—a comparison group of Utah university environmental researchers not associated with a research center. We draw bibliometric data from Web of Science and from Google Scholar. We gather publications for a period before iUTAH had been established (2010-2012) and a period after (2014-2016). We compare these research outputs in terms of publications and citations thereto. We also measure interdisciplinarity using Integration scoring and generate science overlay maps to locate the research publications across disciplines.

Findings: We find that participation in the iUTAH project appears to increase research outputs (publications in the After period) and increase research citation rates relative to the comparison group researchers (although CG1 research remains most cited, as it was in the Before period). Most notably, participation in iUTAH markedly increases co-authoring among researchers—in general; and for junior, as well as senior, faculty; for men and women: across organizations; and across disciplines.

Research limitations: The quasi-experimental design necessarily generates suggestive, not definitively causal, findings because of the imperfect controls.

Practical implications: This study demonstrates a viable approach for research assessment of a center or program for which random assignment of control groups is not possible. It illustrates use of bibliometric indicators to inform R&D program management.

Originality/value: New visualizations of researcher collaboration provide compelling comparisons of the extent and nature of social networking among target cohorts.

Key words: Bibliometrics, Interdisciplinarity, Multidisciplinarity, Research evaluation, Research collaboration mapping, Science visualization, Scientometrics, Social network analysis

Garner, Jon, Alan L. Porter, Andreas Leidolf, and Michelle Baker. "Measuring and Visualizing Research Collaboration and Productivity." Journal of Data and Information Science, vol.3, no.1, 2018, pp.54-81. DOI: 10.2478/jdis-2018-0004

References

[1]	Campbell D.,& Stanley ,J.(1963). Experimental and quasi-experimental designs for research. Chicago: Rand-McNally.
[2]	Cook T., &Campbell ,D.A. (1979). Quasi-experimentation. New York:Houghton Mifflin.
[3]	Carley S.,& Porter ,A.L.(2012). A forward diversity index. Scientometrics, 90(2), 407-427.
[4]	Carley S., Porter A.L., Rafols I., & Leydesdorff, L.(under review). Visualization of disciplinary profiles: Enhanced science overlay maps. Journal of Data and Information Science, 2017(3), 68-111.
[5]	De Bellis,N. (2009).Bibliometrics and citation analysis：From the science citation index to cybermetrics, Lanham, MD: Scarecrow Press From the science citation index to cybermetrics,Lanham, MD: Scarecrow Press.
[6]	de Nooy,W., Mrvar , A., & Batgelj V. (2011). Exploratory social network analysis with Pajek (2nd Edition) .New York,NY: Cambridge University Press.
[7]	Garner J., Porter A.L., Borrego M., Tran E., & Teutonico R. (2013). Facilitating social and natural science cross-disciplinarity: Assessing the human and social dynamics program, Research Evaluation, 22(2), 134-144.
[8]	Garner J., Porter A.L., & Newman N.C. (2014). Distance and velocity measures: Using citations to determine breadth and speed of research impact. Scientometrics, 100(3), 687-703.
[9]	Garner J., Porter A.L., Newman N.C., & Crowl T.A. (2012). Assessing research network and disciplinary engagement changes induced by an NSF program. Research Evaluation, 21(2), 89-104.
[10]	Gl?nzel W.,& Schubert ,A.(2003). A new classification scheme of science fields and subfields designed for scientometric evaluation purposes. Scientometrics, 56(3), 357-367.
[11]	Hall K.L., Stokols D., Stipelman B.A., Vogel A.L., Feng A., Masimore B., Morgan G., Moser R.P., Marcus S.E., & Berrigan D. (2012). Assessing the value of team science: A study comparing center- and investigator-initiated grants. American Journal of Preventive Medicine, 42(2), 157-163.
[12]	Hirsch J.E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102(46), 16569-16572.
[13]	Klavans R.,& Boyack ,K.W.(2009). Toward a consensus map of science. Journal of the American Society for Information Science and Technology, 60(3), 455-476.
[14]	Klavans R.,& Boyack ,K.W.(2017). Which type of citation analysis generates the most accurate taxonomy of scientific and technical knowledge? Journal of the American Society for Information Science and Technology, 68(4), 984-998.
[15]	Kwon S.,Solomon G.E.A., Youtie , J., & Porter, A.L.(under review). A measure of interdisciplinary knowledge flow between specific fields: Implications for impact and funding . PLoS One.
[16]	Leydesdorff L. (2008). On the normalization and visualization of author co-citation data: Salton’s Cosine versus the Jaccard Index. Journal of the American Society for Information Science and Technology, 59(1), 77-85.
[17]	Leydesdorff L.,& Bornmann ,L.(2016). The operationalization of “fields” as WoS subject categories (WCs) in evaluative bibliometrics: The cases of “library and information science” and “science & technology studies.” Journal of the American Society for Information Science and Technology, 67(3), 707-714.
[18]	Leydesdorff L., Carley S., & Rafols I. (2013). Global maps of science based on the new Web-of-Science Categories. Scientometrics, 94(2), 589-593.
[19]	Leydesdorff L.,& Rafols ,I.(2009). A global map of science based on the ISI subject categories. Journal of the American Society for Information Science and Technology, 60(2), 348-362.
[20]	National Academies Committee on Facilitating Interdisciplinary Research, Committee on Science,Engineering and Public Policy(COSEPUPx) (2005). Facilitating Interdisciplinary Research. Washington,DC: National Academies Press.
[21]	Peck L.R. (2016), Social experiments in practice: The what, why, when, where, and how of Experimental Design & Analysis. New Directions for Evaluation, No. 152 (Winter), New York: Wiley.
[22]	Porter A.L., Cohen A.S., Roessner J.D., & Perreault M. (2007). Measuring researcher interdisciplinarity. Scientometrics, 72(1), 117-147.
[23]	Porter A.L., Garner J., & Crowl T. (2012). The RCN (Research Coordination Network) experiment: Can we build new research networks? BioScience, 62, 282-288.
[24]	Porter A.L., Newman N.C., Myers W., & Schoeneck D. (2003). Projects and publications: Interesting patterns in U.S. Environmental Protection Agency research.Research Evaluation, 12(3), 171-182.
[25]	Porter A.L.,& Rafols ,I.(2009). Is science becoming more interdisciplinary? Measuring and mapping six research fields over time. Scientometrics, 81(3), 719-745.
[26]	Porter A.L., Roessner J.D., & Heberger A.E. (2008). How interdisciplinary is a given body of research? Research Evaluation, 17(4), 273-282.
[27]	Porter A.L., Schoeneck D.J., & Carley S. (2013). Measuring the extent to which a research domain is self-contained. In Proceedings of the 14th International Conference on Scientometrics and Informetrics (ISSI2013), July 15-19, Vienna, Austria.
[28]	Porter A.L., Schoeneck D.J., Roessner D., & Garner J. (2010). Practical research proposal and publication profiling. Research Evaluation, 19(1), 29-44.
[29]	Porter A.L., Schoeneck D.J., Solomon G., Lakhani H., & Dietz J. (2013). Measuring and mapping interdisciplinarity: Research & evaluation on education in science & engineering (“REESE”) and STEM. In American Education Research Association Annual MeetingApril 27-May 1, San Francisco.
[30]	Rafols I. (2014). Knowledge integration and diffusion: Measures and mapping of diversity and coherence. In Ding, Y., Rousseau, R., & Wolfram, D.(Eds.) Measuring scholarly Impact: Methods and Practice (pp. 169-190). Berlin: Springer.
[31]	Rafols I.,& Leydesdorff ,L.(2009). Content-based and algorithmic classifications of journals: Perspectives on the dynamics of scientific communication and indexer effects. Journal of the American Society for Information Science and Technology, 60(9), 1823-1835.
[32]	Rafols I.,& Meyer ,M.(2010). Diversity and network coherence as indicators of interdisciplinarity: Case studies in bionanoscience. Scientometrics, 82(2), 263-287.
[33]	Rafols I., Porter A., & Leydesdorff L. (2010). Science overlay maps: A new tool for research policy and library management. Journal of the American Society for Information Science and Technology, 61(9), 1871-1887.
[34]	Riopelle K., Leydesdorff L., & Jie L. (2014). How to Create an Overlay Map of Science Using the Web of Science. Retrieved from .
[35]	Stirling A. (2007). A general framework for analysing diversity in science, technology and society. Journal of the Royal Society Interface, 4(15), 707-719.
[36]	Wagner C.S., Roessner J.D., Bobb K., Klein J.T., Boyack K.W., Keyton J., Rafols I., & B?rner K. (2011). Approaches to understanding and measuring interdisciplinary scientific research (IDR): A review of the literature. Journal of Informetrics, 5(1), 14-26.
[37]	Wang J., Thijs B., & Gl?nzel W. (2014. Interdisciplinarity and impact: Distinct effects of variety, balance and disparity (December 22, 2014). Retrieved from 2014). Interdisciplinarity and impact: Distinct effects of variety, balance and disparity (December 22, 2014). Retrieved from or .
[38]	Yegros-Yegros A., Amat C. B., d’Este P., Porter A.L., & Rafols I. (2010). Does interdisciplinary research lead to higher scientific impact? In Science and Technology Indicators (STI) Conference, September 8-11, Leiden, the Netherlands.
[39]	Yegros-Yegros A., Rafols I., & d’Este P. (2015). Does interdisciplinary research lead to higher citation impact? The different effect of proximal and distal interdisciplinarity, PLOS ONE, 10(8) . Retrieved from .
[40]	Youtie J., Porter A.L., Shapira P., Tang L., & Benn T. (2011). The use of environmental, health and safety research in nanotechnology research. Journal of Nanoscience and Nanotechnology, 11(1), 158-166.
[41]	Youtie J., Solomon G.E.A., Carley S., Kwon S., & Porter A.L. (2017). Crossing borders: A citation analysis of connections between Cognitive Science and Educational research and the fields in between. Research, 26(3), 242-255.
[42]	Zhang J., Ning Z., Bai X., Kong X., Zhou J., & Xia F. (2017). Exploring time factors in measuring the scientific impact of scholars. Scientometrics, 112(3), 1301-1321.
[43]	Zhang L., Rousseau R., & Gl?nzel W. (2016). Diversity of references as an indicator for interdisciplinarity of journals: Taking similarity between subject fields into account. Journal of the American Society for Information Science and Technology, 67(5), 1257-1265.