Please wait a minute...
Journal of Data and Information Science  2018, Vol. 3 Issue (1): 54-81    DOI: 10.2478/jdis-2018-0004
Research Paper     
Measuring and Visualizing Research Collaboration and Productivity
Garner Jon1,L. Porter Alan2,(),Leidolf Andreas3,Baker Michelle3,4
1Search Technology, Inc., Norcross, GA, USA
2School of Public Policy, Georgia Institute of Technology, Atlanta, USA
3Ecology Center, Utah State University, Logan, UT, USA
4Department of Biology, Utah State University, Logan, UT, USA
Download: PDF (10967 KB)      HTML  
Export: BibTeX | EndNote (RIS)      

Abstract  

Purpose: This paper presents findings of a quasi-experimental assessment to gauge the research productivity and degree of interdisciplinarity of research center outputs. Of special interest, we share an enriched visualization of research co-authoring patterns.

Design/methodology/approach: We compile publications by 45 researchers in each of 1) the iUTAH project, which we consider here to be analogous to a “research center,” 2) CG1— a comparison group of participants in two other Utah environmental research centers, and 3) CG2—a comparison group of Utah university environmental researchers not associated with a research center. We draw bibliometric data from Web of Science and from Google Scholar. We gather publications for a period before iUTAH had been established (2010-2012) and a period after (2014-2016). We compare these research outputs in terms of publications and citations thereto. We also measure interdisciplinarity using Integration scoring and generate science overlay maps to locate the research publications across disciplines.

Findings: We find that participation in the iUTAH project appears to increase research outputs (publications in the After period) and increase research citation rates relative to the comparison group researchers (although CG1 research remains most cited, as it was in the Before period). Most notably, participation in iUTAH markedly increases co-authoring among researchers—in general; and for junior, as well as senior, faculty; for men and women: across organizations; and across disciplines.

Research limitations: The quasi-experimental design necessarily generates suggestive, not definitively causal, findings because of the imperfect controls.

Practical implications: This study demonstrates a viable approach for research assessment of a center or program for which random assignment of control groups is not possible. It illustrates use of bibliometric indicators to inform R&D program management.

Originality/value: New visualizations of researcher collaboration provide compelling comparisons of the extent and nature of social networking among target cohorts.



Key wordsBibliometrics      Interdisciplinarity      Multidisciplinarity      Research evaluation      Research collaboration mapping      Science visualization      Scientometrics      Social network analysis     
Published: 19 March 2018
Corresponding Authors: L. Porter Alan     E-mail: alan.porter@isye.gatech.edu
Cite this article:

Garner Jon,L. Porter Alan,Leidolf Andreas,Baker Michelle. Measuring and Visualizing Research Collaboration and Productivity. Journal of Data and Information Science, 2018, 3(1): 54-81.

URL:

http://manu47.magtech.com.cn/Jwk3_jdis/10.2478/jdis-2018-0004     OR     http://manu47.magtech.com.cn/Jwk3_jdis/Y2018/V3/I1/54

Publications 2010-2012 2014-2016
iUtah CG1 CG2 iUtah CG1 CG2
# of group authors with papers 45 44 40 45 45 36
Total Records 702 636 293 788 666 295
Total from GS 431 367 166 407 337 165
Total from WoS 271 269 127 381 329 130
Average Times Cited 9.55 12.81 10.02 2.68 3.65 2.60
Average Times Cited GS 6.49 7.33 6.62 1.13 4.06 1.75
Average Times Cited WoS 14.42 20.49 14.46 3.81 5.13 3.52
Median Times Cited 2 3 1 0 0 0
Median Times Cited GS 0 0 0 0 0 0
Median Times Cited WoS 9 9 4 2 2 1
Cites/Year WoS 2.43 3.47 2.35 2.39 3.08 1.92
H-Index 35 42 22 19 22 14
H-Index GS 26 26 15 13 11 10
H-Index WoS 29 36 18 17 20 12
Integration score 0.535 0.457 0.428 0.523 0.489 0.468
Table 1 Publication and citation metrics of authors in three groups of environmental researchers from Utah, 2010-2016.
Figure 1. Publications indexed by Web of Science for authors in three groups of researchers from Utah, 2010-2016.
Figure 2. Average times cited per year since publication (based on Web of Science) for authors in three groups of researchers from Utah, 2010-2016.
Figure 3. iUTAH publications overlaid on a science map based on Web of Science categories, 2014-2016.
Figure 5. Co-Author map of iUTAH researchers for the Before period (2010-2012), separated by institution and discipline.
Figure 6. Co-Author map of iUTAH researchers for the After period (2014-2016), separated by institution and discipline.
iUTAH CG1
Before After Before After
Average degree 1.2 5.422 0.133 0.178
Density 0.027 0.123 0.003 0.004
Total links 27 122 3 4
Links within discipline 12 63 3 3
Links across discipline 15 59 0 1
Links within rank 12 48
Links across rank 15 74
Links within gender 17 63
Links across gender 10 59
Links within university 24 14
Links across university 3 108
Table 2 Networking statistics within Utah researcher cohorts.
Figure 7. Co-Author map of CG1 researchers for the Before period (2010-2012), separated by institution and discipline.
Figure 8. Co-Author map of CG1 researchers for the After period (2014-2016), separated by institution and discipline.
[1]   Campbell D.,& Stanley ,J.(1963). Experimental and quasi-experimental designs for research. Chicago: Rand-McNally.
[2]   Cook T., &Campbell ,D.A. (1979). Quasi-experimentation. New York:Houghton Mifflin.
[3]   Carley S.,& Porter ,A.L.(2012). A forward diversity index. Scientometrics, 90(2), 407-427.
doi: 10.1007/s11192-011-0528-1
[4]   Carley S., Porter A.L., Rafols I., & Leydesdorff, L.(under review). Visualization of disciplinary profiles: Enhanced science overlay maps. Journal of Data and Information Science, 2017(3), 68-111.
doi: 10.1515/jdis-2017-0015
[5]   De Bellis,N. (2009).Bibliometrics and citation analysis:From the science citation index to cybermetrics, Lanham, MD: Scarecrow Press From the science citation index to cybermetrics,Lanham, MD: Scarecrow Press.
[6]   de Nooy,W., Mrvar , A., & Batgelj V. (2011). Exploratory social network analysis with Pajek (2nd Edition) .New York,NY: Cambridge University Press.
[7]   Garner J., Porter A.L., Borrego M., Tran E., & Teutonico R. (2013). Facilitating social and natural science cross-disciplinarity: Assessing the human and social dynamics program, Research Evaluation, 22(2), 134-144.
doi: 10.1093/reseval/rvt001
[8]   Garner J., Porter A.L., & Newman N.C. (2014). Distance and velocity measures: Using citations to determine breadth and speed of research impact. Scientometrics, 100(3), 687-703.
doi: 10.1007/s11192-014-1316-5
[9]   Garner J., Porter A.L., Newman N.C., & Crowl T.A. (2012). Assessing research network and disciplinary engagement changes induced by an NSF program. Research Evaluation, 21(2), 89-104.
doi: 10.1093/reseval/rvs004
[10]   Gl?nzel W.,& Schubert ,A.(2003). A new classification scheme of science fields and subfields designed for scientometric evaluation purposes. Scientometrics, 56(3), 357-367.
doi: 10.1023/A:1022378804087
[11]   Hall K.L., Stokols D., Stipelman B.A., Vogel A.L., Feng A., Masimore B., Morgan G., Moser R.P., Marcus S.E., & Berrigan D. (2012). Assessing the value of team science: A study comparing center- and investigator-initiated grants. American Journal of Preventive Medicine, 42(2), 157-163.
doi: 10.1016/j.amepre.2011.10.011
[12]   Hirsch J.E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102(46), 16569-16572.
doi: 10.1073/pnas.0507655102
[13]   Klavans R.,& Boyack ,K.W.(2009). Toward a consensus map of science. Journal of the American Society for Information Science and Technology, 60(3), 455-476.
doi: 10.1002/asi.v60:3
[14]   Klavans R.,& Boyack ,K.W.(2017). Which type of citation analysis generates the most accurate taxonomy of scientific and technical knowledge? Journal of the American Society for Information Science and Technology, 68(4), 984-998.
doi: 10.1002/asi.23734
[15]   Kwon S.,Solomon G.E.A., Youtie , J., & Porter, A.L.(under review). A measure of interdisciplinary knowledge flow between specific fields: Implications for impact and funding . PLoS One.
[16]   Leydesdorff L. (2008). On the normalization and visualization of author co-citation data: Salton’s Cosine versus the Jaccard Index. Journal of the American Society for Information Science and Technology, 59(1), 77-85.
doi: 10.1002/asi.20732
[17]   Leydesdorff L.,& Bornmann ,L.(2016). The operationalization of “fields” as WoS subject categories (WCs) in evaluative bibliometrics: The cases of “library and information science” and “science & technology studies.” Journal of the American Society for Information Science and Technology, 67(3), 707-714.
doi: 10.1002/asi.23408
[18]   Leydesdorff L., Carley S., & Rafols I. (2013). Global maps of science based on the new Web-of-Science Categories. Scientometrics, 94(2), 589-593.
doi: 10.1007/s11192-012-0784-8 pmid: 23335826
[19]   Leydesdorff L.,& Rafols ,I.(2009). A global map of science based on the ISI subject categories. Journal of the American Society for Information Science and Technology, 60(2), 348-362.
doi: 10.1002/asi.20967
[20]   National Academies Committee on Facilitating Interdisciplinary Research, Committee on Science,Engineering and Public Policy(COSEPUPx) (2005). Facilitating Interdisciplinary Research. Washington,DC: National Academies Press.
[21]   Peck L.R. (2016), Social experiments in practice: The what, why, when, where, and how of Experimental Design & Analysis. New Directions for Evaluation, No. 152 (Winter), New York: Wiley.
[22]   Porter A.L., Cohen A.S., Roessner J.D., & Perreault M. (2007). Measuring researcher interdisciplinarity. Scientometrics, 72(1), 117-147.
doi: 10.1007/s11192-007-1700-5
[23]   Porter A.L., Garner J., & Crowl T. (2012). The RCN (Research Coordination Network) experiment: Can we build new research networks? BioScience, 62, 282-288.
doi: 10.1525/bio.2012.62.3.9
[24]   Porter A.L., Newman N.C., Myers W., & Schoeneck D. (2003). Projects and publications: Interesting patterns in U.S. Environmental Protection Agency research.Research Evaluation, 12(3), 171-182.
doi: 10.3152/147154403781776609
[25]   Porter A.L.,& Rafols ,I.(2009). Is science becoming more interdisciplinary? Measuring and mapping six research fields over time. Scientometrics, 81(3), 719-745.
doi: 10.1007/s11192-008-2197-2
[26]   Porter A.L., Roessner J.D., & Heberger A.E. (2008). How interdisciplinary is a given body of research? Research Evaluation, 17(4), 273-282.
doi: 10.3152/095820208X364553
[27]   Porter A.L., Schoeneck D.J., & Carley S. (2013). Measuring the extent to which a research domain is self-contained. In Proceedings of the 14th International Conference on Scientometrics and Informetrics (ISSI2013), July 15-19, Vienna, Austria.
[28]   Porter A.L., Schoeneck D.J., Roessner D., & Garner J. (2010). Practical research proposal and publication profiling. Research Evaluation, 19(1), 29-44.
doi: 10.3152/095820210X492512
[29]   Porter A.L., Schoeneck D.J., Solomon G., Lakhani H., & Dietz J. (2013). Measuring and mapping interdisciplinarity: Research & evaluation on education in science & engineering (“REESE”) and STEM. In American Education Research Association Annual MeetingApril 27-May 1, San Francisco.
[30]   Rafols I. (2014). Knowledge integration and diffusion: Measures and mapping of diversity and coherence. In Ding, Y., Rousseau, R., & Wolfram, D.(Eds.) Measuring scholarly Impact: Methods and Practice (pp. 169-190). Berlin: Springer.
doi: 10.1007/978-3-319-10377-8_8
[31]   Rafols I.,& Leydesdorff ,L.(2009). Content-based and algorithmic classifications of journals: Perspectives on the dynamics of scientific communication and indexer effects. Journal of the American Society for Information Science and Technology, 60(9), 1823-1835.
doi: 10.1002/asi.v60:9
[32]   Rafols I.,& Meyer ,M.(2010). Diversity and network coherence as indicators of interdisciplinarity: Case studies in bionanoscience. Scientometrics, 82(2), 263-287.
doi: 10.1007/s11192-009-0041-y
[33]   Rafols I., Porter A., & Leydesdorff L. (2010). Science overlay maps: A new tool for research policy and library management. Journal of the American Society for Information Science and Technology, 61(9), 1871-1887.
doi: 10.1002/asi.21368
[34]   Riopelle K., Leydesdorff L., & Jie L. (2014). How to Create an Overlay Map of Science Using the Web of Science. Retrieved from .
[35]   Stirling A. (2007). A general framework for analysing diversity in science, technology and society. Journal of the Royal Society Interface, 4(15), 707-719.
doi: 10.1098/rsif.2007.0213 pmid: 17327202
[36]   Wagner C.S., Roessner J.D., Bobb K., Klein J.T., Boyack K.W., Keyton J., Rafols I., & B?rner K. (2011). Approaches to understanding and measuring interdisciplinary scientific research (IDR): A review of the literature. Journal of Informetrics, 5(1), 14-26.
doi: 10.1016/j.joi.2010.06.004
[37]   Wang J., Thijs B., & Gl?nzel W. (2014. Interdisciplinarity and impact: Distinct effects of variety, balance and disparity (December 22, 2014). Retrieved from 2014). Interdisciplinarity and impact: Distinct effects of variety, balance and disparity (December 22, 2014). Retrieved from or .
[38]   Yegros-Yegros A., Amat C. B., d’Este P., Porter A.L., & Rafols I. (2010). Does interdisciplinary research lead to higher scientific impact? In Science and Technology Indicators (STI) Conference, September 8-11, Leiden, the Netherlands.
[39]   Yegros-Yegros A., Rafols I., & d’Este P. (2015). Does interdisciplinary research lead to higher citation impact? The different effect of proximal and distal interdisciplinarity, PLOS ONE, 10(8) . Retrieved from .
[40]   Youtie J., Porter A.L., Shapira P., Tang L., & Benn T. (2011). The use of environmental, health and safety research in nanotechnology research. Journal of Nanoscience and Nanotechnology, 11(1), 158-166.
doi: 10.1166/jnn.2011.3840 pmid: 21446421
[41]   Youtie J., Solomon G.E.A., Carley S., Kwon S., & Porter A.L. (2017). Crossing borders: A citation analysis of connections between Cognitive Science and Educational research and the fields in between. Research, 26(3), 242-255.
[42]   Zhang J., Ning Z., Bai X., Kong X., Zhou J., & Xia F. (2017). Exploring time factors in measuring the scientific impact of scholars. Scientometrics, 112(3), 1301-1321.
doi: 10.1007/s11192-017-2458-z
[1] Cinzia Daraio. A Framework for the Assessment of Research and Its Impacts[J]. Journal of Data and Information Science, 2017, 2(4): 7-42.
[2] Stephen Carley, Alan L. Porter, Ismael Rafols, Loet Leydesdorff. Visualization of Disciplinary Profiles: Enhanced Science Overlay Maps[J]. Journal of Data and Information Science, 2017, 2(3): 68-111.
[3] Luan Chunjuan, Porter Alan L.. Insight into the Disciplinary Structure of Nanoscience & Nanotechnology[J]. Journal of Data and Information Science, 2017, 2(1): 70-88.
[4] Zhao Dangzhi, Cappello Alicia, Johnston Lucinda. Functions of Uni- and Multi-citations: Implications for Weighted Citation Analysis[J]. Journal of Data and Information Science, 2017, 2(1): 51-69.
[5] van Raan Anthony F.J.. Patent Citations Analysis and Its Value in Research Evaluation: A Review and a New Approach to Map Technology-relevant Research[J]. Journal of Data and Information Science, 2017, 2(1): 13-50.
[6] Frederik T. Verleysen, Arie Weeren. Mapping Diversity of Publication Patterns in the Social Sciences and Humanities: An Approach Making Use of Fuzzy Cluster Analysis[J]. Journal of Data and Information Science, 2016, 1(4): 33-59.
[7] Yuqing Mao & Zhiyong Lu. Mining Related Articles for Automatic Journal Cataloging[J]. Journal of Data and Information Science, 2016, 1(2): 45-59.
[8] Fred Y. Ye, Mu-Hsuan Huang & Dar-Zen Chen. Comparative Study of Trace Metrics between Bibliometrics and Patentometrics[J]. Journal of Data and Information Science, 2016, 1(2): 13-31.
[9] Yating LI & Bin ZHANG. A comparative study on institutional collaboration characteristics in EM and LIS in China[J]. Journal of Data and Information Science, 2015, 7(4): 62-77.
[10] Simon S. LI & Fred Y. YE. Evaluating journals' yearly impact with altmetric indicators[J]. Journal of Data and Information Science, 2015, 7(2): 25-38.
[11] Xian ZHANG, Haiyun XU, Shu FANG, Zhengyin HU & Shuying LI. Building potential patent portfolios: An integrated approach based on topic identification and correlation analysis[J]. Journal of Data and Information Science, 2015, 7(2): 39-51.
[12] Lihua ZHANG & Zhiqiang ZHANG. Detection of early-stage research fronts—An example of complex networks research[J]. Journal of Data and Information Science, 2014, 7(4): 77-94.
[13] Chunlei YE. Mapping the evolution of research topics using ATM and SNA[J]. Journal of Data and Information Science, 2014, 7(4): 46-62.
[14] Yang ZHANG & Wanyang LING. A comparative study of information diffusion in weblogs and microblogs based on social network analysis[J]. Journal of Data and Information Science, 2012, 5(4): 51-66.
[15] Junping QIU, Feifei WANG & Houqiang YU. A case study on knowledge communication based on friends-list links in the science blogging community at Sciencenet.cn[J]. Journal of Data and Information Science, 2011, 4(3/4): 49-67.