A Framework for the Assessment of Research and Its Impacts

doi:10.1515/jdis-2017-0018

Abstract

Abstract:

This paper proposes a holistic framework for the development of models for the assessment of research activities and their impacts. It distinguishes three dimensions, including in an original way, data as a main dimension, together with theory andmethodology. Each dimension of the framework is further characterized by three main building blocks: education, research, and innovation (theory); efficiency, effectiveness, and impact (methodology); and availability, interoperability, and “unit-free” property (data). The different dimensions and their nine constituent building blocks are attributes of an overarching concept, denoted as “quality.” Three additional quality attributes are identified as implementation factors (tailorability, transparency, and openness) and three “enabling” conditions (convergence, mixed methods, and knowledge infrastructures) complete the framework. A framework is required to develop models of metrics. Models of metrics are necessary to assess the meaning, validity, and robustness of metrics. The proposed framework can be a useful reference for the development of the ethics of research evaluation. It can act as a common denominator for different analytical levels and relevant aspects and is able to embrace many different and heterogeneous streams of literature. Directions for future research are provided.

Key words: Evaluation of research, Efficiency, Effectiveness, Impacts, Modeling, Responsible metrics, Ethics of research evaluation

Cinzia Daraio. "A Framework for the Assessment of Research and Its Impacts." Journal of Data and Information Science, vol.2, no.4, 2017, pp.7-42. DOI: 10.1515/jdis-2017-0018

References

[1]	Abramovitz M. (1956). Resource and output trends in the United States since 1870. The American Economic Review, 46(2), 5-23.
[2]	Acemoglu D., Aghion p., and Zilibotti F. (2003). Vertical integration and distance to frontier. Journal of the European Economic Association, 1(2-3), 630-638.
[3]	Acemoglu D., Aghion p., and Zilibotti F. (2006). Distance to frontier, selection, and economic growth. Journal of the European Economic Association, 4(1), 37-74.
[4]	Adams ,J.D. (1990). Fundamental stocks of knowledge and productivity growth. Journal of political Economy, 98(4), 673-702.
[5]	Agarwal ,R., and Dhar ,V. (2014). Editorial big data, data science, and analytics: The opportunity and challenge for IS research. Information Systems Research, 25(3), 443-448.
[6]	Aghion ,p. (2009). Growth and education. Commission on growth and development working paper, no. 56. Washington, DC: World Bank publications.
[7]	Aghion p., and Howitt, p. (2009). The economics of growth. Cambridge, MA: The MIT press.
[8]	Aghion p., David p.A., and Foray D. (2009). Science, technology and innovation for economic growth: Linking policy research and practice in ‘STIG Systems’. Research policy, 38(4), 681-693.
[9]	Anderson C. (2008). The end of theory: The data deluge makes the scientific method obsolete. Wired Magazine. Retrieved on July 30, 2017, from https://www.wired.com/2008/06/pb-theory/.
[10]	Antonelli ,C., and Link ,A.N. (2014). Routledge handbook of the economics of knowledge. London: Routledge.
[11]	Audretsch D.B., Bozeman B., Combs K.L., Feldman M., Link A.N., Siegel D.S., .. and Wessner C. (2002). The economics of science and technology. The Journal of Technology Transfer, 27(2), 155-203.
[12]	B?din L., Daraio C., and Simar L. (2012). How to measure the impact of environmental factors in a nonparametric production model. European Journal of Operational Research, 223(3), 818-833.
[13]	B?din L., Daraio C., and Simar L. (2014). Explaining inefficiency in nonparametric production models: The state of the art. Annals of Operations Research, 214(1), 5-30.
[14]	Ballard B.W.(2000). Understanding MacIntyre. Lanham, MD: University press of America.
[15]	Bammer ,G. (2016. What constitutes appropriate peer review for interdisciplinary research? palgrave Communications, 2 (palcomms201617). Retrieved on July 30, 2017, from 2016). What constitutes appropriate peer review for interdisciplinary research? palgrave Communications, 2 (palcomms201617). Retrieved on July 30, 2017, from .
[16]	Barré ,R. (2004). S and T indicators for policy making in a changing science-society relationship. In H. Moed, W. Gl?nzel, and U. Schmoch (Eds.), Handbook of Quantitative Science and Technology Research (pp. 115-132). Dordrecht: Springer Netherlands.
[17]	Batini C., and Scannapieco, M. (2016). Data and information quality. Cham, Switzerland: Springer International publishing.
[18]	Benessia A., Funtowicz S., Giampietro M., pereira ?.G., Ravetz J., Saltelli A.,. and van der Sluijs , J.p. (2016). Science on the verge. Tempe, AZ: Consortium for Science, policy, and Outcomes at Arizona State University.
[19]	Blaug, M. (1966. Economics of education; a selected annotated bibliography (No. 370.193 B5). Retrieved on July 30, 2017, from 1966). Economics of education; a selected annotated bibliography (No. 370.193 B5). Retrieved on July 30, 2017, from .
[20]	Bogetoft p., Fried H.O., and Eeckaut, p.V. (2007). The university benchmarker: An interactive computer approach. In A. Bonaccorsi, and C. Daraio (Eds.), Universities and Strategic Knowledge Creation: Specialization and performance in Europe (pp. 443-462). Cheltenham: Edward Elgar publishing.
[21]	Bonaccorsi, A.,, and Daraio ,C., (2004). Econometric approaches to the analysis of productivity of RD systems. production functions and production frontiers. In H.F. Moed, W. Gl?nzel, and U. Schmoch (Eds.), Handbook of Quantitative Science and Technology Research (pp. 51-74). Dordrecht: Springer Netherlands.
[22]	Borgman C.L.(2015). Big data, little data, no data: Scholarship in the networked world. Cambridge, MA: The MIT press.
[23]	Bornmann, L. (2013). What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society for Information Science and Technology, 64(2), 217-233.
[24]	Borrás, S., and Edquist ,C. (2013). The choice of innovation policy instruments. Technological Forecasting and Social Change, 80(8), 1513-1522.
[25]	Bozeman, B., and Melkers , J.(Eds). (1993). Evaluating R and D impacts: Methods and practice. New York: Springer.
[26]	Brunsson N., and Jacobsson, B. (Eds.). (2002a). A world of standards. Oxford: Oxford University press.
[27]	Brunsson N., and Jacobsson, B. (2002b). The pros and cons of standardization—An epilogue. In A World of Standards (pp. 169-173). Oxford: Oxford University press.
[28]	Capano G., Howlett M., and Ramesh M. (Eds.). (2015). Varieties of governance. Hampshire, UK: palgrave Macmillan.
[29]	Carayannis E.G., and Campbell ,D.F. (2009). ‘Mode 3’ and ‘Quadruple Helix’: Toward a 21st century fractal innovation ecosystem. International Journal of Technology Management, 46(3-4), 201-234.
[30]	Checchi, D. (2006).The economics of education:Human capital, family background and inequality. Cambridge: Cambridge University press.
[31]	Chen C.M., Delmas M.A., and Lieberman M.B. (2015). production frontier methodologies and efficiency as a performance measure in strategic management research. Strategic Management Journal, 36(1), 19-36.
[32]	Chesbrough, H. (2012). Open innovation: Where we’ve been and where we’re going. Research-Technology Management, 55(4), 20-27.
[33]	Cohen, W.M., and Levinthal ,D.A. (1990). Absorptive capacity: A new perspective on learning and innovation. Administrative Science Quarterly, 35(1), 128-152.
[34]	Creswell, J.W., and Clark, V.L.p. (Eds.). (2011). 2nd Ed. Designing and conducting mixed methods research. Thousand Oaks, CA: SAGE publications.
[35]	Cronin B. (2013). Thinking about data. Journal of the American Society for Information Science and Technology, 64(3), 435-436.
[36]	Cronin B., and Sugimoto, C.R. (Eds.). (2014).Beyond bibliometrics: Harnessing multidimensional indicators of scholarly impact Cambridge , MA: The MIT press.
[37]	Cronin B., and Sugimoto ,C.R. (Eds.). (2015). Scholarly metrics under the microscope: From citation analysis to academic auditing. Medford, NJ: Information Today.
[38]	D’Ariano G.M., and perinotti, p. (2016). Quantum theory is an information theory. Foundations of physics, 46(3), 269-281.
[39]	Dahler-Larsen, p.(2012). The evaluation society. California: Stanford University press.
[40]	Daraio, C.(2015). Assessing the efficiency, effectiveness and impact of education in the age of big data: Challenges and a way forward. Keynote presentation at Leuven LEER Workshop ‘Efficiency in Education and the Use of Big Data’, November 19-20, 2015, Leuven (Belgium).
[41]	Daraio, C. (2017a). Assessing research and its impacts: The generalized implementation problem and a doubly-conditional performance evaluation model, paper presented at the ISSI 2017 Conference, October 2017, Wuhan (China).
[42]	Daraio, ,C. (2017b). A doubly conditional performance evaluation model, the democratization of evaluation and altmetrics, paper presented at the STI 2017 Conference, September 2017, paris.
[43]	Daraio C.(in press). Econometric approaches to the measurement of research productivity.In W. Gl?nzel, H.F. Moed, H. Schmoch, and M. Thelwall (Eds.), Springer Handbook of Science and Technology Indicators.
[44]	Daraio, C., and Simar ,L. (2014). Directional distances and their robust versions: Computational and testing issues. European Journal of Operational Research, 237(1), 358-369.
[45]	Daraio, C., and Simar ,L. (2007). Advanced robust and nonparametric methods in efficiency analysis. Methodology and applications. New York: Springer.
[46]	Daraio C., Simar L., and Wilson p.W. (2017). Central limit theorems for conditional efficiency measures and tests of the “separability” condition in nonparametric two-stage models of production. The Econometrics Journal. Retrieved on July 30, 2017, from 1111/ectj.12103.
[47]	Daraio, C., and Gl?nzel ,W. (2016). Grand challenges in data integration—state of the art and future perspectives: An introduction. Scientometrics, 108(1), 391-400.
[48]	Daraio C., Lenzerini M., Leporelli C., Moed H.F., Naggar p., Bonaccorsi A., and Bartolucci A. (2016a). Data integration for research and innovation policy: An ontology-based data management approach. Scientometrics, 106(2), 857-871.
[49]	Daraio C., Lenzerini M., Leporelli C., Naggar p., Bonaccorsi A., and Bartolucci A. (2016b). The advantages of an Ontology-Based Data Management approach: Openness, interoperability and data quality. Scientometrics, 108(1), 441-455.
[50]	Debackere, K. (2016). Let the data speak for themselves: Opportunities and caveats. Journal of Data and Information Science, 1(1), 3-5.
[51]	DeMillo, R.A., and Young, A.J. (2015).Revolution in higher education:How a small band of innovators will make college accessible and affordable.Cambridge, MA: The MIT press.
[52]	Ding Y., Rousseau R., and Wolfram, D. (Eds). (2014). Measuring scholarly impact. Cham, Switzerland: Springer International publishing.
[53]	Ding, Y., and Stirling ,K. (2016). Data-driven discovery: A new era of exploiting the literature and data. Journal of Data and Information Science, 1(4), 1-9.
[54]	Drucker, p.F. (1967). The effective executive. New York: Harper and Row.
[55]	Ebrahim, A., and Rangan ,V.K. (2014). What impact? California Management Review, 56(3), 118-141.
[56]	Edquist, C. (2001). The systems of innovation approach and innovation policy: An account of the state of the art. In Druid Nelson and Winter Conference 2001 (pp. 12-15). Denmark: Aalborg University.
[57]	Edwards p.N., Jackson S.J., Chalmers M.K., Bowker G.C., Borgman C.L., Ribes D., ... and Calvert S. (2013. Knowledge infrastructures: Intellectual frameworks and research challenges (p. 40). Ann Arbor, MI: University of Michigan. Retrieved on July 30, 2017, from 2013). Knowledge infrastructures: Intellectual frameworks and research challenges (p. 40). Ann Arbor, MI: University of Michigan. Retrieved on July 30, 2017, from .
[58]	Egghe, L., and Rousseau ,R. (1990). Introduction to informetrics: Quantitative methods in library, documentation and information science. Amsterdam: Elsevier.
[59]	Egghe, L. (2005). power laws in the information production process: Lotkaian informetrics. Amsterdam: Elsevier.
[60]	Ekbia H., Mattioli M., Kouper I., Arave G., Ghazinejad A., Bowman T., .. and Sugimoto C.R. (2015). Big data, bigger dilemmas: A critical review. Journal of the Association for Information Science and Technology, 66(8), 1523-1545
[61]	Etzkowitz, H., and Leydesdorff ,L. (2000). The dynamics of innovation: From national systems and mode 2 to a triple helix of university-industry-government relations. Research policy, 29(2), 109-123.
[62]	European Commission (2014). Expert Group to support the development of tailor-made impact assessment methodologies for ERA (European Research Area), Brussels, Belgium.
[63]	Fagerberg J., Martin B.R., and Andersen, E.S. (Eds.). (2013). Innovation studies: Evolution and future challenges Oxford: Oxford University press Evolution and future challenges. Oxford: Oxford University press.
[64]	Fealing K.H., Lane J.I.,Marburger, J.H. JIII, and Shipp,S.S. (Eds.) (2011). The science of science policy, a handbook. Stanford: Stanford University press.
[65]	Floridi, L. (Ed.). (2008). The Blackwell guide to the philosophy of computing and information. Hoboken, NJ: John Wiley and Sons.
[66]	Floridi, L. (Ed.). (2010). The Cambridge handbook of information and computer ethics. Cambridge: Cambridge University press.
[67]	Floridi, L. (2012). The road to the philosophy of information. In H. Demir (Eds.), Luciano Floridi’s philosophy of Technology (pp. 245-271). Dordrecht: Springer Netherlands.
[68]	Floridi, L. (2014). The fourth revolution: How the infosphere is reshaping human reality. Oxford: Oxford University press.
[69]	Floridi L., and Illari, p. (Eds.). (2014). The philosophy of information quality (Vol. 358). Cham, Switzerland: Springer International publishing.
[70]	FrickéM. (2015). Big data and its epistemology. Journal of the Association for Information Science and Technology, 66(4), 651-661.
[71]	Frischmann B.M.(2012).Infrastructure: The social value of shared resources.Oxford: Oxford University press.
[72]	Funtowicz, S.O., and Ravetz ,J.R. (1990). Science for policy: Uncertainty and quality. In Uncertainty and Quality in Science for policy (pp. 7-16). Dordrecht: Springer Netherlands.
[73]	Furner, J. (2014). The ethics of evaluative bibliometrics. In B. Cronin, and C. Sugimoto (Eds.), Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact (pp. 85-107). Cambridge, MA: MIT press.
[74]	Galán J.E., Veiga H., and Wiper M.p. (2014). Bayesian estimation of inefficiency heterogeneity in stochastic frontier models. Journal of productivity Analysis, 42(1), 85-101.
[75]	Georgescu-Roegen, N.(1971). The entropy law and the economic process. Cambridge, MA: Harvard University press.
[76]	Gibbons M., Limoges C., Nowotny H., Schwartzman S., Scott p., and Trow M. (1994). The new production of knowledge: The dynamics of science and research in contemporary societies. London: Sage.
[77]	Gingras, Y. (2016). Bibliometrics and research evaluation: Uses and abuses. Cambridge, MA: The MIT press.
[78]	Gl?nzel, W. (1996). The need for standards in bibliometric research and technology. Scientometrics, 35(2), 167-176.
[79]	Gl?nzel, W., and Schoepflin ,U. (1994). Little scientometrics, big scientometrics.and beyond? Scientometrics, 30(2-3), 375-384.
[80]	Gl?nzel, W. (2010). On reliability and robustness of scientometrics indicators based on stochastic models. An evidence-based opinion paper. Journal of Informetrics, 4(3), 313-319.
[81]	Gl?nzel, W. , and Moed ,H.F. (2013). Opinion paper: Thoughts and facts on bibliometric indicators. Scientometrics, 96(1), 381-394.
[82]	Gl?nzel W., Moed H.F., Schmoch H., and Thelwall, M.(Eds).Science and Technology Indicators.
[83]	Godin, B. (2002). Outline for a history of science measurement. Science, Technology, and Human Values, 27(1), 3-27.
[84]	Godin, B. (2004). Measurement and statistics on science and technology: 1920 to the present. London: Routledge.
[85]	Griliches, Z. (1986). Economic data issues. In Z. Griliches, and M.D. Intriligator (Eds.), Handbook of Econometrics Volume III (pp. 1465-1514). Amsterdam: Elsevier.
[86]	Griliches, Z. (1994). productivity, R and D, and the data constraint. American Economic Review, 84(1), 1-23.
[87]	Griliches, Z. (1998). R and D and productivity: The econometric evidence. Chicago: University of Chicago press.
[88]	Hall, B.H. , and Rosenberg ,N. (Eds.). (2010). Handbook of the economics of innovation. Amsterdam: Elsevier.
[89]	Hanson B., Sugden A., and Alberts B. (2011). Making data maximally available. Science, 331(6018), 649-649.
[90]	Hanushek, E.A., and Woessmann ,L. (2007). The role of education quality for economic growth. World Bank policy Research Working paper, No. 4122. Retrieved on July 30, 2017, from https://ssrn.com/abstract=960379.
[91]	Hanushek E.A., Machin S.J., and Woessmann L. (Eds.). (2016). Handbook of the economics of education. Amsterdam: Elsevier.
[92]	Hanushek E.A., Woessmann L., Jamison E.A., and Jamison D.T. (2008. Education and economic growth. Education Next, 8(2). Retrieved on July 31, 2017, from 2008). Education and economic growth. Education Next, 8(2). Retrieved on July 31, 2017, from .
[93]	Helbing, D., and Carbone ,A.F. (Eds.). (2012). participatory science and computing for our complex world. The European physical Journal. Special Topics, Vol. 214. Retrieved on July 31, 2017, from https://epjst.epj.org/index.php?option=com_toc and url=/articles/epjst/abs/2012/14/contents/contents.html.
[94]	Hemlin, S. (1996). Research on research evaluation. Social Epistemology, 10(2), 209-250.
[95]	Henderson R., Jaffe A.B., and Trajtenberg M. (1998). Universities as a source of commercial technology: A detailed analysis of university patenting, 196588. Review of Economics and Statistics, 80(1), 119-127.
[96]	Hicks D., Wouters p., Waltman L., De Rijcke S., and Rafols I. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature, 520, 429-431.
[97]	Hill, S. (2016. Assessing (for) impact: Future assessment of the societal impact of research. palgrave Communications. Retrieved on July 31, 2017, from 2016). Assessing (for) impact: Future assessment of the societal impact of research. palgrave Communications. Retrieved on July 31, 2017, from .
[98]	Hinrichs-Krapels, S., and Grant ,J. (2016). Exploring the effectiveness, efficiency and equity (3es) of research and research impact assessment. palgrave Communications. Retrieved on July 31, 2017, from https://www.nature.com/articles/palcomms201690#t2.
[99]	Horstemeyer, M.F. (2009). Multiscale modeling: A review. In J. Leszczynski, and M. Shukla (Eds.), practical Aspects of Computational Chemistry (pp. 87-135). Dordrecht: Springer Netherlands.
[100]	ISCED . (2011, International Standard Classification of Education, UNESCO Montreal, Canada. Retrieved on December 20, 2016, from 2011), International Standard Classification of Education, UNESCO Montreal, Canada. Retrieved on December 20, 2016, from .
[101]	Johnes, G., and Johnes J. (Eds.). (2004). International handbook on the economics of education. Cheltenham: Edward Elgar.
[102]	Khandker S.R., Koolwal G.B., and Samad H.A. (2010). Handbook on impact evaluation: Quantitative methods and practices. Washington, DC: World Bank publications.
[103]	Kitchin, R. (2014). Big data, new epistemologies and paradigm shifts. Big Data and Society, 1(1), 1-12.
[104]	Kuhn T.S.(1962). The structure of scientific revolutions. Chicago: University of Chicago press.
[105]	Kuhn, T.S. (1969. The structure of scientific revolutions. Retrieved on July 31, 2017, from 1969). The structure of scientific revolutions. Retrieved on July 31, 2017, from .
[106]	Largent, M.A., and Lane ,J.I. (2012). STAR METRICS and the science of science policy. Review of policy Research, 29(3), 431-438.
[107]	Lenzerini, M. (2011). Ontology-based data management. CIKM, proceedings of the 20th ACM International Conference on Information and Knowledge Management (pp. 5-6). New York: ACM.
[108]	Leydesdorff, L. (2012). The triple helix, quadruple helix, and an N-tuple of helices: Explanatory models for analysing the knowledge-based economy? Journal of the Knowledge Economy, 3(1), 25-35.
[109]	Lutz, C.S. (2017. Alasdair Chalmers MacIntyre. In The Internet Encyclopedia of philosophy. Retrieved on January 7, 2017, from 2017). Alasdair Chalmers MacIntyre. In The Internet Encyclopedia of philosophy. Retrieved on January 7, 2017, from .
[110]	MacIntyre, A. (2006). The end of education: The fragmentation of the American university. Commonweal, 133(18), 10-14.
[111]	MacIntyre, A . (2007).After virtue:A study in moral theory. 3rd ed. Notre Dame, Indiana: University of Notre Dame press.
[112]	MacIntyre, A . (1988). Whose justice? Which rationality? Notre Dame, Indiana: University of Notre Dame press.
[113]	Mansfield, E. (1991). Academic research and industrial innovation. Research policy, 20(1), 1-12.
[114]	Mansfield, E. (1995). Academic research underlying industrial innovations: Sources, characteristics, and financing. The Review of Economics and Statistics, 77(1), 55-65.
[115]	Martin, B.R. (2016). Introduction to discussion paper on the sciences are different and the differences are important. Research policy, 9(45), 1691.
[116]	Mingers, J. (2006). Realising systems thinking: Knowledge and action in management science. Boston, MA: Springer.
[117]	Mirowski p., and Sent, E.M. (2002).Science bought and sold:Essays in the economics of science Chicago: University of Chicago press Essays in the economics of science. Chicago: University of Chicago press.
[118]	Moed, H.F. (2016). Altmetrics as traces of the computerization of the research process. In C.R. Sugimoto (Ed.), Theories of Informetrics and Scholarly Communication. A Festschrift in Honor of Blaise Cronin (pp. 360-371). Berlin: De Gruyter.
[119]	Moed, H.F., and van Leeuwen,T.N. (1996). Impact factors can mislead. Nature, 381(6579), 186.
[120]	Moed H.F., Gl?nzel W., and Schmoch U. (2004). Handbook of quantitative science and technology research. Dordrecht: Springer Netherlands.
[121]	Moore S., Neylon C., Eve M.p., O’Donnell D., and pattinson D. (2017). “Excellence R Us”: University research and the fetishisation of excellence. palgrave Communications. Retrieved on July 31, 2017, from https://www.nature.com/articles/palcomms2016105.
[122]	Moreau L., Freire J., Futrelle J., McGrath R.E., Myers J., and paulson p. (2008). The open provenance model: An overview. In International provenance and Annotation Workshop (pp. 323-326). Berlin: Springer.
[123]	Morris, W.T. (1967). On the art of modeling. Management Science, 13(12), B707-B717.
[124]	Munafò M.R., Nosek B.A., Bishop D.V., Button K.S., Chambers C.D., du Sert N.p., . and Ioannidis J.p. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021.
[125]	Myung, I.J. (2000). The importance of complexity in model selection. Journal of mathematical psychology, 44(1), 190-204.
[126]	National Research Council.(2014).Science of science and innovation policy: principal investigators’ conference summary . Washington, DC: The National Academies press.
[127]	Nelson, R.R., and phelps, E.S.(1966). Investment in humans, technological diffusion, and economic growth. The American Economic Review, 56(1/2), 69-75.
[128]	Nielsen, M. (2012).Reinventing discovery:The new era of networked science. princeton: princeton University press.
[129]	Nielsen M.A., and Chuang, I.L. (2010). Quantum computation and quantum information. 10th Anniversary Edition. Cambridge: Cambridge University press.
[130]	Nowotny H., Scott p., and Gibbons M. (2001). Re-thinking science: Knowledge and the public in an age of uncertainty. Cambridge: polity.
[131]	O’Donnell, C.J. (2016). Using information about technologies, markets and firm behaviour to decompose a proper productivity index. Journal of Econometrics, 190(2), 328-340.
[132]	OECDv. (2002. Frascati Manual: proposed standard practice for surveys on research and experimental development. Retrieved on July 31, 2017, from 2002). Frascati Manual: proposed standard practice for surveys on research and experimental development. Retrieved on July 31, 2017, from .
[133]	OECD. (2005). Oslo Manual:Guidelines for collecting and interpreting innovation data. 3rd edition. paris: OECD publishing.
[134]	OECD. (2011). Quality framework and guidelines for OECD statistical activities. paris: OECD publishing.
[135]	OECD (2015a). Making open science a reality. OECD Science, Technology and Industry policy papers No. 25. paris: OECD publishing.
[136]	OECD. (2015b. Frascati Manual 2015: Guidelines for collecting and reporting data on research and experimental development. Retrieved on July 30, 2017, from 2015b). Frascati Manual 2015: Guidelines for collecting and reporting data on research and experimental development. Retrieved on July 30, 2017, from .
[137]	Owen R., Macnaghten p., and Stilgoe J. (2012). Responsible research and innovation: From science in society to science for society, with society. Science and public policy, 39(6), 751-760.
[138]	parent C., and Spaccapietra, S. (2000)Database integration:The key to data interoperability. In M.p. papazoglou, and Z. Zari (Eds.), Advances in Object-Oriented Data Modeling (pp. 221-253). Cambridge: , MA: The MIT press.
[139]	parmeter C.F., Wan A.T., and Zhang X. (2016). A model averaging stochastic frontier estimator, paper presented at the NApW 2016 Quebec City, Canada, June 2016.
[140]	perkmann M., Tartari V., McKelvey M., Autio E., Brostrom A., D’Este p., .. and Krabel S. (2013). Academic engagement and commercialisation: A review of the literature on university-industry relations. Research policy, 42(2), 423-442.
[141]	pollock S.M. (1976). Mathematical modeling: Applying the principles of the art studio. Engineering Education, 67(2), 167-171.
[142]	Romer p.M. (1986). Increasing returns and long-run growth. The Journal of political Economy, 94(5), 1002-1037.
[143]	Romer p.M. (1994). The origins of endogenous growth. The Journal of Economic perspectives, 8(1), 3-22.
[144]	Roper C.D. , and Hirth M.A. (2005). A history of change in the third mission of higher education: The evolution of one-way service to interactive engagement. Journal of Higher Education Outreach and Engagement, 10(3), 3-21.
[145]	Rosenberg N., and Nelson R.R., (1994). American universities and technical advance in industry. Research policy, 23(3), 323-348.
[146]	Saltelli A. , and Funtowicz ,S. (2014). When all models are wrong. Issues in Science and Technology, 30(2), 79-85.
[147]	Saltelli A. , and Funtowicz ,S. (2015). Evidence-based policy at the end of the Cartesian dream: The case of mathematical modelling.In G. pereira, and S. Funtowicz (Eds.), Science, philosophy and Sustainability: The End of the Cartesian Dream. Beyond the Techno-Scientific Worldview. Routledge’s Series: Explorations in Sustainability and Governance (pp. 147-162). London: Routledge.
[148]	Saltelli A., Guimaraes pereira A., van der Sluijs J.p., and Funtowicz S. (2013). What do I make of your latinorum? Sensitivity auditing of mathematical modelling. International Journal of Foresight and Innovation policy, 9(2-3-4), 213-234.
[149]	Saltelli A., Ratto M., Andres T., Campolongo F., Cariboni J., Gatelli D., .. and Tarantola S. (2008). Global sensitivity analysis: The primer. Chichester, UK: John Wiley and Sons.
[150]	Salter A.J., and Martin ,B.R. (2001). The economic benefits of publicly funded basic research: A critical review. Research policy, 30(3), 509-532.
[151]	Scharnhorst A., Borner K., van den Besselaar p. (Eds.). (2012). Models of science dynamics: Encounters between complexity theory and information sciences. Berlin: Springer.
[152]	Simar L. , and Wilson ,p.W. (2015). Statistical approaches for nonparametric frontier models: A guided tour. International Statistical Review, 83(1), 77-110.
[153]	Solow R.M. (1957). Technical change and the aggregate production function. The Review of Economics and Statistics, 39(3): 554-562.
[154]	Stephan p.E.(2012). How economics shapes science. Cambridge, MA: Harvard University press.
[155]	Stilgoe J., Owen R., and Macnaghten p. (2013). Developing a framework for responsible innovation. Research policy, 42(9), 1568-1580.
[156]	Teixeira p.N. , and Dill,D.D. (Eds.) . (2011). public vices, private virtues? Assessing the effects of marketization in higher education (Vol. 2). Rotterdam:Sense publishers.
[157]	Teixeira p., Jongbloed B., Dill D., and Amaral A. (2004). Markets in higher education: Rhetoric or reality? Dordrecht: Kluwer.
[158]	Vandenbussche J., Aghion p., and Meghir C. (2006). Growth, distance to frontier and composition of human capital. Journal of Economic Growth, 11(2), 97-127.
[159]	Veugelers R. , and Del Rey,E. (2014. The contribution of universities to innovation, (regional) growth and employment. EENEE Analytical Report. Munich, Germany: EENEE. Retrieved on July 31, 2017, from 2014). The contribution of universities to innovation, (regional) growth and employment. EENEE Analytical Report. Munich, Germany: EENEE. Retrieved on July 31, 2017, from .
[160]	Vinkler p. (2010).The evaluation of research by scientometric indicators. Cambridge: Chandos publishing.
[161]	von Hippel,E. (2005). Democratizing innovation. Cambridge, MA: The MIT press.
[162]	von Hippel, E. (2016). Free innovation. Cambridge, MA: The MIT press.
[163]	Weinan E. (2011). principles of multiscale modeling. Cambridge: Cambridge University press.
[164]	West J., Salter A., Vanhaverbeke W., and Chesbrough H. (2014). Open innovation: The next decade. Research policy, 43(5), 805-811.
[165]	Whitley R. , and Gl?ser ,J. (Eds.). (2007). The changing governance of the sciences: The advent of research evaluation systems. Dordrecht: Springer Netherlands.
[166]	Wilsdon J., Allen L., Belfiore E., Campbell p., Curry S., Hill S., Jones R., ... and Johnson B. (2015. The metric tide: Report of the independent review of the role of metrics in research assessment and management. Retrieved on July 31, 2017, from 2015). The metric tide: Report of the independent review of the role of metrics in research assessment and management. Retrieved on July 31, 2017, from .
[167]	Willemain T.R. (1994). Insights on modeling from a dozen experts. Operations Research, 42(2), 213-222.
[168]	Zucchini W. (2000). An introduction to model selection. Journal of Mathematical psychology, 44(1), 41-61.