TSI launched its first Working Paper on impact assessment. In this paper the TSI team from Austria, Ruth Simsa, Olivia Rauscher et.al. review existing scientific literature, official reports and policy documents on impact measurement methodologies in order to elaborate on impact indicator domains, agreed upon by the TSI consortium and put to the test by third sector stakeholders across Europe. They propose a theoretical framework for TSIs research agenda and discuss different methods to allow analysing third sector impact not only on meta- but also on macro-level of society.

Little scientific findings exist concerning third sector impact as a whole. There are no systematic methodological guidelines or for empirical analysis. TSIs objective is to reach a clear understanding of the third sector in Europe in terms of its scope and scale, existing and potential impacts, and barriers to the third sector fully contributing to Europe’s common welfare. Developing a methodology for impact measurement is a central task for TSI researchers.

Working Paper 1 starts with a review of the literature on impact measurement as interdisciplinary topic, drawing from evaluation research, social accounting, ecological and social impact assessment, NPO research, social enterprise and for-profit entrepreneurship (business ethics and CSR). Even the term “impact” has many definitions. One important fact to stress is that impact does not equal outcome. Impact only refers to changes that can actually be attributed to the activities of a particular programme, organization or sector, as defined by Clarke et.al. 2004.

Impact measurement is regarded as one of the most fundamental topics in non-profit studies, especially since impact data are increasingly demanded by funding bodies and policies, but also from within the third sector itself in order to increase its legitimacy. Challenges in finding a common methodology lie among other factors in the heterogeneity of the third sector and in determining causalities. Impact assessment needs clearly defines indicators that are difficult to establish on aggregate level: consensus on what is “good” for society is hard to find. TSI is seeking impact indicators for the areas civic engagement, innovation, citizen well-being, economic development, and human development in a consensus-based approach. Input from third sector and policy stakeholders across Europe support this process.

Theoretical framework
Underlying impact analysis is the impact value chain, a theoretical model that defines the theoretical functioning of a programme, organization or sector. It reflects on input, activities, output, outcome, deadweight and impact. Furthermore, outcome and impact can have economic, social, political, cultural, ecological and physical or psychological dimensions, captured in the Impact Box model that helps to cluster and locate effects. Both models can be combined to measure impact both on meta-level and macro-level.

Method
The Social Return on Investment (SROI) method is currently the most popular approach for measuring impact at societal (macro-) level, trying to assess social value created by social organisations, assessing in it monetary terms while including social cultural, political and ecological aspects (SROI-value). It s a promising method to work with but it will need further development.

Impact indicators
A core questions to reflect in TSIs research methodology will be which impacts can be attributed exclusively to the third sector. To this end types of impacts or impact goals to include in the theoretical model must be defined. The research literature shows a broader consensus for indicators related to well-being, economic and human impacts than in the areas of innovation and civic engagement. Stakeholders in national and EU meetings gave special priority to quality of life and well-being and civic engagement, empowerment, advocacy and community building.

The working paper underlines that assessing causalities, the deadweight, as well as long-term and unintended effects poses serious challenges that TSI researchers must address in 2015. Deciding on an impact assessment methodology will be a crucial step in TSI’s research process, as well as finalising a common definition of the third sector in Europe.

The report is available for download (808KB).