aes18 International Evaluation Conference

Catherine Manley (Miles Morgan Australia), Ebeny Wood (Beacon Foundation), Anna Powell (Beacon Foundation)

During 2017, a team in Perth was seeking out a case study subject for a forthcoming publication on Australian skills development at the local level and came across a live example of transformational systems change in action, right here in Tasmania.
The Beacon Foundations’ Collective Ed is a work-in-progress example of systems change design and practice, and demonstration of the willingness and commitment of Tasmanian community, education, industry, and government. Currently working with six Tasmanian secondary schools, Collective Ed is designed to help schools try and test new ideas and new ways of helping young people complete Year 12.
This special panel brings together practice observers, designers, and evaluators, as well as school leadership associated with the Collective Ed project. The session is designed to stimulate discussion of, and engagement with the panel's perspectives and explore answers to valuable conference questions from both a local and global standpoint:
What are we learning about collaborating with unlikely partners and operating at the systems level?
How is evaluation practice adapting to work at the system level?

Amohia Boulton (Whakauae Research for Maori Health and Development), Gill Potaka Osborne (Whakauae Research for Maori Health and Development, NZ), Lynley Cvitanovic (Whakauae Research for Maori Health and Development, NZ), Sharon Clarke (Women's and Children's Health Network Government of South Australia, AU), Lisa Warner (YWCA Adelaide), Jenni Judd (Central Queensland University), Margaret Cargo (University of Canberra)

The STEPS project has coalesced as a discrete piece of work over several years. Its genesis lies in the desire of a group of Indigenous and non-Indigenous evaluators in NZ and Australia to improve evaluation undertaken in Indigenous settings. Mixed-method concept mapping methodology was used to brainstorm practices and strategies to support culturally safe evaluation; 106 strategies were consolidated and sorted into conceptually meaningful groups; each strategy was rated on relative importance and achievability. Approximately 400 participants were involved in this work. Concept maps for each country were developed using multi-dimensional scaling and hierarchical cluster analyses. The 12 cluster Australia map reflects three thematic regions: (1) An Evaluation Approach that Honours Community; (2) Core Heart of the Evaluation; (3) Cultural Integrity of the Evaluation. The 11 cluster New Zealand map reflects four regions: (1) Authentic Evaluation Practice; (2) Building Māori Evaluation Expertise; (3) Integrity in Māori Evaluation; (4) Putting Community First.   Both maps highlight the importance of cultural integrity in evaluation. Differences include the distinctiveness of the Respecting Language Protocols concept in the Australia map with language being embedded within the concept of Knowing Yourself as an Evaluator in a Māori Evaluation Context in the NZ map. The ratings on importance and achievability highlight that all concepts are important though differences exist between countries in perceived achievability. In both countries the concepts of Evaluator Qualities and Evaluator Integrity were rated as very important and as most achievable. We will present an overview of the concept maps and highlight importance and achievability ratings. Participants will be invited to discuss how resources can best be harnessed to 'grow' evaluation that works for Indigenous communities. 

Rebecca Denniss (First Person Consulting), Matthew Healey (First Person Consulting), Liz Smith (Litmus / AES), Amy Gullickson (University of Melbourne, Centre for Program Evaluation), Nerida Buckley (Sustainability Victoria), Sally Hartmanis (Deloitte Access Economics)

The beauty of evaluation as a discipline and a professional practice is that it involves diverse skills, capabilities, mindsets and approaches that can be applied across diverse contexts, cultures, landscapes and sectors. While this presents opportunities, it can also be overwhelming.

New and emerging evaluators often get told what they need and what they should be doing - so, instead, we're asking them for their perspective. For this panel session, we have brought together a collection of movers and shakers with a range of experiences, including: - a young up-start who started up his own evaluation firm - new and emerging evaluators from the government and non-government sectors - an evaluation educator who challenges and inspires evaluators across all stages of their careers - an experienced evaluator and senior AES member who describes herself as a 'disrupter'.

If you are a new or emerging evaluator, this is your chance to ask questions, seek mentoring and advice, share experiences and, most importantly, tell us what you need to transform your career.
If you're an experienced evaluator, it's your chance to meet some of the region's brightest new evaluators—and talk to them about all the things you wish you'd known in the early stages of your career!

Facilitated by new and emerging evaluators for new and emerging evaluators, this panel session will involve discussion about: - capabilities, mindsets, approaches and skills - learning from failures and f**k ups - mentoring and support - professional pathways

After hearing a bit about the stories of each of the panellists, the majority of this session will be dedicated to questions and answers, and facilitated audience discussion.

Zazie Tolmer (Clear Horizon), Mila Waise (Department of Health and human Services)

The presenters are involved in delivering the Children and Youth Area Partnerships (CYAP) a Victorian government-led Collective Impact initiative delivered through place-based area partnerships in eight sites across Victoria. The Area Partnership members are intentionally diverse and together a) identify systemic and local factors that contribute to the vulnerability of children, young people and their families, b) design and test new ways of thinking and prototypes to overcome these and c) seek to influence uptake of successful prototypes by government, business, philanthropy, community and others. Ultimately, the initiative aims to work out how government can lead collaborative place-based approaches that result in real and sustainable positive change for those experiencing vulnerability.

A key component of the approach is to embed evaluative thinking and strategic learning.  We are finding that in order for the learning and local innovative practices to drive system change, a strong authorising environment and collaborative governance is needed. There needs to be a strong collective forum where learning can be further tested and innovative practice can be implemented. There needs to be a culture where partners feel 'safe to fail' and learn while continuously refining their work. There needs to be an environment where accountability is well balanced with learning and power is shared. Only this has the potential to lead to true transformation at the local and system levels and within each component/actor in the system.

The following questions will be explored in the presentation:1. What does evaluative thinking and strategic learning mean and look like in a government-led Collective impact initiative? What are the tensions and 'easy fits'?2. What difference has it made to our work? What are the implications on our resources, the intensity of the work, the impacts and ripples?3. Yeah but, so what? Has any of this actually sparked the transformation we are after? Procedure: The presentation will be delivered by three presenters, which will include perspectives and expertise from:• one Principal Advisor that is a place-based practitioner that leads the initiative within an area level, who is also the local backbone and  drives the change process locally• a representative from the central government unit that provides whole-of-initiative backbone support  and leads  transformation  within government; and • an evaluator who has been engaged to provide practice advise and embed a learning culture across the initiative. 

The panellists will each deliver a presentation of about 10 minutes each where their different perspectives will be explored.  The presentations will be followed by a 15 minutes question time from the audience where answers will be provided to generate a short discussion on themes that the audience will be most interested in.  It is anticipated that time will permit for about three to five questions that will be followed by answers from the most relevant panelists and brief commentary from the audience.

David Morton (Department of Finance), Brad Cook (Department of Finance)

The Australia Parliament - through the Joint Committee on Public Accounts (JCPAA) -  has encouraged the Department of Finance (Finance) and others to support capacity-building to further implement the enhanced Commonwealth performance framework. Evaluators have a key role. They will need to be clear about what they have offer, and how they can help deliver better performance information to government, the Parliament and public more broadly. They will need to be willing to adapt what evaluators do and know today, and participate in developing the flexible approaches needed in the future. The performance frameworks calls for approaches that deliver performance information that simultaneously supports  accountability to the taxpaying public and everyday operational decisions.  We encourage the Australian evaluation community to reflect on  what it has to offer and how it can work with others to shape the evolution the of the performance framework. 

The performance framework commenced on 1 July 2015. It succeeds if it enables the Australian Parliament and public to understand the benefits of Commonwealth activity. The framework encourages entities and companies to move past over-reliance on input and output-based performance measures. There is a clear role for evaluators to contribute to this important adjustment. Opportunities lie in helping a larger cross-section of the Commonwealth public sector understand and use the evaluators' toolbox - for example, program theory and qualitative analysis - to improve the quality of published performance information available to stakeholders. The evaluation community has the opportunity to be at the centre of key expertise, and to make a critical contribution to building the capability of 'performance professionals' across the public sector.

Samantha Togni (RMIT University), Nan Wehipeihana (Kinnect Group), Kate McKegg (Kinnect Group), Sonya Egert (Inala Indigenous Health Service)

Innovation is required in Indigenous settings to strengthen communities and address challenging and complex social issues. Evaluation in these contexts is important to understand innovation effectiveness and takes place at the interface of different knowledge systems. Therefore, the challenge for evaluation in these contexts is to transform the power and privilege inherent in evaluation and to be centred on Indigenous voices, values and aspirations.

Developmental evaluation is designed to support innovation development in complex and dynamic contexts. Informed by complexity theory and systems thinking, developmental evaluation is relationship-based and pays attention to different perspectives, inter-relationships, context, boundaries and emergence. As the practice of developmental evaluation continues to evolve, recognition of its ability to respond to different cultures, diverse communities and Indigenous peoples' worldviews is increasing. Understanding how this is achieved is important. 

Indigenous and non-Indigenous evaluator panellists will critically reflect on our developmental evaluation practice experience in New Zealand and Australian Indigenous contexts in relation to transforming power relations and at the interface of different knowledge systems. The panellists will reflect on what genuine co-creation that recognises different worldviews looks like in practice, the dynamic role and orientation of the evaluator and how developmental evaluation grounded in culture can address power and privilege, facilitate collaboration in innovation and support Indigenous peoples' aspirations. We will also discuss the challenges and limitations of using developmental evaluation in culturally diverse contexts. To promote rich discussion, we will invite audience participation through questions and sharing of experiences of developmental evaluation in Indigenous or culturally diverse contexts.

The history of evaluation too often has been detrimental to, and marginalised, Indigenous people and communities. Our western frames of thinking and reasoning are simply not adequate for meeting the aspirations of Indigenous communities. Developmental evaluation offers an approach to include diverse knowledges in these pursuits

Delyth Lloyd (Victorian Department of Health and Human Services), Vanessa Hood (Rooftop Social), Megan Kerr (Victorian Department of Education and Training), Kate Nichols (Victorian Department of Economic Development, Jobs, Transport and Resources), Amanda Reeves (Victorian Department of Education and Training), Roberta Thorburn (Australian Government Department of Industry, Innovation and Science), Eleanor Williams (Victorian Department of Health and Human Services), Martin Hall (New South Wales Department of Education)

Building an organisation's evaluation culture and capability is not an exact science. Different approaches are suited to different contexts and must be responsive to the organisation's individual characteristics. Factors such as leadership, systems, processes, staff attitudes and skills will inform what strategies will be most effective in transforming an organisation's evaluation culture.  Who leads the evaluation capability effort and the resources available will also determine the approach. Sometimes evaluation capability building is led by a central team, other times it is dispersed throughout the organisation, or contracted-in via external consultants. Sometimes the funding and resources are flowing, while other times there is only a trickle.

So what is the current situation in the public sector at State and Commonwealth level, a sector undergoing marked transformation and reform with increased demand for accountability, outcomes-thinking, evaluation and evidence-driven ways of working? In this context, what different approaches are being used to help strengthen organisational evaluation culture and capability building? Are evaluation capability building endeavours equipping government organisations to thrive in this time of change?

This interactive session will explore the current evaluation capability and culture building approaches being used in five large State and Commonwealth government departments, including the:

1. Victorian Department of Education and Training
2. Victorian Department of Health and Human Services
3. Victorian Department of Economic Development, Jobs, Transport and Resources
4. Australian Department of the Environment and Energy
5. New South Wales Department of Education
6. Australian Government Department of Industry, Innovation and Science

The session will be invaluable for those who work in, or with, any government agency as well as those interested in evaluation capability building more broadly. Each organisation will showcase different evaluation culture and capability building approaches tailored to their context. A facilitated mini-workshop will then invite participants to reflect on the implications for their own organisations and co-create practical strategies for enhancing evaluation capability and culture building practice in different contexts.

Deborah Askew (The University of Queensland), Samantha Togni (S2 Consulting), Philip Schluter (University of Canterbury), Sonya Egert (Inala Indigenous Health Service)

We implemented a transformative model of primary health care for Aboriginal and Torres Strait Islander peoples with complex chronic disease. This research project used developmental evaluation to develop, adapt and understand why and how our intervention had the impact it did.  Therefore, this project brought together different paradigms, different priorities and different languages. Our challenge was to unite these different perspectives to improve health outcomes for Indigenous people.

Quantitative research is characterised as being value-free, structured, logical and reductionist, with the researcher being distant and independent to the research. In contrast, developmental evaluation requires flexibility, innovation, tolerance for ambiguity, and the evaluator is inseparable to the process of refinement and adaptation of the intervention. Improvements in the health of Australia's Indigenous people requires honouring the Aboriginal definition of health. Bringing these worldviews together required identification of shared values and beliefs.

Indigenous and non-Indigenous researcher and evaluator panellists will critically reflect on the challenges, opportunities and successes we experienced implementing, refining, adapting and evaluating our model of care and bringing together these different knowledge systems. The panellists will reflect on how their personal ideologies and values created a space where the importance of each different worldview was recognised and given its rightful place in the project; how tensions at the interface were recognised and celebrated as opportunities to learn; and how developmental evaluation facilitated the successful conduct of the research project and improved Indigenous peoples' health. To promote audience participations, we will facilitate paired discussions and feedback where participants can share their own stories of successes, failures, and learnings in similar situations.

The history of research and evaluation has too often privileged outcomes that are frequently of little benefit to Indigenous people and communities. Developmental evaluation offers an approach to facilitate symbiotic relationships rather than tragic culture clashes.

Nan Wehipeihana (Research Evaluation Consultancy Limted - a member of the Kinnect Group), Judy Oakden (Pragmatica Limited - a member of the Kinnect Group), Kate McKegg (The Knowledge Institute - a member of the Kinnect Group), Julian King (Julian King & Associates - a member of the Kinnect Group)

Evaluation rubrics are a powerful and influential approach to evaluation-specific methodology that can be used in collaborative/participatory or independent evaluations to build a clear, shared understanding of how quality, value, and effectiveness are defined. Evaluative rubrics make explicit the basis for evaluative judgments about effectiveness or performance, as well as importance. 

Drawing from their experience of using rubrics in many evaluation settings, the presenters in this panel session will provide an overview of rubrics, as well as more detail about different kind of rubrics and their uses, their strengths and weaknesses, and the ability of rubrics to explore and integrate shared values providing a clear and transparent basis for making decisions.
Objectives:In this panel presentation, participants will gain insights from panel members' practice about rubrics in the following areas: 
An overview of rubrics • What are rubrics?• Where do they come from?• What are the components of a rubric?• Why are they useful / transformative for evaluation practice?
Different kinds of rubrics:• What different types of rubrics are there?• What are their key features?• What are the design considerations for each? • What is the comparative value of each type for makingevaluative judgments?

The strengths and weaknesses of rubrics• What are the strengths of rubrics?• Troubleshooting, faults and mishaps - overcoming the weaknesses of rubrics in practice?• How do they transform evaluation practice?

Using rubrics to integrate shared values• Whose perspectives and values count when using rubrics?• How do you weave different values into the design and use of a rubric?• Why does this matter?

This panel will invoke conversations that explore ideas that will challenge the status quo in evaluation. The session will also seek to establish a community of practice for emerging evaluators.  
The panel will introduce emerging evaluators from a range of professional backgrounds. With the focus on ideas that challenge the status quo, each panel member will offer their unique perspective and experience, drawing on ideas around the role of evaluators in alleviating poverty, how evaluation can drive Aboriginal sovereignty, and opportunities for inclusivity and integrating lived experience into evaluation. Facilitated by emerging evaluators, the session will include opportunities for the audience to pose questions to the panel.
Panel members

• Skye Bullen, PCT Consulting
• Fran Demetriou, Lirata Consulting
• Joanna Farmer, beyondblue 
• Sarah Leslie, Fred Hollows
• Rini Mowson, Clear Horizon

Facilitators: Eunice Sotelo and Nathan Delbridge from Clear Horizon
Structure of the session – 50 minutes 

• Part 1 – Panel introduction (15 min)
• Part 2 – Panel discussion and audience questions (30 min)
• Part 3 – Close and next steps (5 min) 

Dan Borg, Jennifer Thompson (VicRoads), Victoria Cook (Department of Economic Development, Jobs, Transport and Resources), Ellie McDonald (Department of Health and Human Services)

What are the common traps for young players newly transitioned to evaluation? New to evaluation and asking this question - then this is the session for you. Come and hear about the lessons learnt from a panel of practitioners who have recently transitioned to evaluation through diverse pathways. Hear also from evaluators with dedicated roles in building evaluation capability (and the common issues encountered). You'll also have the opportunity in this facilitated session to share your experiences and lessons learnt. 

Part panel, part facilitated session/panel forum, we will discuss pathways into evaluation practice; successes and challenges in making the transition; all aspects of the evaluation journey (from first conversations with clients/commissioners to evaluation reporting); lessons in maximising evaluation use; and, where to turn to for help. 

The session will involve a mix of a facilitated panel; audience Q&As and facilitated group activities designed to encourage audience participation and sharing of experiences.  

Sarah Kearney (Our Watch), Anna Trembath (Our Watch), Elise Holland (OurWatch)

In recent decades, there have been growing efforts to apply intersectional theory to the field of gender equality, health promotion, and other areas of social policies. While much of the focus so far has been on understanding how to apply an intersectional lens to policy and programming, of equal importance is the application of an intersectional approach to monitoring and evaluation and its potential to reveal meaningful distinctions and similarities in order to better a) understand the impact of social interventions and b) monitor progress toward social policy outcomes.  

The panel consists of practice specialists with expertise in evaluation and monitoring from Our Watch, the national foundation for the prevention of violence against women. Each panellist applies an intersectional approach to designing either project-level evaluations or monitoring frameworks for tracking population-level change.  This panel will open by exploring the concept of intersectionality and its role in the development of transformative social policy.
Building on this theoretical understanding, the panellists will be interviewed by a facilitator on how they have embedded intersectionality into their monitoring and evaluation projects, drawing primarily from examples of violence prevention interventions and initiatives aimed at promoting gender equality. Examples will include: the evaluation of a national cultural change campaign (delivered across digital platforms) and the development of monitoring mechanism which tracks population-level progress towards the prevention of violence against women.   

The panel will conclude with an interactive facilitated discussion. Audience members will be asked to interrogate evaluation case studies (provided by the panellists), discussing whether the examples are intersectional, and identifying practical steps to advance an intersectional approach of the case study. At the conclusion of the panel, participants will be directed toward relevant resources to support them to move from 'inclusive' evaluations that simply recruit for diversity, towards transformative, intersectional evaluation design.

Jess Dart (Clear Horizon), Anna Powell (Beacon Foundation), Ebeny Wood (Beacon Foundation), Jo Taylor (Paul Ramsay Foundation), Kitty te Riele (Peter Underwood Centre, University of Tasmania)

There is a gradual shift in realisation that intractable or wicked problems are going to require different types of solutions - and different ways of working together.  There has been considerable energy in setting up and establishing collaborative initiatives to disrupt and change systems. These initiatives don't fit the usual confines of a program or service. They often work across sectors, are emergent,  with long establishment phases.  With the challenge of how to work with this new kind of initiative comes the challenge of how we meaningfully evaluate in this space. It's a big topic, and in this panel we focus on evaluation in the establishment phase of a systems change initiative. 

Early ideas include looking to diagnose what is holding the system in a non-optimal state, to looking for key anchors that enable systems change such as adaptive leadership, collaborative health, and trusting relationships.

This panel session brings you four different perspectives on the topic of evaluating initiatives with systems change endeavours. The philanthropist funder, the backbone leader, the project director and the evaluators. Each present their challenges and their ideas for how to evaluate systems change projects on a real example of a 5 year project here in Tasmania in 5 schools.