aes18 International Evaluation Conference

Lewis Atkinson (Haines Centre for Strategic Management LLC)

The systems thinking approach is an important tool for evaluators
because it is a way to:

• clarify the system level that you are trying to change
• be people-centric by having a focus on clarity of measures of their ‘better-off-ness’ Rapidly build evaluation capacity
• establish a common language for measuring impact focus on evidence-based practice and continuous
• improvement Turn strategic reflection into practical action
• ensure a participative process with stakeholders to codesign to create systems change Have a low tech/low cost introduction to measuring outcomes of programs
• use iterative hypothesis testing to validate theory of change for programs
• ensure accountability for theory of change over time, by whom and at which systems level
• Create a narrative that is evidence-based and reported as a contribution to social impact

The authors use group reflection & discussion based on a Results Based Accountability (RBA) adoption case study within a medium-sized NFP company delivering community services in Queensland. Practitioners will be exposed to how RBA and other systems thinking tools and participative methods for stakeholder engagement are used build evaluation capacity, create an evaluative culture, encourage timely utilisation of feedback loops and a commitment to strategic learning.

By the end of this session, participants will be able to:

• understand how systems thinking that can be applied facilitate evaluative thinking
• understand how Results Based Accountability (RBA) is used to Build Evaluation Capacity
• understand how to use RBA to report change for people at different system levels
• understand that the systems thinking approach can accommodate any validated method of measurement of change over time.

Keryn Hassall (The Australia and New Zealand School of Government)

Values underpin evaluative decision-making, and evaluation experts have advocated for clarity of values in evaluation. But there are conceptual and practical challenges to working with values. This session draws on the findings of values research in social psychology and combines this with techniques for values inquiry developed by evaluation thought leaders.
The learning objectives of this skill-building session are to (a) give participants some understanding of the research into values, to enable them to comfortably talk about values, and (b) provide ways to include values explicitly through all stages of evaluation, and particularly for evaluative synthesis.

Values are trans-situational goals that motivate people's action and serve as guiding principles in their lives. In a society or organisation, values are the broadly shared abstract ideas about what is good and desirable. These social values serve to justify actions that are taken in pursuit of these goals, and are implicitly and explicitly embedded in policies and programs. Values underpin the programs we evaluate, and how we evaluate them. Working with values allows evaluators to make clearer decisions about evaluative criteria, evaluation methods, to interpret the distribution of outcomes, and make evaluative judgements.

Participants will learn about research into values, with a framework for understanding values that can be used to facilitate discussions about values in evaluation. The session will guide participants through using this framework to interpret and map values as they appear in a social context - in policies, programs, documents and organisations. It will show the importance of understanding and being explicit about values in all stages of program development - through the process of needs analysis and developing a program theory.

Participants will learn about techniques for eliciting and clarifying values, and discuss ways to incorporate values in each stage of an evaluation, and how this clarity about values can enable more effective evaluative synthesis.

Ruth Pitt (Department of Social Services)

Despite the increasing popularity of visual presentation methods, writing is still a core skill for evaluators.  Evaluators need to write for diverse audiences and produce attractive, error-free reports while facing tight deadlines and budgets. I have previously worked as an editor, an evaluator and a consultant supporting organisations to improve their evaluation documents. In my current role, I receive and review numerous evaluation reports. These experiences have given me insight into the common problems with evaluation reports, why they occur and how to fix them. In this skill building session, I will share tips and tricks for improving your writing when facing a deadline, whether the final version is due in one hour, one day or one week.One hour: the clock is ticking and you've only just finished writing. I'll demonstrate affordable editing software that can quickly reduce errors and improve consistency. I'll also provide a handout outlining the features and costs of other options. One day: The final report is due tomorrow and your draft is...okay. I'll provide a checklist of common problems that can be addressed in one day, and demonstrate Word features that will help you find and fix them. One week: you planned carefully and left plenty of time for revising and editing. I'll share practical steps for improving the structure, readability and visual appeal of your reports.  The skill building session is suitable for evaluators of any level of experience who would like refresher training in writing and editing, particularly on how technology can support (rather than thwart) efforts to deliver a quality report on time. 

Dan Borg 

The online or hard copy survey is one of the go-to data collection tools in the evaluator's tool box. Easy to use online software is making these kinds of surveys more accessible than ever before with hundreds of questions for the budding survey designer to choose from. But what makes a quality survey? How do you know that your survey is well constructed and has the right kinds of questions designed to elicit high quality and reliable responses? 

In this skill-building session, aimed at those new to survey questionnaire design, we will explore the art of designing a good survey questionnaire.

We will work through the fundamentals of survey design, including overall structure; common question types and good practice in their use. We will also work through some common ways in which the design of surveys can influence responses (either increasing or decreasing reliability).  By understanding how we can commonly go wrong in the design of surveys, we will highlight strategies for avoiding these problems. 

Participants will leave the session with an increased understanding of:

• The role of the survey designer in influencing the reliability of survey results
• Common survey problems (and strategies for how to avoid these problems) 
• Foundational principles for good survey design