Idea Transcript
18 Apr 2016
Technical Competency Framework Evaluation Cadre 1. Introduction About the Advisory Cadres DFID advisors help identify, generate and utilise the best evidence, knowledge, technology and ideas to improve the effectiveness and impact of the UK’s aid programme. Advisors have key roles in programme design, management and evaluation, and in the broader development and implementation of policy. They have strong links with development networks, research organisations and professional bodies both in the UK and internationally. There are thirteen professional cadres with a combined total of more than 850 accredited advisors, around a third of the total DFID workforce. All DFID advisors need to continually update and refresh their skills, through their cadre programmes of continuing professional development (CPD), in line with the latest trends and knowledge in global development, poverty reduction and UK policy.
How the framework should be used for the Evaluation Cadre Each cadre framework follows similar design principles, and is used for advisor recruitment, performance management, Technical Competency Assessment (TCA), quality assurance, career development and workforce planning. The framework may also be used by other government departments and external partner organisations for collaboration and partnership working. The framework applies to all members of DFID’s Evaluation Cadre.
The Evaluation Cadre Monitoring and evaluation strengthen evidence-based decision-making, safeguard accountability, and improve portfolio quality and value for money of DFID’s programmes. They affect all of DFID’s policies, programmes and projects and DFID has explicit requirements to build these in at design stage. Members of DFID’s Evaluation Cadre support the generation of a wide range of timely evaluative information and context-specific knowledge that is relevant and tailored to the diverse needs of policy makers, programme managers and stakeholders. They also ensure this evidence can be used to feed into key programme and policy decisions and to manage programme and portfolio risks. DFID’s monitoring and evaluation system is decentralised which means that each spending unit is responsible for resourcing and delivering their own monitoring, evaluation, results reporting and learning. The system is supported and delivered by staff who have evaluation competencies and whose roles and skills complement each other. DFID’s Evaluation Cadre consists of two types of members: Evaluation Advisers (specialists who primarily work in evaluation posts) and Evaluation Page 1
Managers (staff with skills in monitoring and evaluation but who primarily work in non-cadre roles across DFID). DFID has over 30 FTE staff dedicated to Evaluation Advisory posts, making Evaluation Advisers one of the smallest Advisory groups. The majority of these staff resources (80%) are based in the UK, with the remainder in Africa and Asia. DFID’s Evaluation Cadre also includes a further 130 Evaluation Managers who have a level of accreditation to the evaluation cadre and may have some evaluation responsibilities as part of their role or an existing skills base in monitoring and evaluation. Most spending units will have a combination of Evaluation Advisers and Evaluation Manager resources to meet their evidence needs and deliver DFID’s large portfolio of monitoring and evaluation commitments. The primary role of Evaluation Advisers is to translate high-quality evidence into pragmatic solutions to real-world problems and to design and commission monitoring and evaluation solutions for programmes. Their primary skills are therefore in monitoring, evaluation, analysis, research and learning, while Evaluation Managers will have some but not necessarily the required technical depth in all of the relevant competencies, or they may be deployed in other specialist roles. All cadre members are trained in designing, commissioning and interpreting monitoring and evaluation approaches, methods and findings. Many have also developed specialist knowledge in the evidence base related to specific development sectors or methods and approaches. Evaluation Cadre members have particular expertise in cost-effectively designing programmes and policies in ways that allow for results and impacts to be measured and used to improve delivery and to test what works. All DFID’s Evaluation Advisers are required to be able to:
Support robust programme and policy decisions through leading the design, commissioning, management and delivery of programme and project reviews, monitoring and evaluation plans and other evidence products. Identify and analyse existing evidence base, assessing its reliability and relevance, and identify requirements to collect additional evidence across a range of different development sectors to meet stakeholder’s needs and support key decisions. Apply a range of different data collection and analytical approaches and methods to design tailored monitoring and evaluation plans, reviews and in some cases research programmes. Communicate complex technical monitoring, evaluation and research concepts in ways that non-experts can readily understand, and synthesise findings and share learning from monitoring and evaluation to empower stakeholders to understand and apply findings. Help departments and stakeholders to translate evidence, monitoring and evaluation results into practice. Lead stakeholder engagement and collaboration on monitoring and evaluation with representatives of other organisations (whether evaluation suppliers, NGOs, multilaterals or governments) to ensure DFID’s needs are met.
Page 2
Be able to respond to stakeholders’ changing needs by keeping abreast of latest developments in monitoring and evaluation theory and practice, including new and innovative approaches and methods. Build capacity and mentor colleagues to manage results, monitoring and evaluation in DFID, and support the capacity building of external organisations (e.g. programme implementers, multilaterals, OGDs). Work flexibly and confidently across DFID’s monitoring, evaluation, results and VfM agenda to support programme delivery and to ensure DFID has evidence of the impact of its programmes.
Competencies required from Evaluation Advisers at Accreditation In addition to minimum entry requirements, advisers are expected to demonstrate three different types of competencies at the point of accreditation: 1. Advisory Competencies which are common to all DFID advisory cadres and describe the essential skills required to be an adviser in DFID. These are used alongside the primary competencies in recruitment and promotion. 2. Primary Competencies: All Evaluation Advisers accredited to the cadre will need to demonstrate the four primary competencies and CPD should be used to maintain and improve them. 3. Specialist Competencies: All Evaluation Advisers accredited to the cadre are expected to demonstrate one specialist competency from a menu of technical specialisms. Job-specific competencies tested at recruitment Certain posts in DFID may require staff to acquire additional skills in specific areas. Examples of post-specific competencies are given in section 2 to illustrate the types of experience that may be requested by recruiting departments. All DFIDs advisers are expected to be able to deploy their own specialism while taking account of key cross cutting issues:
Understand political economy context in which DFID is operating, including how power is exercised and by whom, and how the context will shape the drivers the use and dissemination of data and statistical evidence.
Understand gender dimensions of development, in particular relating to sex disaggregation of data and the collection of sensitive data on gender issues. Compliance with the Gender Equality Act (2014). Further they should have a good understanding of the importance of wider data disaggregation and the importance of this for inclusive development.
Understand economic concepts and policy, including the importance of good quality economic data for economic planning.
Understand the potential of digital technology to improve the reach and value for money of development interventions. Integrate digital effectively in
Page 3
programmes, and use digital tools to work smarter, and to collaborate and communicate effectively .
The cadre specific competencies are underpinned by the principles of the civil service, international, and where appropriate the Policy Profession skills and knowledge frameworks. These civil service wide competencies provide the foundation upon which the DFID technical competencies are built. In the delivery of technical expertise advisers must demonstrate the core values of the civil service and the behaviours set out by these broader frameworks. Advisory Grades The Civil Service competency framework provides a short narrative and examples of the desired and less desired behaviours of civil servants at all the different grades. This provides a good basis for determining grades in terms of generalist competencies and many of the descriptions also hold true for technical activities. As in the case of Generalist Civil Service competencies Technical Evaluation competencies are not different for different grades. What changes by grade are the desired behaviours and context for the technical knowledge required and the levels of risk, complexity and responsibility they will be expected and able to cover. 1 For example: B1/A2L advisers will meet the key competency areas and be able to provide sound technical advice in the areas they have experience of. They will be able to design, manage and lead clearly defined projects, policy issues and technical processes such as dialog and stakeholder engagement with lower levels of risk and complexity. A2 advisers should be able to lead project and programme design, management and oversight and be able to operate at country and or departmental level independently. They should be aware of the relevant evidence in their technical area and be able to provide sound technical advice and support and lead technical discussions. This will include some level of setting direction and technical strategic leadership not just narrow technical guidance. A1 technical advisers are expected to be able to represent DFID in technical discussions at the highest levels. Be able to design, manage and oversee the most complex, high risk and high value projects and advice on complex and tricky technical issues. They will be able to apply their generalist leadership capabilities such as setting direction in their technical work.
Evaluation Managers in DFID DFID’s programme and project delivery model relies on all staff having an awareness of monitoring and evaluation when designing and implementing 1
Some cadres will maintain a guide to how to make the judgement between different grades in practice. These are advisory not a check list.
Page 4
programmes. DFID’s evaluation accreditation model is designed to enable staff (generalists or advisers of other cadres) whose primary role or career is not in monitoring and evaluation to develop their skills and to get recognised for their skills and contribution to the M&E agenda through accreditation. It also allows spending units to assess and consider the level of skills available to implement their monitoring and evaluation portfolio. Some Evaluation Managers may seek to build a career in monitoring and evaluation while others may use it to complement existing skills base. DFID’s Evaluation Managers will have:
Awareness of different monitoring and evaluation approaches and methods to address stakeholders varying evidence needs; Knowledge and ability to engage with DFID’s monitoring and evaluation commissioning processes; Ability to manage DFID’s monitoring and evaluation projects and ensure timely, relevant findings to feed into programme decisions; Ability to articulate and demonstrate clear results, impacts and value for money, even where harder to measure, and play a key role in brokering the acceptance and understanding of evaluation findings. Skills to build networks and partnerships with other donors, partner governments, civil society, policy and research communities to leverage greater results and impacts. Work flexibly across and apply their skills to DFID’s monitoring, evaluation, results and VfM agendas to support programme delivery and to ensure DFID has evidence of the impact of its programmes
Evaluation Accreditation for Evaluation Managers (Levels 1-3) Evaluation Manager accreditation is a technical assessment of staff M&E skills and competencies and it is not linked to existing grades. Staff seeking accreditation at L1-L3 will only be asked to demonstrate Primary Competencies at the point of application. However, they can offer to demonstrate a Specialist Competency in addition to the Primary Competencies. They will not need to demonstrate Advisory Competencies (see Competency Summary on p.6). Those applying for Level 3 accreditation will also need to meet the Minimum Entry Requirement (as set out on page 7). There is no minimum entry requirement for Levels 1-2. DFID’s evaluation accreditation for Evaluation Managers is not linked to grade or post. All staff with the relevant skills and competencies can apply for accreditation and accreditation will be awarded based on the skills demonstrated at the following levels: Level 1: Foundation Level 2: Competent
Basic understanding and some but limited experience of applying specific methods or managing monitoring and evaluations. Good all round understanding of evaluation with relevant practical experience, but still needs to develop depth and breadth.
Page 5
Level 3: Skilled
Significant specialist theoretical and practical expertise in evaluation, including on technical methods. Has led the process of commissioning, designing and managing of evaluations (or other comparable research projects). Has closely related analytical and research experience.
Level 3 accreditation is close to A2 Evaluation Adviser accreditation in terms of technical expectations. Most evaluation adviser roles will be opened to existing Evaluation Advisers as well as those accredited at Level 3, and it is expected that those accredited at Level 3 will demonstrate other Advisory Competencies as part of interviews for specific posts.
2. Competency Summary Type Advisory Competencies
Competencies A1: Advisory Competencies
Primary Competencies
P1: Best Practice in Evaluation Approaches and Methods
(i) (ii) (iii) (iv)
International Development. Using evidence to inform policy and programming. Design and manage programmes and projects Develop and Implement Policy
Understands and makes appropriate use of evaluation concepts and best practice in evaluation design, methods and approaches in line with departmental policy, and relevant international and professional technical standards.
Core Competencies
P2: Collecting and Analysing Data and Evidence for Effective Decision Making Collects and analyses appropriate and relevant evidence, data and information for monitoring and evaluation from a range of sources, identifying relevant material, assessing quality, spotting gaps and presenting information in a structured logical and useful way to feed into programme and policy decisions
P3: Communicating and Sharing Monitoring & Evaluation Findings, Knowledge and Expertise Improves influence and use of evaluations throughout the evaluation cycle, including communications, planning and dissemination, high quality advice on evidence uptake, training and guidance to stakeholders.
P4: Leading, Managing and Delivering Evaluations Manages evaluation resources and relationships with stakeholders including external partnerships to deliver high quality monitoring and evaluation plans and related objectives on time and to departmental standards.
S1: Specialist Evaluation Competencies:
Monitoring and evaluation for adaptive management Survey methods Impact Evaluation Econometric methods and modelling Qualitative design and analysis Monitoring, Evaluation and Results systems design
Page 6
Job Specific Competencies
M&E or statistics capacity building Digital tools and technologies for M&E Evaluation resource management and planning M&E and data collection in conflict and fragile contexts
Posts that are open to multiple cadres, additional competencies are likely to be tested. For posts open to statistics and evaluation advisers, some of the following GSS competencies are likely to be included: Acquiring data and understanding customer needs Data analysis Presenting data effectively
3. Competency Detail ENTRY REQUIREMENT for Evaluation Advisers and Level 3 Normally Master’s degree or equivalent postgraduate qualification in a relevant technical area but a first degree in relevant technical area can be sufficient, as can NVQ at the level equivalent to Master’s degree. Strong demonstration of the competences through relevant experience may substitute for a Master’s. A1: ADVISORY COMPETENCIES (i) International Development. Understand the processes, institutions and organisations that shape the changing international development context in which DFID operates, including the Sustainable Development Goals, ODA rules, UK legislation and government structures, and the roles of multilateral organisations and processes in development.
(ii)
Use evidence to inform policy and programming. Analyse and critically appraise a wide range of data and evidence, design projects and engage with and implement high quality evaluation processes in a way that enables DFID to learn from its projects and programmes.
(iii)
Design and manage programmes and projects. Have skills to undertake the Senior Responsible Owner role, including working with partners/suppliers, understanding and practising risk management, financial management, commercial awareness and value for money at all stages of the programme cycle .
(iv)
Develop and implement policy including open and collaborative policy making and scaling-up policy solutions. (defining policy as the agreed principles and approach we apply when addressing an issue, problem or challenge.)
PRIMARY COMPETENCIES Competency Best Practice in Evaluation Approaches and Methods Collecting and Analysing Data and Evidence for Effective Decision Making Communicating and Sharing Monitoring and Evaluation Findings, Knowledge and Expertise
Enabling Evaluation Advisers to: Design high quality, timely and influential monitoring and evaluation systems for projects and programmes, utilising global best practice. Assess quality of data, analyse results from monitoring and survey outputs. Design and advice partners on indicators and data collection processes and instruments. Support evidence uptake, communicate monitoring and evaluation findings, translate and use the result to influence programme and policy decision. Advice and train colleagues and stakeholders on monitoring and evaluation.
Page 7
Leading, Managing Delivering Evaluations
and
Design monitoring and evaluation activities that support programme and policy decisions. Set up governance and management arrangements for these and ensure monitoring and evaluation in DFID is done to high quality and complies with ethical and other departmental standards.
SPECIALIST COMPETENCIES The list of specialist competencies (see page 6) will be updated annually as DFID business needs change. It is intended that each Evaluation Adviser will demonstrate one specialist competency in addition to all the primary competencies and advisory competencies.
Page 8
Best Practice in Evaluation Approaches and Methods Understands and makes appropriate use of evaluation concepts and best practice in evaluation design, methods and approaches in line with departmental policy, and relevant international and professional technical standards. Best Practice in Evaluation Approaches and Methods
Evaluation Adviser Provides expert and strategic guidance to others (in DFID and to key external stakeholders) on all aspects of evaluation design, methods and approaches. Helps to develop monitoring, evaluation and research methods and approaches and pushes forward the frontiers of best practice in methods and approaches with partners, including through engagement with evaluation and other researchers. Demonstrates expert knowledge and application of a range of different quantitative and qualitative methods and approaches and where necessary proposing and testing innovative approaches. Recognised leader in DFID or externally on evaluation methods, approaches and designs. Ability to establish credibility on methods and approaches at senior levels and influence internal and external partners. Able to apply methods and approaches to support programme and policy design at all stages of programme management.
Level 3: Skilled Demonstrates advanced and specialist knowledge and ability to apply and interpret a range of evaluation and research methods and approaches. Demonstrates advanced knowledge and application of several quantitative methods used for evaluation and research. Demonstrates advanced knowledge and application of several qualitative methods used for evaluation and research. Understands the importance and relevance of multiple and mixed methods in evaluation and is able to bring a range of multidisciplinary approaches to bear in evaluation design, depending on context. Demonstrates ability to prioritise and focus evaluation questions (and negotiate these with partners) to produce relevant and useful evaluation. Recognises, addresses and escalates complex ethical issues at different stages of evaluation.
Level 2: Competent Demonstrates a good knowledge and understanding of various approaches and methods in evaluation and research. Has evidence of practical application of these. Demonstrates a strong understanding and application of either qualitative or quantitative evaluation or research methods. Understands the importance and relevance of integrated mixed methods in evaluation and research, and applies them appropriately.
Level 1: Foundation Demonstrates knowledge and has evidence of practical application of at least one evaluation or research methodology. Has participated in designing, quality assuring or reviewing at least one type of evaluation approach or methodology, and can communicate and explain this to a non-technical audience.
Page 9
Collecting and Analysing Data and Evidence for Effective Decision Making Collects and analyses appropriate and relevant evidence, data and information for monitoring and evaluation from a range of sources, identifying relevant material, assessing quality, spotting gaps and presenting information in a structured logical and useful way to feed into programme and policy decisions Collecting and Analysing Data and Evidence for Effective Decision Making
Evaluation Adviser Demonstrates expertise in a range of data collection techniques used in evaluation and research, and has experience of implementing several of them in the field. Confidently and competently analyses data from diverse sources, identifying key trends and gaps in information, and is able to quality assure data collection mechanisms for evaluation. Is able to lead the design and quality assure a range of different types of inception and baseline reports. Expertly uses analytical skills to synthesise data and present well founded results /recommendations to support higher level strategic decision making on evaluation or programme design. Can confidently provide leadership and advice to others on data collection and analysis, including on innovations in data collection.
Level 3: Skilled Demonstrates substantial knowledge of a range of data collection techniques used in evaluation and research, and has experience of implementing some of them. Confidently and competently analyses data from diverse sources, identifying key trends and gaps in information. Uses analytical skills to synthesise data and present well founded results /recommendations to support higher level strategic decision making on evaluation or programme design. Is able to design and quality assure inception and baseline reports. Leads by example, ensuring evidence, data and information used in evaluation is appropriate, accurate, high quality and collected and analysed ethically.
Level 2: Competent Demonstrates a good knowledge of some data collection techniques used in monitoring, evaluation and research. Is able to confidently advice on the design of data collection for monitoring, evaluation and research. Demonstrates good attention to detail and analytical skills by reviewing data quality for content and accuracy. Uses analytical skills to contribute to evaluation process. Works on preparing data for analysis, such as testing for data accuracy, with minimal guidance.
Level 1: Foundation Demonstrates knowledge and ability to apply some data collection techniques used in monitoring, evaluation and research. Is able to advice on the design of data collection for monitoring, evaluation and research. Is able to use some analytical techniques to analyse data and information that is collected. Provides constructive feedback on the quality and standard of information.
Page 10
Communicating and Sharing Monitoring and Evaluation Findings, Knowledge and Expertise Improves influence and use of evaluations throughout the evaluation cycle, including communications, planning and dissemination, high quality advice on evidence uptake, training and guidance to stakeholders. Communicating and Sharing Monitoring and Evaluation Findings, Knowledge and Expertise
Evaluation Adviser Level 3: Skilled Demonstrates highly developed skills Confidently develops, delivers and advises others on in developing evaluation communication plans for recommendations and applying them monitoring and evaluation. to key areas of DFID’s work, relating effective them to programmes and policies in Demonstrates communication skills when a practical and effective way conveying information on Presents difficult evaluation and evaluation processes, ensuring evidence results with authority and the message is clear, concise confidence and objectively. and suitable for the recipient’s Uses evidence to support and shape level of knowledge and skill. future strategy, communicating key sound trends and recommendations within Demonstrates judgement when advising DFID the department and to external and partners on evaluation partners. policy and procedure, taking Is seen as a leader in the field and into account complex and regularly presents monitoring, sensitive issues. evaluation and research papers, Confidently presents on delivers training and/or represents monitoring, evaluation or DFID in both internal and external research at internal and meetings/conferences. external fora to inform policy Uses highly developed influencing and programme decisions. skills to encourage others within supports the DFID and external partners to build Confidently delivery of training for other in evaluation processes at the DFID/HMG staff on monitoring, inception of a project or policy, and evaluation or research. to continually monitor performance Demonstrates influencing and throughout the project and policy negotiating skills to ensure cycle. DFID and partners promote Demonstrable ability to build sound evaluation processes capacity and quality of partners in throughout the duration of monitoring and evaluation, and projects and programmes. related fields (e.g. statistics).
Level 2: Competent Confidently defends evidence based results, stands their ground and has conviction when delivering the message. Confidently advices colleagues and stakeholders on use of evaluation guidance, protocols and processes. Able to develop a communication plan, tailored to the specific audience and needs. Able to translate and communicate technical information to non-technical audiences. Understands factors which strengthen the influence of evaluations and applies these in own work and when advising others.
Level 1: Foundation Proactively seeks new opportunities to communicate and share monitoring and evaluation best practice within DFID and stakeholders. Demonstrates commitment to ensuring own knowledge is up to date. Works jointly with partners to achieve shared evaluation outcomes. Uses a range of evaluation and research communication tools to promote evaluation and achieve impact. Understands how the evaluation process can influence the use of evaluations, and provides appropriate, timely and focused evidence to influence programme decisions and policy development.
Page 11
Leading, Managing and Delivering Evaluations Manages evaluation resources and relationships with stakeholders including external partnerships to deliver high quality monitoring and evaluation plans and related objectives on time and to departmental standards. Leading, Managing and Delivering Evaluations
Evaluation Adviser: Has significant experience in leading all aspects of the processes of commissioning, designing and managing a diverse range of different types of monitoring and evaluation plans (or other comparable research projects). Demonstrates project management skills to a senior and strategic level, ensuring appropriate resource use. Works effectively to establish annual and long term monitoring, evaluation and research programme strategies, set objectives and standards, and negotiate key milestones and budget. Demonstrates expert leadership and management skills by ensuring that all members of staff are supported and feel confident to use their level of authority effectively to achieve results. Displays expert skills in managing and negotiating expectations of multiple constituencies (e.g. World Bank, NGOs, partner Governments). Confidently manages monitoring, evaluation and research programmes for DFID and OGDs and ensure evidence uptake.
Level 3: Skilled Has led the processes of commissioning, designing and managing a range of different types of evaluations (or other comparable research projects). Able to prioritise and focus evaluation questions (and negotiate these) to produce relevant and useful evaluation. Able to quality assure evaluation products for others. Able to set up management arrangements for evaluations (including ToR and contacts) to ensure independence, ethics and quality. Expert application of Smart Rules and project management cycle to all aspects of evaluation work. Establishes and assesses risk, builds and applies contingency plans in order to mitigate risks in the delivery of monitoring and evaluations. Establishes relevant stakeholder groups, and consults, informs and manages a range of expectations. Able to confidently chair steering or reference groups, or evaluation management groups.
Level 2: Competent Level 1: Foundation Develops high quality Participates in joint terms of reference to and internal deliver complex evaluations of evaluations projects, incorporating all programmes and relevant aspects of topics, providing evaluation design support and working Demonstrates people with others with management and more experience and relationship knowledge. management to Has a good maintain momentum knowledge of DFID’s and complete evaluation deadlines. commissioning and Sets up and management participates in processes, and is management and able to apply them. governance Works collaboratively arrangements for with third parties (eg evaluations. NGOs and external Supports consultants) ensuring improvements in how all parties are fully evaluations in DFID aware of their are managed to responsibilities and maximise influence key goals/deadlines and use. are met. Knows and Makes appropriate applies professional use of global and evaluation standards, DFID evaluation including ethical guidance and standards, and frameworks. ensures contractors abide by them.
Page 12
Page 13