Evaluation is the 4th stage of Evidence-Based Practices (EBP) implementation and is core to identifying strengths, opportunities, aspirations, and results. From key tasks to tips and resources, the following documents guide agencies through this critical stage. For early adopters, stage 4 is an opportunity to assess your implementation and identify areas to be revisited. 

 Implementation of Stage 4: Evaluation

The 4th stage of implementation, evaluation, is a stage that should be revisited often because it informs program effectiveness. This stage takes approximately 12–18 months and focuses specifically on:

• Performance management
• EBP policy and practice alignment
• Gap analysis and service matrix
• Sustainability check

The Implementing Evidence-Based Practices: A Staged Approach—Stage 4 document details an overview that includes a chart with the stage’s key tasks, subtasks, implementation tips, and resources. Also included in the document are an implementation plan template, a cost-benefit analysis worksheet, and a fidelity and quality assurance checklist.

Resource: Implementing Evidence-Based Practices: A Staged Approach—Stage 4

 Gap Analysis

A gap analysis is a systematic method of assessing an organization’s performance. Essentially, it answers three key questions:

• Where are we now?
• Where do we want to be?
• How do we get there?

The five-step model outlined in Performing a Program and Gap Service Analysis aligns most closely with the needs of justice system agencies:

1. Define the focus
2. Assess current needs
3. Determine available programs/services
4. Identify gaps
5. Create a plan

A thorough gap analysis can and should be regularly conducted on processes, policies, practices, programs, personnel, services, staffing, training, and work culture.

Resource: Performing a Program and Gap Service Analysis

 Data and Continuous Quality Improvement

To develop and maintain systems that consistently yield better outcomes, policy discussions and decisions should be based on credible data. Internal collection, analysis, and sharing of data helps agencies evaluate effectiveness, efficiency, satisfaction and quality, and timeliness. Additionally, data guides continuous quality improvement. The EBP Brief: Evidence-Based Practices Research and Data Series: Collecting, Analyzing, and Sharing Data expounds on the importance of data and includes steps for developing performance measures and sample performance measures.

Using quality assurance (QA) and continuous quality improvement (CQI) processes, agencies are better able to improve performance and process outcomes, consistently and continuously. The EBP Brief: Evidence-Based Practices Research and Data Series: Continuous Quality Improvement includes continuous quality improvement activities and tips from experts.

Resources:
EBP Brief: Evidence-Based Practices Research and Data Series: Collecting, Analyzing, and Sharing Data

EBP Brief: Evidence-Based Practices Research and Data Series: Continuous Quality Improvement

Dr. Bryanna Davis, Quality Assurance Supervisor, Berks County

Pro Tip:

Stage 4 of EBP implementation focuses on evaluating current practices and how well an agency is adhering to fidelity. This is done by identifying process and outcome measures for each EBP. For example, agencies may have officers complete a test-case each year on the risk/needs assessment to ensure fidelity to the tool. To evaluate the use of motivational interviewing skills, a coding system, such as the Motivational Interviewing Treatment Integrity (MITI) 4.2.1, may be utilized.

At this stage, policies and procedures are also developed to evaluate those processes and data is collected and analyzed. This is where the data and spreadsheet-lovers in your department come in handy! By using data-driven evaluation processes, agencies can identify areas for improvement or where to focus booster training and, more importantly, showcase officers’ strengths in their use of EBPs.

Tips:

• Include staff in developing how processes and outcomes will be evaluated, especially your trainers.
• Explain to staff what data is being collected and why it is being collected.
• Share outcomes with staff at all levels and include infographics. (Who doesn’t like a pie chart?)
• Evaluate processes regularly and update as needed.

Stage 4 in Practice:

In Berks County, 18 staff were trained in the MITI 4.2.1 to evaluate the department’s use of motivational interviewing skills and they helped develop policy and procedure for this evaluation process. Through data collection and analysis of scores, we showed that nearly 80 percent of staff were competent in the skills of cultivating change talk, softening sustain talk, empathy, and partnership. We were also able to identify skills to focus on for future booster training sessions. These outcomes are shared with staff as encouragement and in full transparency of what the data being collected is being used for. Moreover, we plan to include these outcomes in our annual report to showcase to stakeholders the great work officers are doing to encourage positive behavior change in those we supervise.

Next Month:  Developing Professional Alliance 

Positive change is possible for everyone in Pennsylvania. We look forward to continuing to enhance your EBP knowledge and invite you to submit education/resource requests to askppcji@gmail.com.