Guest Column

Don't skimp on evaluation

Make sure changes are effective

By Patrice Spath, RHIT
Brown-Spath & Associates
Forest Grove, OR

It is pointless being really busy improving your case management services if you don't also make sure that the changes being made are as effective as they could be. You also need to be sure that you are making the right changes. This is where evaluation becomes important. Evaluation is all about checking that you are doing things right and doing the right things. Without evaluation, the value of new case management initiatives cannot be adequately judged. Evaluation will help to determine whether your goals have been achieved as well as look at what worked best. Evaluations can be small and simple, as well as complex. They need not be labor-intensive, particularly if they use routinely collected information.

Evaluation of your improvement activities, however small, also helps spotlight the achievements of the case management department. Through dissemination of findings, accomplishments can be recognized and shared with others. In addition, the findings from many small evaluations in various departments provide a wealth of information about service improvement, both what works within case management services areas and across the entire organization.

Often, the most challenging part of evaluation is determining what to measure. Your choice of measures is influenced by what you are trying to achieve. There are three different areas of interest for an evaluation. They are:

  • Project monitoring: looking at the routine functioning of your improvement work. Is it doing what you wanted it to do?
  • Process evaluation: looking at the way in which your improvement work is implemented and runs. What can you learn from the process?
  • Impact evaluation: looking at whether your improvement work is delivering the objectives set. Are you getting the outcomes you planned for?

These areas are not mutually exclusive. They can each be the sole focus of an evaluation or can be combined for the evaluation. It is important to be clear about the focus of the evaluation from the outset.

At the planning stage, you also need to be clear about what information is required for the evaluation. There may be routinely collected information that can be used, or additional information may be required. If nonroutine data are to be collected, you'll need to determine whether the information can be gathered with sufficient reliability and whether the people assigned data collection responsibilities have adequate time to do so given their other job demands.

To ensure your data collection efforts provide worthwhile information, a number of issues must be address prior to the start of the evaluation. Start by developing a plan for evaluating the activity. For instance, you may have recently expanded case manager coverage to the weekends. Prior to this time, case managers were only on-call during the weekend. Now a case manager is in-house during the day on Saturday and Sunday. You'd like to evaluate this change in service. To help get you started on the evaluation, develop an evaluation plan that considers:

  • the questions to be answered;
  • the data to be collected, together with methods of collection;
  • how long the evaluation will take;
  • who will do what in the evaluation;
  • how the results of the evaluation will be disseminated.

Be clear about the data needed. Where possible, data that are routinely available should be used. Specific data may be required for the evaluation which is not already collected routinely by case managers. It is critical that a practical approach to collecting the data is developed and that those collecting the data are able to collect it in a way that does not impact on their day-to-day work. Your evaluation can fail if this has not been thoroughly considered.

There are several approaches that can be chosen to evaluate your improvement activity.

Consider the following four questions in determining the type of evaluation you need:

  • Do I want to assess the impact or outcome of the new or changed case management process against stated goals, or do I want to assess the impact by evaluating intended and unintended effects?
  • Do I want to assess whether outcomes are directly due to the new or changed case management process, or do I want to explore what is affecting outcomes?
  • Do I want to measure the success of the new or changed case management process by judging it against the goals of a single stakeholder (e.g., case managers) or against the goals of a wide range of stakeholders?
  • Do I want to determine whether the new or changed case management process actually works as expected, or do I want to determine where improvements are needed?

In preparing the design of the evaluation, there are several key areas to consider, each of which will shape the design of the evaluation and methods used. You may wish to design a summative evaluation. This is a one-time evaluation used to make a judgment about the success of an improvement project to show whether the project worked and whether it met its goals. Key questions might be:

  • Did the improvement work achieve its objective?
  • What improvements were created?
  • What benefits did the project deliver compared to what it cost?

A formative evaluation is ongoing. This type of evaluation looks at the new or changed case management process as it evolves and suggests ways in which it can be improved. The emphasis in a formative evaluation is to determine why or how a project produces specific results. Key questions might be:

  • What have we learned?
  • What were the drivers for change?
  • What were the obstacles for change?
  • How did the initiative change over time?

Once your evaluation plan is formulated, explore the plan with the case managers as well as other stakeholders. Get everyone's input into the key aspects of the evaluation and who is responsible for what.

If you set out knowing what you want to achieve with new or changed case management processes but don't plan effectively, it will take a very long time to achieve your goals. You may never actually realize your improvement objectives. Planning for the evaluation of your improvement project is just as important as the planning that goes into making changes. Evaluation is an essential and regular part of all improvement activities. It will help you determine whether your aims have been achieved as well as point out what additional improvements are needed.