An Indiana research compliance program found that IRB staff and board assessments help keep the program on track and running well.
“The primary benefit of assessments is to help us identify problems we were not aware of and to confirm things we think we’re doing well to keep on doing them that way,” says John R. Baumann, PhD, associate vice president for research compliance at Indiana University in Indianapolis. The institution has seven IRBs with approximately 160 IRB members, including six chairs.
The following is an example of the assessment’s use in identifying issues: IRB staff and member assessments of one IRB chair indicated a problem.
“We had a feeling the person wasn’t performing that well as a chair,” Baumann says, adding that the assessment feedback confirmed there was a problem.
“That person is no longer a chair,” Baumann notes.
Assessments also can point out systemic problems. For instance, the research compliance office responded to an apparent IRB office crisis by surveying researchers anonymously about their experiences with the IRB.
“A few years ago, the office was in a real crisis; staff were demoralized, researchers were unhappy with us,” Baumann recalls. “The IRB’s turnaround and throughput were bad, so quantitatively and qualitatively, our measures were not very good.”
The first survey had a good response rate and made it clear that the IRB had to make changes. “Our rate of satisfied or very satisfied was 40% to 60%, depending on the question asked,” Baumann says.
After the IRB made changes and corrected what researchers said was not working well, follow-up investigator surveys improved dramatically.
“We made a lot of changes, and it went above 90% who said they were satisfied or very satisfied,” he says.
The organization originally began to create assessments as part of an AAHRPP accreditation standard that required that IRB members and staff be periodically evaluated and given feedback.
This is accomplished via a variety of mechanisms, including member self-assessment, assessment of IRB members by IRB staff, of IRB staff by IRB members, and researchers assessing the IRB process.
The staff’s survey of IRB members was designed to provide insight into the working relationship between IRB staff and individual board members, Baumann notes.
“The first time we had the staff evaluating IRB members was in 2013,” says Shawn Axe, CIP, director of the human research protection program in the office of research compliance at Indiana University.
Axe and Baumann explain how the assessments work:
- Staff assessment of IRB members and IRB members’ assessment of staff. “We evaluate each IRB member based on how often they worked with the member on five or more areas,” Axe says. “We ask how well they understand the federal regulations and how well they apply them.”
The assessment also looks at how robust their documentation is and how accessible the member is.
The staff’s evaluation of IRB members is scored from one — extremely unsatisfied — to 5 — extremely satisfied. IRB employees are advised to not score members unless they are familiar with them.
The following are the scoring items:
- ability to apply federal regulations, ethical principles, and IU IRB policies and procedures to research;
- completion of reviewer requirements: completed reviewer checklists, clear provisions (changes requested by the IRB reviewer to the IRB application), and presentation to IRB;
- accessibility: willingness to review minutes, willingness to consult with staff/investigators, willingness/timeliness of review expedited or high-priority submissions, and willingness to serve as an alternate or attend off-cycle meetings; and
- working relationship with staff: positive interactions, supportive of staff requests, and responsive to communication.
Based on the survey IRB members complete about the staff, the IRB office has learned that its most effective educational format is the 10-15 minute presentation by staff at the start of board meetings, Axe says.
“We have general regulatory or ethical topics and a PowerPoint presentation, usually with an opportunity for questions and answers,” Axe says. “This month, the topic was on promptly reportable events, focused on unanticipated problems and noncompliance.”
Other topics have included regulatory updates, vulnerable populations, informed consent, recruitment, and regulatory/ethical compliance.
The assessments are anonymous and include room for comments. “That’s how we found out the concern over one chair was pervasive,” Axe notes.
- Researcher surveys about IRB experience. Researcher surveys are held twice a year over a four-week period. Survey invitations go out to each principal investigator of a study that is approved during those time periods.
“We ask a series of seven questions and we’re clear that we’re asking about their experience with a particular approval,” Baumann says. “Then we have additional questions about the time to approval, the new electronic system, or things like that.”
The researcher survey simply asks investigators to answer each of six questions, based on ratings from extremely unsatisfied to neutral to extremely satisfied. A seventh question is open-ended: “What can we do to improve these responses?”
Each question begins with “How satisfied are you with..:
- Your working relationship with the IU Human Subjects Office staff?
- The staffs’ pre-review/screening of materials for review?
- Your IRB Chair?
- Your IRB Vice Chair?
- The qualifications and performance of the board?
- How the IRB meetings are run?”
The narrow nature of the questions has made the ratings more useful, Baumann notes.
“It’s pretty much spot-on,” he says. “We don’t find very many petty comments or personal comments other than what is directly related to performance.”
The counterpart assessments sometimes reveal an expectation or communication problem.
For example, one board member’s assessment of IRB staff noted that staff’s pre-review missed issues in protocol submissions that the board member felt should have been found prior to the IRB meeting, Axe recalls.
But the problem was not as it appeared. The staff had done their jobs as they were instructed. They conducted the pre-review for the purpose of identifying regulatory omissions in protocols. The issues the board member raised were related to grammatical errors.
“Staff are not the grammar police and nor should they be,” Axe says. “These are not the things that hold up approval.”
So instead of changing how staff handled the pre-reviews, Axe spoke with board members about how to adjust their expectations of what the pre-review is intended to accomplish.
Perform Self-assessments
- IRB self-assessments. “We’ve had self-assessment by IRB members and confirmation by chairs since we’ve been accredited,” Baumann says. “Every IRB member is required to complete a self-assessment.”
The self-assessments include staff-added performance data.
The IRB chair meets with Baumann or Axe and reviews the board member’s self-assessment and the IRB staff’s assessment of that member to confirm the member is performing as expected, Baumann says.
Self-assessments are identifiable, and assessments also include a confidential set of questions that are not identifiable about the IRB chair, vice chair, and staff. “We have everyone reflecting and assessing everybody,” Baumann says.
The self-assessment form, like the other assessments, asks for ratings from extremely satisfied to extremely unsatisfied.
Members are asked to describe “How satisfied are you with the following:
- Your knowledge of the federal regulations and ethical principles in research;
- Your ability to apply the federal regulations and ethical principles in research;
- Your knowledge of the IU IRB policies and processes;
- Your ability to apply the IU IRB policies and processes;
- Your attendance at IRB meetings;
- Your availability for conducting expedited reviews; and
- Your participation in IRB meetings.”
There also are several open-ended questions, including:
- What can we do to help you improve as an IRB member?
- What additional educational resources may we provide?
- Any additional comments?
The staff pull data about each IRB member’s meeting attendance, CITI completion, total new study reviews for the full and expedited boards, and other information.
Most assessments are positive, and they’re given positive feedback via email. “We might email to say, ‘Everything looks good! Here’s your appointment letter for the next round,’” Baumann says.
When IRB chairs and Axe meet with IRB members or staff about their assessments, it’s usually because there were red flags that have to be addressed face-to-face.
“It’s always done with respect to confidentiality,” Baumann says. “It’s all designed around, ‘How can we make you a better IRB member?’”
