Analysis, Review/Self-Audit: Measuring Progress and Sustaining Outcomes

“Mathematics” and “measurement” are words that send many people scurrying to find cover, but in the world of organizational change, numbers play an important part in helping YSOs to gauge progress toward their goal of keeping children and youth safe. Consider, for example, beginning a weight loss or fitness program. Without periodically collecting numbers like weight, inches, or heart rate and blood pressure, how would a person determine if he or she was making consistent progress toward the goal of better health? Similarly, without any type of measurement, it would be difficult to figure out if a person’s efforts to make changes in diet and/or exercise were having the intended effects. Numbers collected over time can tell us if we’re heading in the right direction and – later – once we (hopefully) reach the desired goal weight, waist size or heart rate, sustaining the accomplishment into the future is likely to depend on continued, periodic measurement. The same can be said for the programs, changes and goals that YSOs set in place to keep children safe.

The overall goal of the initiatives described in this report is one of primary prevention, i.e., to create and help to facilitate an environment in the youth-serving organizations, businesses and institutions of Massachusetts that, once in place, would serve to protect children and youth by preventing child sexual abuse before it occurs. A second goal is that if a child or youth in the care of a YSO becomes the target of sexual abuse or human trafficking/sexually exploited child, that child or youth would have knowledge sufficient to distinguish safe from unsafe touching and relationships, and know what to do if unsafe touch occurs – including how and when to seek assistance from a trusted adult – and have the skills and language to report the abuse. A final goal is that, should child abuse or neglect be suspected, observed or disclosed to any administrator, supervisor, staff member, employee, or volunteer, that individual would possess the knowledge, information, and resources to ensure it is reported to the appropriate organizational and civil authorities according to the reporting laws of the Commonwealth of Massachusetts.

Given these goals, the investment in time and effort necessary to put a safety framework together, and the need to provide feedback to the organization, the following questions will naturally come up:

• How do we know if it’s working?
• Are children and youth in the YSO any safer now than they were before these new programs and structures were put into place?
• Is the YSO more capable of keeping children safe?
• If they are safer, what were the primary factors that had the greatest effect?
• If they are not safer, what factors lead to this conclusion and what needs to be changed, corrected, strengthened or eliminated?
• How effective are the YSO’s existing compliance and audit efforts?
• An essential element of developing, implementing, and sustaining a successful child abuse prevention framework is data – data about the ongoing programs, how they are being integrated into the YSO, what is working, what is not working, and what needs to be improved. The questions can be “qualitative” (as above) seeking answers that are more subjective, or “quantitative” (as below) seeking objective information like numbers, percentages and quantities that can help to gauge progress.
• How many people are there in the organization (and in each job classification) who require the safety training?
• How many have received the required training?
• How many are left to train?
• Have the safety materials been distributed?
• Has everyone who received the required training also received a background and criminal history check?
• Are the safety curricula (if appropriate) being taught to the children/youth?
• Have all staff, employees and volunteers received and signed the Code of Conduct?
• How many reports of suspected or alleged abuse and neglect have been filed?
• Have the reports been handled correctly (i.e., within the required timeframes)?

Depending on the size of the YSO, the data to be collected and analyzed – or even simply summarized – could be considerable. Therefore, no matter what the size of the organization is, it is important in the early stages of building a child safety framework to also think about the kinds of questions that need to be periodically answered, and the data that need to be collected to answer those questions. 

Organizations can use simple checklists with “yes” or “no” responses, and a tally of the numbers of employees and volunteers who have attended the required training, signed the code of conduct, and complied with other requirements set by the leadership. This allows a YSO to gather basic, numbers-oriented information about the implementation of the safety program, and then use those numbers to provide feedback to both leadership and staff about progress, and the level of compliance with the new program requirements.

The numbers can also be used to keep parents and others informed about what the YSO does to ensure their child’s safety before enrolling them. A YSO that does basic data collection can say to parents “We are committed to child safety: We have a set of published safety policies and procedures, and all our staff and volunteers are screened. All members have a signed code of conduct on file, and have been trained in child/youth safety, the prevention and recognition of maltreatment, and in their abuse reporting responsibilities under Massachusetts law”.  Hearing a statement like this, and/or seeing a statement to this effect on YSO brochures and materials can help to put parents more at ease. A sample “Child Sexual Abuse Prevention Evaluation Tool” that can be used to tally the required safety elements in a comprehensive safety program is provided in the Resources section below.

Changes in staff practices and behaviors are a bit more complicated to measure and rely more on ongoing observation and monitoring to assess improvements (see also the section on Code of Conduct). A good rule to remember is that all improvement requires change, but not all changes lead to improvement. Unless there is an ongoing commitment to gauge staff attitudes about the desired changes, and to gather information about the behaviors that will indicate whether those attitudes are being translated into the desired actions, YSOs may be tempted to stop measurement altogether when the policies are published and everyone has been trained. This does little to determine if the changes have been adopted into the culture of the YSO.

Some basic tools – both introductory and advanced – that exist for the measurement and evaluation of child abuse prevention programs are listed on the website of the Child Welfare Information Gateway. ¹ An annual data collection process or “self-audit” can help all YSOs to understand (and explain) the basics of how the implementation is progressing. A simpler variation of the Child Sexual Abuse Prevention Evaluation Tool mentioned above entitled “Sample Self-Audit Form for YSOs” is included in the Resources section below. More comprehensive data collection can also enable them to measure organizational, cultural, and behavioral change, and whether or not what is being implemented supports the desired outcome of making the environment safer.

Some YSOs may want only to collect information about program implementation, and basic information about the number and types of “internal” allegations encountered during the year (i.e., the number of reports filed involving staff, volunteers or other children and youth) and how they were handled.  Other YSOs may want (or need) to take a broader look to answer questions about the effectiveness of their programs from a wider variety of stakeholders like funders, insurers, boards of trustees, etc.

For example, a broader perspective would consider that since the YSO has been working to build or improve its child protection policies and procedures, modify organizational structures to respond effectively to abuse reports, update codes of conduct, and educate adults (and children if appropriate) about how to recognize dangerous situations and report them, that all types of abuse and neglect are likely to be reported – including domestic, extended family, dating, and neighborhood situations.

In terms of measuring the effectiveness of what the YSO has instituted, initial assessments could focus on gathering data on the numbers and types of reports being filed, how (and through whom) they came to the attention of the YSO (observed, suspected, self-disclosed), how quickly they were reported to the YSO’s reporting authority and/or to DCF, whether those to whom the abuse was reported knew what to do, and any information about outcomes of cases. The data can be collected without using the names or other identifying information, but should be treated as confidential and kept in a secure location. Asking respondents to also supply examples of observed “safe” behaviors among children and youth, or increased awareness of the safety rules, would underscore their response with additional anecdotal information. An example of a basic “Incident Report” is included in the Resources section below.

In terms of behaviors and desired outcomes, one could argue that by collecting data of this type on an annual basis, a YSO could have an ongoing awareness of the numbers and types of abuse and neglect cases its children are experiencing, and could at least have some indication that children are learning the skills, have used those skills to report to a trusted adult within the YSO community, and that the adult knew what to do and did it within the required timeframes as trained. Again, the focus is on the behaviors and skills that we would hopefully expect to be outcomes of the safe environment efforts. Certainly, there are more sophisticated analytic and research-oriented methods for determining program effectiveness, but the data described above can be collected with minimal effort. The more complex program effectiveness analyses should not be attempted without appropriate expertise and guidance.

In addition to an annual self-audit, it is strongly recommended that the policies and procedures themselves be evaluated periodically (every 2-3 years). This comprehensive analysis is primarily qualitative and is designed to interview individuals and groups who have carried out the various responsibilities described in the policies, and to get their feedback about how what is written (and intended) is actually working in practice. The product of this analysis will be a document that assesses each functional area of the policies and makes recommendations for changes and updates to the policies and procedures that:

• Reflect the way a policy area or procedure has evolved in its practice that is contrary to what is written, but is working successfully
• Restructure the text to address weaknesses or failures in a policy and/or procedural area and make them more effective or efficient
• Strengthen and/or clarify policy sections or procedures
• Develop policies and procedures for new issues that emerge from the analysis

In this way, a YSO can identify the policy’s strengths and weaknesses, uncover issues that were not anticipated or addressed adequately, and change/update them accordingly. Once updated, the areas of recommended improvement can be followed and observed for a period of time (say, 6 months) to determine their effect (and quality improvement) on the overall policy implementation.

In summary, measurement/data collection recommendations in a YSO’s child protection framework can also serve to help to sustain the programs over time and should include:

• An annual self-audit to assess current state of the implementation, to include a checklist of the data to collect
• A comprehensive analysis of the organization’s policies and procedures every 2 – 3 years designed to identify what is working in practice, what needs improvement, or what needs to be added to the policies and procedures in order to strengthen them
• A “continuous quality improvement” initiative that takes a subset of any identified policy or practice improvements, addresses them, follows their implementation for a period of time and issues a report.

---

¹ (https://www.childwelfare.gov/topics/preventing/evaluating/)