Program Evaluation

The Five-Tiered Approach

Background: Family and youth preservation programs such as the S.E.A.L. are being held accountable for their effects by funders, policymakers, staff, and clients. However, most programs ( and we are no exception) have limited resources for conducting evaluations of our activities. To this end the five-tiered evaluation model is well suited to the needs of any program in general and to the S.E.A.L. Project in particular.

We have adapted this approach to help us plan and prepare for more sophisticated assessments as needed. Processes and outcomes are addressed throughout instructions that explain how to balance program characteristics, stakeholder interests, and the resources of the agency. The five levels include: needs assessment; monitoring and accountability; quality review and program clarification; achieving outcomes; and establishing impact.

The needs assessment stage includes hyperlinks to S.E.A.L. specific resources, and as well, involves tasks such as reviewing available data and identifying additional information necessary for setting goals for the program. The second tier, monitoring and accountability, builds on the needs assessment data to create procedures for data collection about clients, services, staff, and costs. Quality review and program clarification includes the collection of participants' opinions about the program and comparison of the program with standards and expectations. The fourth level begins to consider outcomes by focusing on short-term objectives that should be assessed and the identification of measurable indicators of success. Data at this level are analyzed to determine the effects of the program on people served. The final stage, establishing impact, utilizes more rigorous research methods with control groups to measure the impact of the program.

The accompanying guide - Making it Count: Evaluating Family Preservation Services. A Guide for State Administrators explains data sources and collection strategies, the ethics of evaluation research, the role of management information systems in evaluations, and the specific tasks necessary to complete each level of evaluation. Numerous references, numerous figures, numerous tables.

Level/Title

Purpose of Evaluation

Audience

Tasks

Kinds of Data to Collect/Analyze

Tier One - Needs Assessment

 

 

Guide Sections

  1. To determine the size and nature of a public problem

  2. To determine unmet needs in a community

  3. To propose program and policy options to meet needs

  4. To set a data baseline from which later progress can be measured

  5. To broaden the base of support for a proposed program

  1. Policymakers

  2. Funders

  3. Community stakeholders

  1. Review existing community, county, region and state data

  2. Determine additional data needed to describe problem and potential service users

  3. Conduct "environmental scan" of available resources

  4. Identify service, support and/or funding resource gaps and unmet needs

  5. Set goals and objectives for intervention

  6. Recommend one program model for range of options

  1. Extant data on target population; services currently available

  2. Interviews with community leaders

  3. Interviews or survey data from prospective participants

  4. Information about similar programs in other locations

Tier Two - Monitoring and Accountability

  1. To monitor program performance

  2. To meet demands for accountability

  3. To build a constituency

  4. To aid in program planning and decision-making

  5. To provide a groundwork for later evaluation activities

  1. Program staff and administrators

  2. Policymakers

  3. Funders

  4. Community stakeholders

  5. Media

  1. Determine needs and capacities for data collection and management

  2. Develop clear and consistent procedures for collecting essential data elements.

  3. Gather and analyze data to describe program along dimensions of clients, services, staff, and costs

  1. MIS (management information system) data; collected at program, county, and/or state level.

  2. Case material; obtained through record reviews, program contact forms, etc.

Tier Three - Quality REVIEW AND Program Clarification

  1. To develop a more detailed picture of the program as it is being implemented

  2. To assess the quality and consistency of the intervention.

  3. To provide information to staff for program improvement

  1. Program staff and administrator

  2. Policymakers

  3. Community stakeholders

  1. Review monitoring data

  2. Expand on program description using information about participants' views

  3. Compare program with standards and expectations

  4. Examine participants' perceptions about effects of program

  5. Clarify program goals and design

  1. MIS monitoring data

  2. Case material

  3. Policy Standards

  4. Other qualitative and quantitative data on program operations, customer satisfaction, and perceived effects; obtained using questionnaires, interviews, observations, and focus groups.

Tier Four - Achieving Outcomes

  1. To determine changes, if any, have occurred among beneficiaries

  2. To attribute changes to the program .   

  3. To provide information to staff for program improvement

 

  1. Program staff and administrators

  2. Policymakers

  3. Community stakeholders

  4. Funders

  5. Other programs

  1. Choose short-term objectives to be examined

  2. Choose appropriate research design, given constraints and capacities

  3. Determine measurable indicators of success for outcome objectives

  4. Collect and analyze information about effects on beneficiaries

  1. Client-specific data; obtained using questionnaires, interviews, goal attainment scaling, observations, and functional indicators

  2. Client and community social indicators

  3. MIS data

Tier Five - Establishing Impact

  1. To contribute to knowledge development in the field

  2. To produce evidence of differential effectiveness of treatments

  3. To identify models worthy of replication

  1. Academic and research communities

  2. Policymakers

  3. Funders

  4. General public

  1. Decide on impact objectives based on results of Tier Four evaluation efforts

  2. Choose appropriately rigorous research design and comparison group

  3. Identify techniques and tools to measure effects in treatment and comparison groups

  4. Analyze information to identify program impacts

  1. Client-specific data; obtained using questionnaires, interviews, goal attainment scaling, observations, and functional indicators

  2. Client and community social indicators MIS data

  3. Comparable data for control group

Source: Making it Count: Evaluating Family Preservation Services. A Guide for State Adminstrators.
Jacobs, F. H. Kapuscik, J. L. Williams, P. H. Kates, E.; Tufts Univ., Medford, MA.

Virginia Resources:

Services

Supports

Funding
   
  1. Source: Child Care Subsidy and Services