|Please let us know if you are using any of these measures — we would welcome the opportunity to share other professional learning materials with you!|
RESOURCES TO SUPPORT
As schools across the country continue designing, implementing, and evaluating Ci3T models of prevention, we are pleased to offer some new resources to support use of data to inform decision-making. Check back here for additional resources related to Ci3T measures. If you have questions or interests around measurement in Ci3T models, we encourage you to attend our monthly Ci3T Trainers and Coaches calls (visit ci3t.org/pl for additional information)
Ci3T PROFESSIONAL LEARNING SERIES MEASURESMeasures Overview TRANSCRIPT
The DEMOGRAPHIC FORM (updated in August of 2020) consists of thirteen items covering preferred pronoun, age, ethnicity, school employee years of experience and grade level(s) currently teaching (if applicable), primary school role, certification status, highest degree earned, classroom management course history, and professional development history with regard to academic and behavior screenings. This form now also includes questions in which people indicate their familiarity with Ci3T and what professional learning they have accessed. The demographic form is completed by all participating faculty and staff at each school site so that those designing, implementing, and evaluating the Ci3T model of prevention can be clearly described. This allows others who are interested in adopting a Ci3T model to make comparisons between their own faculty and staff and those of implementing schools.
The SCHOOLWIDE EXPECTATIONS SURVEY FOR SPECIFIC SETTINGS (SESSS; Lane, Oakes, & Menzies, 2010) allows school-based faculty and staff to identify behaviors that are critical for student success at their particular school. The survey is completed by all faculty and staff working at and/or supporting the school site. The survey is designed to gather information on three broad categories: respect, responsible, and best effort for seven school settings: classroom, hallway, cafeteria, playground, restroom, bus, and arrival/dismissal. The expectations were selected based on the most prevalent expectations of schools implementing positive behavioral interventions and supports (Lynass, Tsai, Richmond, Cheney, 2012). Each behavior is rated on importance using a 3-point Likert-type scale where 0 = not important for success in this setting, 1 = important for success in this setting, and 2 = critical for success in this setting. Scores for each item are aggregated and reported as the number and percentage of faculty and staff who scored the behavior as critical for success. The Ci3T school leadership team then uses the SESSS data to construct the first draft of the schoolwide expectation matrix so that the draft reflects the behaviors important to the majority of the school (see additional detail in Lane, Oakes, Jenkins, Menzies, & Kalberg, 2014).
SESSS (8.5" x 14" single page legal size PDF to print).
SESSS Large Format (8.5" x 11" multi-page letter size PDF to print).
SESSS Report Template (aggregate data reported back to school site; 8.5" x 14" multi-page legal size Excel file to print). SESSS Report Sample.
The KNOWLEDGE, CONFIDENCE AND USE (KCU) SURVEY is a multi-item survey for educators to rate their knowledge about, confidence in using, and usefulness of key constructs of the Ci3T model of prevention, and a brief assessment of their actual knowledge of these concepts. Self-reported KCU of items are rated using a 4-point Likert-type scale ranging from 0 (I have no KNOWLEDGE of this concept or strategy; I am not CONFIDENT in my ability to use or implement this concept or strategy; This concept or strategy is neither USEFUL nor relevant to my teaching) to 3 (I have a substantial amount of KNOWLEDGEabout this concept or strategy; I am very CONFIDENT in my ability to use or implement this concept or strategy; This concept or strategy is very USEFUL and/or relevant to my teaching), with higher numbers reflecting more knowledge, confidence, and perceived usefulness of the construct. A set of open-ended items are completed and scored as an assessment of actual knowledge. Educators complete the survey prior to engaging in a professional learning series, such as a full year training series to design a Ci3T model of prevention, after the completion of the professional learning series, and then once the new practice has been implemented (about three months after the start of implementation) to look for changes in KCU of constructs between designing and implementing the new practice. Pre, post, and follow up mean scores are computed for comparison. There are versions of the KCU for the Ci3T prevention model; low intensity strategies, classroom management, and instructional delivery practices (included in Lane, Menzies, Bruhn, & Crnobori, 2011); and for one systematic model of functional assessment-based interventions (Umbreit, Ferro, Liaupsin, & Lane, 2007).
Ci3T Version 2: Multiple-choice questions
Ci3T KCU Multiple Choice PDF to print.
Ci3T KCU Multiple Choice Report Template MS-Word document to edit and print.
Ci3T KCU Multiple Choice Qualtrics Survey and Report
FABI KCU Version 1: Write-in questions
KCU FABI Write-in PDF to print.
FABI KCU Version 2: Multiple-choice questions
KCU FABI Multiple Choice PDF to print.
The PRIMARY INTERVENTION RATING SCALE (PIRS) by Lane, Robertson, and Wehby (2002; Lane, Kalberg, Bruhn, Driscoll, Wehby, & Elliott, 2009) is an adapted version of the Intervention Rating Profile-15 (Witt & Elliott, 1985), a widely used measure of social validity. Social validity refers to stakeholders' views on the social significance of the intervention goals, acceptability of the intervention procedures, and the social importance of the intervention outcomes (Wolf, 1978). During the training year when a school’s Ci3T leadership team is designing their Ci3T prevention plan, school faculty and staff are encouraged to provide feedback on the first full draft of the primary plan using the PIRS. First, the team shares the draft primary prevention plan and answers any questions posed by faculty and staff. Then, those who had the opportunity to learn about the primary plan provide feedback. PIRS results are aggregated (i.e., average item and overall scores) and comments typed up to protect confidentiality before sharing back with the team. The Ci3T team reviews the results and responds with revisions to the plan (as appropriate) and information back to the faculty and staff on decisions made based on feedback. That is, some suggestions do not result in changes based on various reasons (e.g., feasibility, resources, or the science supporting certain aspects of the plan). In some instances, the narrative feedback provides direction on where additional professional learning is needed (e.g., understanding the difference between bribery and reinforcement; how to use tickets to reinforce behaviors that facilitate the instructional processes).
To complete, each individual from the school site who has reviewed the Ci3T primary prevention plan (or other primary prevention plans) rates his or her opinion on 17 items using a 6-point Likert-type scale ranging from (1) strongly disagree to (6) strongly agree, and then completes four open-ended questions regarding suggestions for changes to the primary prevention plan, perceptions of student performance as a result of implementing the primary prevention plan, and what is felt to be the least and most beneficial components of the primary prevention plan. The PIRS take approximately 10 minutes to complete. Please see Lane, Oakes, and Magill (2014) in the volume 58, issue 3, 2014 special issue of Preventing School Failure, Designing, Implementing, and Evaluating Comprehensive, Integrated, Three-Tiered Models of Prevention: A Step-by-Step Guide for additional details and a sample of the school level report of results.
Pre-Implementation PIRS PDF to print.
Pre-Implementation PIRS Report Template MS-Word document to edit and print.
Pre-Implementation PIRS and Demographic Qualtrics Survey and Report.zip
Adapted from the Primary Prevention Plan: Feedback Form (Lane, 2002), the Ci3T Model of Prevention: Feedback Form allows teams to gather additional feedback from school stakeholders on the full draft of the Ci3T prevention plan. The feedback form is the second formal step in soliciting stakeholder feedback (assessing social validity) of the draft Ci3T plan (also referred to as Ci3T blueprint). The feedback form is used at the end of the training series (completed after Session 5 and results returned at Session 6) and allows stakeholders a voice in final revisions made during Session 6 as they prepare to implement during the final year. It is completed by stakeholders who have reviewed the revised plan to provide feedback to the school’s Ci3T leadership team. The team then uses stakeholders’ feedback for a final round of revisions to the plan, which will then be implemented in the following academic year. Similar to scoring the PIRS, the five-point, Likert-type items are averaged (means and standard deviations reported) and the four open ended item comments are typed and combined into one report. The team assesses the level of agreement with the plan and themes of comments to determine additional professional learning needs and final refinement to the Ci3T plan before implementation. Please see Lane, Oakes, Jenkins, Menzies, & Kalberg (2014) for additional detail.
Ci3T Feedback Form PDF to print.
Ci3T Feedback Form Report Template MS-Word document to edit and print.
Ci3T Feedback Form and Demographic Qualtrics Survey and Report.zip
Ci3T school leadership teams complete PROFESSIONAL LEARNING SERIES SESSION EVALUATIONS after they attend each of the six Ci3T sessions throughout the training year. The feedback is used to improve content delivery and determine if the sessions are meeting the needs and expectations of each school’s leadership team to be able to successfully design a Ci3T model of prevention. Professional learning series session attendees rate seven items (e.g., The format of the training kept me engaged, The objectives of the training session were addressed) on a 5-point Likert-type scale ranging from 1 = unsuccessful to 5 = successful. Space for additional comments, recommendations, and commendations is provided.
Implementation Measures TRANSCRIPT
The Ci3T Treatment Integrity: Teacher Self-Report (Ci3T TI: TSR; Lane, 2009a) and Ci3T Treatment Integrity: Direct Observation Tool (Ci3T TI: DO; Lane, 2009b) are used to measure the level of implementation of the school's Ci3T plan core components. The Ci3T TI: TSR is a 38-item checklist assessing educator use of Ci3T practices. There are three subscales: Procedures for Teaching (16 items; e.g., Did I use clear routines for classroom procedures?), Procedures for Reinforcing (10 items; e.g., Did I use behavior-specific praise when giving tickets to students?), and Procedures for Monitoring (12 items; e.g., Did I use behavior and academic data together [in conjunction with each other] to inform my instruction?; Lane, K. L., 2009). Teachers rate their use of each item using a 5-point Likert-type scale (not at all = 0, some of the time = 1, most of the time = 2; all of the time = 3; no opportunity = 7). Scores are computed by dividing total score by total possible (adjusted for missing items) and multiplying by 100; school percentages by calculating the mean score for everyone who completed the Ci3T TI: TSR. This allows the school Ci3T leadership team to plan for summer revisions, professional learning opportunities, and coaching supports. Additionally, teams can monitor integrity of the core components over time and in conjunction with student data. For example, if there are increases in behavioral risk on screening measures or office discipline referrals, teams would examine the integrity data to ensure students have access to the Ci3T primary plan before assuming non-responsiveness. That is, consider if the Ci3T plan is being implemented at sufficient integrity to produce the desired results for students. We recommend treatment integrity assessments be conducted at least twice per year, once during fall semester (at least 6-8 weeks after school began) and once during spring semester. An initial evaluation of the Ci3T TI: TSR indicated adequate to desirable reliability (see Bruhn, 2011).
Ci3T Treatment Integrity: Teacher Self-Report PDF to print
Fall 2017 (previous version with separate survey and report files):
The Ci3T Treatment Integrity: Direct Observation (Ci3T TI: DO; Lane, 2009b) tool is a subset of items from the Ci3T Treatment Integrity: Teacher Self-Report (Ci3T TI: TSR; Lane, 2009a). It includes 13 procedures for teaching and eight procedures for reinforcing. Trained observers enter a classroom for 30 minutes and score each item using a 5-point Likert-type scale, indicating the level of implementation observed during that session for each item (not at all = 0, some of the time = 1, most of the time = 2; all of the time = 3; no opportunity = 7). After the observation time and at a natural break in instruction, the teacher completes the same direct observation tool and scores his or her perception of implementation during the same 30 minute time period. Integrity is computed by dividing total score by total possible (adjusted for missing items) and multiplying by 100, with the goal of examining the extent to which the observer and teacher converge and diverge in their views of implementing the Ci3T primary plan. Results can be used to celebrate strengths and to inform professional development offerings to support particular components of the plan (Lane, Oakes, & Magill, 2014).
The SCHOOL-WIDE EVALUATION TOOL (SET; Todd et al., 2012) version 2.1 is a treatment integrity tool used to evaluate implementation progress of the positive behavioral interventions and supports (PBIS) component of a school’s Ci3T plan. It contains 28 items to assess the seven critical features of PBIS: (a) school-wide behavioral expectations are defined, (b) school-wide expectations are taught explicitly to all students, (c) reinforcement is provided for meeting school-wide behavioral expectations, (d) a continuum of consequences for responding to problem behavior is implemented consistently, (e) problem behavior patterns are monitored and resulting data are used to inform ongoing decision making, (f) an administrator is involved with and supports the school-wide PBIS framework, and (g) the school district offers assistance as evidenced by functional policies, staff training opportunities, and data collection options. Multiple sources of information are gathered including permanent products (e.g., school improvement plan goals, school action plan, office discipline referral form), observations, staff interviews, and student interviews. SET items are scored using specific criteria for each item, where 0, 1, or 2 points are earned based on how fully the evaluation question was satisfied. For example, “Are the agreed upon rules & expectations publicly posted in 8 of 10 locations?” is worth 0 points if expectation matrices were observed in 0-4 locations, 1 point for 5-7, and 2 points for 8-10 locations. Percentage scores for each subscale and an overall score are calculated. In our work, the SET is typically completed twice during the academic year, once in fall and once in spring. Results are used by schools to determine annual goals, design and revise procedures as needed, compare efforts from year to year, and more.
The SET and the SET Muanual are available at https://www.pbis.org/resource/school-wide-evaluation-tool-set.
The TIERED FIDELITY INVENTORY (TFI; Algozzine et al., 2019) version 2.1 is a treatment integrity tool used to evaluate implementation progress of the positive behavioral interventions and supports (PBIS) component of a school’s Ci3T plan. It is divided into three sections corresponding to Tier 1, Tier 2, and Tier 3 features. Sections can be used separately or all together to evaluate to what degree of fidelity core features of PBIS are in place at each tier.
The TFI is available at https://www.pbis.org/resource/tfi.
The PRIMARY INTERVENTION RATING SCALE (PIRS) by Lane, Robertson, and Wehby (2002; Lane, Kalberg, Bruhn, Driscoll, Wehby, & Elliott, 2009) is an adapted version of the Intervention Rating Profile-15 (Witt & Elliott, 1985), a widely used measure of social validity. Social validity refers to stakeholders' views on the social significance of the intervention goals, acceptability of the intervention procedures, and the social importance of the intervention outcomes (Wolf, 1978). The PIRS is a brief, individual-completed rating scale designed to assess social validity of primary prevention plans prior to intervention onset (pre-intervention) and after the primary prevention plan has been put in place (implementation). To complete, each individual from the school site who has reviewed the Ci3T primary prevention plan (or other primary prevention plan) rates his or her opinion on 17 items using a 6-point Likert-type scale ranging from (1) strongly disagree to (6) strongly agree, and then completes four open-ended questions regarding suggestions for changes to the primary plan, perceptions of student performance as a result of implementing the primary plan, and what is felt to be the least and most beneficial components of the primary plan. The PIRS take approximately 10 minutes to complete. Please see Lane, Oakes, and Magill (2014) in the volume 58, issue 3, 2014 special issue of Preventing School Failure, Designing, Implementing, and Evaluating Comprehensive, Integrated, Three-Tiered Models of Prevention: A Step-by-Step Guide for additional details and a sample of the school level report of results.Primary Intervention Rating Scale (implementation) PDF to print
Fall 2017 (previous version with separate survey and report files): TSR_PIRS_Implementation_Qualtrics_Survey_Template_1718.qsf
TIER 2 & 3 SOCIAL VALIDITY MEASURES
Please see the Systematic Screening tab.
- Bruhn, A. L. (2011). Measuring primary plan treatment integrity of comprehensive, integrated three-tiered prevention models. (Unpublished doctoral dissertation). Vanderbilt University, Nashville, TN.
- Elliott, S. N., & Gresham, F. M. (2008). Social Skills Improvement System (SSiS): Performance Screening Guides. San Antonio, TX: PsychCorp Pearson Education.
- Feil, E. G., Walker, H. M., & Severson, H. H. (1995). The Early Screening Project for young children with behavior problems: Research and development of the early screening project. Journal of Emotional and Behavioral Disorders, 3(4), 194-202.
- Lane, K. L. (2002). Primary Prevention Plan: Feedback Form. Unpublished rating scale.
- Lane, K. L. (2009a). Teacher self-report form. Unpublished instrument.
- Lane, K. L. (2009b). CI3T treatment integrity: Direct observation tool. Unpublished instrument.
- Lane, K. L., Kalberg, J. R., Bruhn, A. L., Driscoll, S. A., Wehby, J. H., & Elliott, S. (2009). Assessing social validity of school-wide positive behavior support plans: Evidence for the reliability and structure of the Primary Intervention Rating Scale. School Psychology Review, 38, 135-144.
- Lane, K. L., Kalberg, J. R., Bruhn, A. L., Mahoney, M. E., & Driscoll, S. A. (2008). Primary Prevention Programs at the Elementary Level: Issues of Treatment Integrity, Systematic Screening, and Reinforcement. Journal of Education and Treatment of Children. 31, 466-494.
- Lane, K. L., Little, M. A., Casey, A. M., Lambert, W., Wehby, J., Weisenbach, J. L., & Phillips, A. (2009). A comparison of systematic screening tools for emotional and behavioral disorders. Journal of Emotional and Behavioral Disorders, 17(2), 93-105. DOI: 10.1177/1063426608326203
- Lane, K. L., Oakes, W. P., Common, E. A., Zorigian, K., Brunsting, N. C., & Schatschneider, C. (2014). A comparison between SRSS-IE and SSiS-PSG scores: Examining convergent validity. Assessment for Effective Intervention, 1-13. doi: 10.1177/1534508414560346
- Lane, K. L., Oakes, W. P., Ennis, R. P., & Hirsch, S. E. (2014). Identifying students for secondary and tertiary prevention efforts: How do we determine which students have Tier 2 and Tier 3 needs?Preventing School Failure, 58, 171-182.
- Lane, K. L., Oakes, W. P., Harris, P. J., Menzies, H. M., Cox, M., & Lambert, W. (2012). Initial evidence for the reliability and validity of the student risk screening scale for internalizing and externalizing behaviors at the elementary level. Behavioral Disorders, 99-122.
- Lane, K. L., Oakes, W. P., Jenkins, A., Menzies, H. M., & Kalberg, J. R. (2014). A team-based process for designing comprehensive, integrated, three-tiered (CI3T) models of prevention: How does my school-site leadership team design a CI3T model? Preventing School Failure: Alternative Education for Children and Youth, 58 (3), 129-142. DOI: 10.1080/1045988X.2014.893976
- Lane, K. L., Oakes, W. P., & Magill, L. (2014). Primary prevention efforts: How do we implement and monitor the tier 1 component of our comprehensive, integrated, three-tiered (ci3t) model?Preventing School Failure: Alternative Education for Children and Youth, 58(3), 143-158. doi: 10.1080/1045988X.2014.893978
- Lane, K. L., Oakes, W. P., Swogger, E. D., Schatschneider, C., Menzies, H., M., & Sanchez, J. (2014). Student risk screening scale for internalizing and externalizing behaviors: Preliminary cut scores to support data-informed decision making. Behavioral Disorders.
- Lane, K. L., Robertson, E. J., & Wehby, J. H. (2002). Primary Intervention Rating Scale. Unpublished rating scale.
- Lynass, L., Tsai, S., Richmond, T., & Cheney, D. (2012). Social expectations and behavioral indicators in schoolwide positive behavior supports: A national study of behavior matrices. Journal of Positive Behavior Interventions, 14, 153-161.
- Oakes, W.P., Lane, K.L., Jenkins, A., & Booker, B.B. (2013). Three-tiered models of prevention: Teacher efficacy and burnout. Education and Treatment of Children, 36(4), 95-126.
- Walker, H. M., Severson, H. H., & Feil, E. G. (2014). Systematic screening for behavior disorders (SSBD) technical manual: Universal screening for preK–9 (2nd ed.). Eugene, OR: Pacific Northwest Publishing.