Evaluation with the University of South Florida

Discovering What Works

Discovering What Works

The Department of Child and Family Studies at the University of South Florida (USF) is conducting a national evaluation of the Making Connections initiative. The purpose of this multi-site, multi-year evaluation is to understand how communities improve the mental health and wellbeing of men and boys through community-level prevention efforts. Overall, the evaluation team is tracking:

  1. How communities create successful and scalable community change,
  2. The barriers and facilitators to successful implementation across different sites, 
  3. What factors influence community engagement, and
  4. How effective are these community-level change initiatives in improving the mental health and wellbeing of men and boys.

You can visit the Making Connections Evaluation website for more information https://www.mcievaluation.com/.

The national evaluation uses a multi-level, mixed-method design that is framed by the initiative’s theory of change, which assumes that community-driven, community-level strategies can help improve the mental health and wellbeing of men and boys in high need communities, including those of color, service members, and veterans. The core components of the evaluation design include the process evaluation, outcome evaluation, and evaluation techinical asssitance.

 

Process Evaluation

The process evaluation is designed as a mixed-method explanatory case study in which each grantee site serves as a case/the unit of analysis. Data collection methods include monthly grantee calls, annual on-site data collection (site visits), and a coalition/partnership assessment (via social network analysis and the coalition effectiveness survey).  A cross-site process assessment is also conducted on the grantee’s capacity to create change in their communities. The evaluation team developed the Capacity Building to Create Community Change (CBC3) model (based on project data), which assesses grantee progress in each of the eight domains below.

 

Outcomes Evaluation: The Mental Wellbeing Inventory         

To demonstrate site-based and cross-site improvement over time, in one or more areas of mental wellbeing at the population of focus and community levels, the evaluation developed a mental wellbeing inventory (MWBI). This measure was developed based on the results of concept mapping conducted with all grantees.

The results identified 33 items in eight overall domains: 1) Dignity and Respect, 2) Safety, 3) Institutional Relationships, 4) Community Roles, 5) Community Connections, 6) Social Connections, 7) Positive Self-worth and 8) Positive Masculinity that are foundational to supporting the mental health and wellbeing of men and boys in this initiative.  Each domain includes a variety of statements that operationalize the meaning of that particular domain.  For questions or more information about the MWBI, contact Roxann McNeish mcneish@usf.edu.

 

Evaluation Technical Assistance (TA)

Technical assistance from the national evaluation team is based on principles of theory-driven evaluation which supports results-based accountability. Evaluation TA focuses on using evaluation data to support program implementation, continued quality improvement, and data-driven decision making.

Meet the Team

Meet the Team

Roxann McNeish, Principal Investigator

Tom Massey, Co-Principal Investigator

Connie Walker, Grantee Liaison

Chris Simmons, Grantee Liaison

Khary Rigg, Grantee Liaison

Cathy Sowell, Grantee Liaison

Alexandra Albizu-Jacob, Grantee Liaison

Angela Vatalaro, Data Collection Lead

Tracy-Ann Gilbert-Smith, Project Manager