Evaluation with the University of South Florida

The national evaluation uses a multi-level, mixed-method design that is framed by the initiative’s theory of change, which assumes that community-driven, community-level strategies can help improve the mental health and wellbeing of men and boys in high need communities, including those of color, service members, and veterans. The core components of the evaluation design include the process evaluation, outcome evaluation, and evaluation techinical asssitance.

 

Process Evaluation

The process evaluation is designed as a mixed-method explanatory case study in which each grantee site serves as a case/the unit of analysis. Data collection methods include monthly grantee calls, annual on-site data collection (site visits), and a coalition/partnership assessment (via social network analysis and the coalition effectiveness survey).  A cross-site process assessment is also conducted on the grantee’s capacity to create change in their communities. The evaluation team developed the Capacity Building to Create Community Change (CBC3) model (based on project data), which assesses grantee progress in each of the eight domains below.

 

Outcomes Evaluation: The Mental Wellbeing Inventory         

To demonstrate site-based and cross-site improvement over time, in one or more areas of mental wellbeing at the population of focus and community levels, the evaluation developed a mental wellbeing inventory (MWBI). This measure was developed based on the results of concept mapping conducted with all grantees.

The results identified 33 items in eight overall domains: 1) Dignity and Respect, 2) Safety, 3) Institutional Relationships, 4) Community Roles, 5) Community Connections, 6) Social Connections, 7) Positive Self-worth and 8) Positive Masculinity that are foundational to supporting the mental health and wellbeing of men and boys in this initiative.  Each domain includes a variety of statements that operationalize the meaning of that particular domain.  For questions or more information about the MWBI, contact Roxann McNeish mcneish@usf.edu.

 

Evaluation Technical Assistance (TA)

Technical assistance from the national evaluation team is based on principles of theory-driven evaluation which supports results-based accountability. Evaluation TA focuses on using evaluation data to support program implementation, continued quality improvement, and data-driven decision making.