MELISSA JENNINGS
Designing the Ally Training Program
Key activities in human centered design research consulting
-
Literature review
-
Research questions (Evaluation questions)
-
Consultations with subject matter experts (SMEs)
-
Pre-post tests
-
Surveys
-
Expert observations
-
Data analysis
-
Actionable recommendations for SMEs to iteratively refine training content
UX Design 2019 Lead Researcher
Data Collection Plan
I led the development of research questions and a data collection and analysis plan (see banner above), including working closely with subject matter experts to develop a scale to address gaps.
Data Collection & Analysis Measured Changes in:
-
Participant knowledge on 32 items were collected before and after the training (pre-post tests)
-
Training quality on 8 items for the training overall, and for each module
-
Via participant satisfaction survey questions
-
Via expert observations
-
-
Participant attitudes on 33 items collected before and after the training.
Our perceptions measure and the Ally Identity Measure (AIM) included the following domains:-
Ability to confront
-
Comfort discussing LGBTQ issues
-
Connectedness to resources
-
Role in environment
-
Knowledge and skills
-
Openness and support
-
Oppression awareness
-
-
​​A sample survey question sounds like, “I know of at least two strategies to respond to hearing a homophobic or transphobic comment at work,” and used a Likert scale to gauge agreement.
The Analysis
-
We examined the changes in participant knowledge and attitudes to identify which modules of the training were most and least effective.
-
Since the questionnaires were anonymous, we could not look at individual knowledge gains. So overall group level knowledge changes were examined by calculating the percentage of participants who got each question correct. We took the absolute difference of the percentage of the questions answered correctly from pre-test to post-test.
-
To gauge quality, we averaged participant ratings on the overall training, training format, facilitators, and each module using a 5-point scale where 1 was poor and 5 was excellent.
-
We examined participant attitude changes using the AIM measure and our own Perceptions Measures. We collapsed positive and negative ends of the Likert scale values for ease of interpretation and averaged the respondent ratings by domain.
​
Example Findings
Qualitative data suggests training content and format help explain the changes in knowledge, Ally Identity Score, and the SGM Perceptions Questionnaire.
Some participants noted the Genderbread activity helped them learn differences in terms and concepts since they’re “visual learners.”
Others noted elements of the training – such as “hearing personal experiences” and “dialogue” made it “more intimate and less abstract” and led them to “feel comfortable and open” or, in some cases, to even “change their view.”
Results & Recommendations
When people feel safe being themselves, they bring the best to our organizations and the world. The data show success across all the metrics measured, including subjective ratings of the training, as well as increases in knowledge and positive changes in participant attitudes. However, there are always ways to improve, regardless of how high the ratings.
The results led to the following actionable recommendations to iteratively refine training content:
-
Decreasing the length of questionnaires
-
Spend more time answering questions and discussing the vocabulary, gender bread person, and do’s and don’ts activities.
-
Spend less time covering history, policy, examples of discrimination, and statistics
-
Maintain videos, personal accounts, and facilitator’s patience with mistakes, which both participants and observers believed improved engagement with the content and participation in the discussion, and ultimately improved learning.
​
Next Steps
After conducting the UX Research on the pilot training described here, iterative improvements were made. This further increased learning scores and favorable attitudes. See our publication of a paper in the Journal of Public Health Management and Practice.
