The ACM CHI Conference on Human Factors in Computing Systems, commonly known as 'kai,' is the premier international conference in Human-Computer Interaction (HCI). This annual gathering brings together a diverse group of researchers and practitioners from around the world, representing various cultures, backgrounds, and perspectives. Their shared goal is to improve the world by developing and applying interactive digital technologies. In this blog post, I delve into the research paper authored by Tawanna R. Dillahunt and Joey Chiao-Yin Hsiao, titled "Positive Feedback and Self-reflection: Features to Support Self-efficacy among Underrepresented Job Seekers," published in the Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems.
In the digital age, finding employment increasingly relies on technology, yet underrepresented job seekers often find themselves disadvantaged due to a lack of support from existing digital tools. This research uses the Theory of Planned Behavior (TPB) to explore the effectiveness of two innovative tools, "Review-Me" and "Interview4", designed to enhance job search self-efficacy, attitudes, and intentions among these job seekers. Their empirical study, which includes a longitudinal deployment of the tools across 23 underrepresented job seekers from resource-constrained areas, highlights the complex role of feedback and self-reflection in enhancing job search outcomes. The study advocates for digital tools that promote positive feedback and self-reflection, as these features have been shown to improve job seekers' efficacy and motivation, thereby supporting their employment goals more effectively. These insights underscore the potential for tailored digital interventions to empower underrepresented job seekers, offering them a better chance to overcome their unique challenges in the labor market.
Introduction
Technology has significantly transformed the employment landscape, influencing job search methods and market dynamics. Despite economic growth in various regions, stable employment opportunities have declined, particularly affecting racial and ethnic minorities, individuals with disabilities, and those without a college degree in the United States. These groups constitute a large portion of the unemployed population. To understand how these job seekers manage their employment search in this challenging environment, the study focused on underrepresented groups and investigated how digital employment tools could support them.
Existing Human-Computer Interaction research has proposed several tools to aid underrepresented and disadvantaged job seekers. This study aims to assess the effectiveness of such interventions in facilitating job searches among these populations. They applied TPB to understand the effectiveness of the aforementioned tools in elevating the self-reflective process among underrepresented job-seekers.
In an eight-month randomized field experiment, they evaluated the tools with 23 underrepresented job seekers from resource-constrained areas. Participants were divided into three groups: Review-Me users, Interview4 users, and a control group. Their methodology included pre- and post-treatment surveys, tool tutorials, semi-structured interviews, and a one-month tool deployment. Participants provided weekly diaries of their job search activities and were compensated for their time. The study revealed several key findings. Both Review-Me and Interview4 positively impacted job seekers’ self-efficacy and subjective norms. Review-Me participants reported increased confidence due to constructive resume feedback, while Interview4 users improved their interview skills through self-assessment and practice. However, attitudes towards the job search showed mixed results, with long-term unemployment negatively affecting participants’ outlooks over time.
These results emphasize the importance of incorporating positive feedback and self-reflection features in digital employment tools to better support underrepresented job seekers, ultimately contributing to more effective job search strategies and improved employment outcomes.
Figure 1 Dillahunt et al.: Adaptation of the TPB model for our research context. This study investigates the impact of two employment tools, Review-Me and Interview4, on the three cognitive factors of the TPB model. These factors are known to positively influence job search intentions and behaviors.
Study Methodology
The study methodology included a randomized field experiment conducted over eight months to evaluate the effectiveness of two digital employment tools, Review-Me and Interview4, on underrepresented job seekers. The primary aim was to investigate how these tools influenced the three cognitive factors of TPB—job search self-efficacy, subjective norms, and job search attitudes—thereby affecting job search intentions and behaviors.
Figure 2 Dillahunt et al.: This diagram outlines the assignment of participants into three distinct groups: the Review-Me group, the Interview4 group, and the control group, for the randomized field experiment. It details the participant count in each phase of the study along with the compensation structure.
Participant Recruitment and Group Assignment: The study participants were primarily from low-resource areas, with a focus on those actively seeking employment for the last six months. Recruitment was executed using both offline and online methods, including connections with local workforce development programs, advertisements on community boards, and digital platforms like Facebook and Craigslist. Once recruited, participants were randomly assigned to either one of the tool groups or a control group in a manner that balanced their job search activity interests. This randomization helped ensure the reliability of the treatment effects across varied job search activities.
Pre-treatment Session: The pre-treatment session of the study was a crucial initial step designed to onboard participants and familiarize them with the tools they would be using—either Review-Me or Interview4. Each session lasted between 60 to 80 minutes and was structured to gather baseline data on the participants’ job search behaviors and attitudes, as well as introduce them to the functionality of the assigned digital tools. At the beginning of the session, participants completed a survey that included measures adapted from existing literature on job search interventions. This survey assessed the three cognitive factors of TPB—self-efficacy, subjective norms, and attitudes towards job searching. Following the survey, participants received a 15-minute tutorial on their assigned tool, which provided them with an overview of how to use the features relevant to their job search activities. For those assigned to Review-Me, this included uploading a resume and reviewing feedback, while for Interview4 participants, it involved making and reviewing recordings of practice interviews. Additionally, a semi-structured interview was conducted during the session. This interview was aimed at enriching the quantitative survey data with qualitative insights into the participants' current employment status, job search experiences, and their expectations from the study. The combination of survey, tutorial, and interview in the pre-treatment session equipped participants with the necessary tools and knowledge for the study while establishing a comprehensive baseline for assessing changes over the course of the intervention.
Tools Evaluated: Review-Me is an application that allows users to upload their resumes and receive feedback from volunteers, including Amazon Mechanical Turk workers and student volunteers. This tool aims to enhance job seekers' resumes by providing constructive feedback and boosting their job search self-efficacy. Interview4, on the other hand, is a video-based tool owned by Hire-Intelligence LLC. It enables job seekers to practice job interviews by recording their responses to standard interview questions. The tool allows for the playback of these recordings, allowing users to self-assess and refine their interview skills based on the feedback provided. Additionally, Interview4 supports sharing practice videos, allowing for external feedback that can further assist job seekers in improving their interview techniques. Both tools were conceptualized to meet the immediate and pressing needs of job seekers by enhancing their skills in critical areas of the job application process—resume crafting with Review-Me and interview practice with Interview4. The evaluation of these tools focused on their potential to positively affect the job search process by improving candidates' preparedness and confidence.
Figure 3 Dillahunt et al.: Screenshots of (a) Review-Me and (b) Interview4. These images have been modified to highlight essential functionalities: Review-Me's volunteer feedback and resume evaluation, and Interview4's features for prompting and recording responses.
One month deployment: During the one-month deployment phase of the study, participants were tasked with actively using their assigned digital tools—Review-Me or Interview4—as part of their regular job search activities. Those using Review-Me were instructed to upload their resumes at least twice, allowing them to receive feedback and make necessary revisions based on the critiques provided. This iterative process aimed to enhance their resume quality over the course of the month. Meanwhile, participants assigned to Interview4 were encouraged to practice job interviewing regularly using the tool. They were provided with a set of default interview questions, and additional sets were offered to diversify their practice sessions. This practice was intended to help participants refine their interview skills through self-assessment and feedback. Additionally, participants were asked to maintain a diary of their job search activities during this period, tracking their efforts and progress, and providing researchers with insights into their usage of the tools and any changes in their job search behavior.
Post-Treatment session: In the post-treatment session, which also lasted between 60 to 80 minutes, participants reconvened to assess the impact of the digital tools on their job search activities after one month of use. This session mirrored the pre-treatment setup, including a survey and a semi-structured interview that mirrored the initial assessments to measure any changes in the participants' self-efficacy, subjective norms, and job search attitudes. The survey questions were slightly adjusted to reflect on the participants' experiences and behaviors over the past month. The qualitative interviews helped gather in-depth feedback on the participants' use of the tools and any shifts in their job search strategies.
Data Collection and Analysis: The study employed a mixed-methods approach for data collection and analysis, integrating both quantitative and qualitative data to provide a comprehensive understanding of the tools' impacts. Quantitative data were gathered through surveys administered during the pre-treatment and post-treatment sessions, which measured changes in job search self-efficacy, subjective norms, and attitudes. Qualitative data were collected via semi-structured interviews and diary entries, offering deeper insights into the participants' experiences and perceptions of the tools. Analysis involved triangulating these data sources to validate findings and uncover trends. Descriptive statistics highlighted general quantitative trends, while thematic analysis of interview transcripts and diary entries helped interpret the qualitative data. This robust approach ensured a thorough examination of how the digital tools affected the job search behaviors and attitudes of underrepresented job seekers.
Results
Quantitative Results: The quantitative results of the study provided mixed insights into the effectiveness of the digital employment tools, Review-Me and Interview4, on influencing the job search behaviors and attitudes of underrepresented job seekers. The primary quantitative data came from pre-treatment and post-treatment surveys, which assessed changes in TPB's three cognitive factors: job search self-efficacy, subjective norms, and job search attitudes.
For job search self-efficacy, participants who used Review-Me showed a significant improvement, indicating that receiving feedback on their resumes positively impacted their confidence in job-seeking activities. Interview4 users also reported slight improvements in self-efficacy, suggesting that practicing interviews and viewing their recordings helped enhance their perceived ability to perform in actual job interviews.
In terms of subjective norms, both tools seemed to encourage a perception of increased social support among users. This was likely due to the feedback mechanisms of both tools, which mimicked social interactions by providing external evaluations of the users' job search materials and techniques.
However, the changes in job search attitudes were less straightforward. While some Interview4 users noted improvements, likely due to increased self-efficacy and perceived social support, Review-Me users had mixed results. This variance could be attributed to the personal relevance of the feedback received or the individual's specific circumstances during the job search period.
Figure 4 Dillahunt et al.: In the diagrams, each circle symbolizes a participant and their relative change in percentage (Valpost − Valpre)/Valpre for each of the three TPB factors. The size of each circle indicates the duration for which participants engaged with their assigned tools; larger circles denote more extensive use. Crosses mark participants who did not utilize the tools throughout the deployment phase. For clarity, one participant is omitted from Figure 4(b) and another from Figure 4(c); both exhibited a 600% change in each factor, but their data are included in the overall analysis.
Qualitative Results: In the study, the three cognitive factors of TPB—job search self-efficacy, subjective norms, and job search attitudes—were critically analyzed to understand their influence on job search intention and behavior among underrepresented job seekers using the tools Review-Me and Interview4.
Job Search Self-Efficacy: The study found a significant positive change in self-efficacy among participants, particularly those using Review-Me. This was attributed to the constructive feedback received on their resumes, which enhanced their confidence in job-seeking tasks. A participant noted the impact of this feedback: "I know that my resume beforehand was not very good, as often as I tried to get help from other people and tried to figure it out. I believe that especially because then [after revising her resume using Review-Me], two or three companies responded immediately with asking for an interview whereby before I was searching and turning in my resume for four years long and getting no responses."
Subjective Norms: There was a notable increase in perceived social support, measured as subjective norms. This factor was positively influenced by both tools, with participants feeling more supported in their job search efforts. One participant expressed, "I don’t have anybody to critique my resume. And uploading it on here [Review-Me], they [volunteers] pointed out a couple different things that are really going to help me fix my resume."
Job Search Attitudes: The impact on job search attitudes was mixed, with some participants experiencing more positive attitudes due to increased self-efficacy and support. However, long-term unemployment seemed to temper these gains, as reflected in one participant's comment: "Interview4 along with just doing the job searching. It just made [me] more aware, like what I want to do... [It] made me think of more career than just getting a job."
Discussion
The study highlights the significant role of positive feedback and self-reflection in enhancing self-efficacy among underrepresented job seekers using the digital tools Review-Me and Interview4. These elements not only improved job search self-efficacy but also seemed to increase the sense of social support, as evidenced by changes in subjective norms. Furthermore, the discussion addresses challenges such as participant attrition and suggests strategies to improve engagement and retention in future studies. It emphasizes the need for further research to explore more comprehensive ways of providing feedback and enhancing social support through digital platforms.
The paper provides insightful analysis on the effectiveness of digital tools in supporting disadvantaged job seekers. However, the high attrition rate poses concerns about generalizability. Future studies could benefit from more robust engagement strategies to ensure broader participation, thereby enhancing the credibility and impact of the findings. Additionally, integrating a more diverse array of digital tools could offer richer insights into different facets of job search behavior and tool efficacy.
Conclusion
- Integrating features that provide positive feedback and promote self-reflection in digital employment tools significantly enhances job seekers’ self-efficacy. These elements are crucial for empowering underrepresented job seekers and improving their job search outcomes.
- Future digital employment tools should directly connect job seekers with feedback providers to build social capital. Additionally, strategies to increase study retention among underrepresented populations are essential for conducting effective longitudinal research in this domain.
- This research provides valuable insights into how positive feedback and self-reflection can enhance the employment outcomes of underrepresented groups, making it a critical tool for empowering these populations. Extending this approach to people with special abilities, such as blind and neurodivergent individuals, could further tailor digital employment tools to meet their unique needs and improve their job search experiences.
References
Comments
Post a Comment