Participatory Research Based Program Evaluation Theoretical Framework
You can read a downloadable version of this article: Participatory Research Based Program Evaluation Theoretical Framework
Abstract
For my dissertation, my current thinking is to use participatory research methods to conduct a program evaluation for a classroom serving students with emotional and behavioral disabilities (EBD). This paper looks at the theoretical framework of participatory research, and its connection with social work, k-12 education, and resilience.
Participatory Research Based Program Evaluation Theoretical Framework
The K-12 educational system faces challenges on many fronts. Students have a diverse set of needs to be addressed in the classroom and school setting. Paired with the challenges of equipping staff to meet those students’ concerns, these varied challenges give insight into the complexity of the education system. Being able to offer high-quality services for students require educators to be able to be reflective about the types of interventions they are employing or the systems that are in place to help support all students. Students with special education services have a higher level of deficits requiring support within the school system. All students who qualify for special education services have a disability as described by the Individuals with Disability Education Act (IDEA) Amendments of 1997. When a student qualifies for special education services, there are many protections and services school districts are required to implement, including the provision of specially designed instruction. One of the disability categories that can qualify a student for special education services is that of an emotional disturbance.
Students with emotional disturbance, which is frequently described at emotional and behavioral disabilities (EBD), demonstrate a significant number of disruptive and difficult behaviors within the classroom, school environment, home, and community. Special education services are a continuum and not just a class. Services range from making modifications and accommodations within a general education setting to being in a self-contained classroom or school (Knight, 1999). Self-contained classrooms and schools are considered highly restrictive environments as they limit the student’s interaction with general education peers, and all the academic instruction is completed in that setting (Lane et al., 2005). The United States Department of Education (2019) describes many students with emotional disturbance (which is also frequently referred to as emotional and behavioral disabilities or EBD) spend a significant amount of their time outside of a general education setting. This report, the Digest of education statistics, 2018, describes that 17.7 percent of students with EBD spend less than 40 percent of their time with their general education peers. These students might be in a resource room setting or a self-contained classroom setting. A sizable number of these students are in an even more restrictive environment, where 12.4 percent are outside of comprehensive schools and in a separate school for students with disabilities.
Students with EBD can also have high rates of traumatic events that have impacted them, adverse childhood experiences, and symptoms related to post-traumatic stress disorder. Cavanaugh (2016) describes that based on the prevalence of traumatic experiences for students with EBD, there is a need for their teachers to be aware of the impact of trauma and effective methods for addressing students’ educational and social needs in light of that trauma. Kirst-Ashman and Hull (2015) describe social workers as engaging with clients through micro practice, mezzo practice, and macro practice. They describe micro as working with individuals, mezzo as working with groups, and macro as working with communities and organizations.
Students are affected by this trauma, but so are staff, and school climate. Compassion fatigue seems to contribute to the difficulty in maintaining special education teachers (Hoffman et al., 2007). There is a need to understand individual students within a classroom looking at their overall needs. This includes their traumas, strengths, and skills they possess for being resilient. Determining these distinguishing factors allows educators to tailor content to student’s direct needs.
Along with understanding the individual needs or micro level, the information can be compared and understood at a small group level within the classroom setting or mezzo level. Dombo and Sabatino (2019) describe eleven schoolwide interventions that are useful for creating a trauma-informed school. These include: recognizing the impact on development and coping strategies, making a recovery from trauma a primary goal, employing an empowerment model, maximizing student’s choice and control in the recovery process, basing services on relational collaboration. Program services also require creating an atmosphere of safety, acceptance, and respect for survivors. Trauma-informed services focus on strengths and resilient adaptation over pathology, minimize the possibility of retraumatization. They provide services in a culturally competent manner, solicit consumer involvement and input, and address vicarious trauma. Staff competence and understanding of the effects of trauma and its counterpart resilience play in the lives of students also should be examined to understand the overall needs of the system. See Figure 1: Resilience and Educational Systems.
The specifics of my dissertation inquiry is still being molded and determined. The following are some of the questions I am looking at investigating: What is resiliency like for students in a self-contained classroom setting for EBD? How are resiliency and trauma understood with a school setting? What are the resilience factors of students within the classroom? What kind of resilience supporting elements does staff implement within the classroom and school setting? In general, I plan to use participatory action research methods as a tool to understand, classify, and make recommendations based on factors of resiliency for students with EBD. Through a facilitated process, staff will discuss student needs, issues, traumas, and how they impact students within their classroom setting. They will consider factors related to increasing resilience within a classroom/school setting and develop a self-created plan for how to increase resiliency.
Theoretical Framework - Participatory Research
Participatory research (PR) is a methodology and theoretical framework that allows the researcher to conduct a rigorous qualitative evaluation that has the potential to make a meaningful contribution to a quantitatively inclined scientific community. This contribution to the academic community is a staple of research methodologies. Participatory research goes further than this scientific contribution and looks towards potentially making a positive impact through social justice and the process completed within the study. This dual focus of PR is to connect science and practice (Bergold & Thomas, 2012; Cochran et al., 2008). Figure 2: Dual Focus of Participatory Research shows this connection. Persons with disabilities benefit from taking a participatory research method. Bergold and Thomas (2012) describe PR as a “methodology that argues in favor of the possibility, the significance, and the usefulness of involving research partners in the knowledge-production process” (p. 192).
Pedagogy of the Oppressed as a Foundation for Participatory Research
Pablo Freire and the Critical Consciousness Movement in Latin America were foundational in the formation of participatory research. Streck and Holiday (2015) divide the start of participatory research into two groups of theorists. First, theorists such as Kurt Lewin, Carl Rogers, and Martin Buber and the start of the humanist movement provided foundation and conceptualization for the beginnings of participatory research. The second set of theorists is related to the concientização (Portuguese for critical consciousness) movement, which was highly influenced by Freire. Boden et al. (2015) described Freire’s Pedagogy of the Oppressed as a seminal work for participatory research.
Freire (1970/2000) is an argument against what he calls the banking model of education where educators view students as if they are empty vessels, like a piggy bank, that needs to be filled with knowledge. Freire counters this way of seeing learners and describes that pedagogy should treat them as co-creators of knowledge. He calls his counter methodology problem-posing education, where there is a dialogue completed between teacher-student and student-teachers. As a text and set of ideas, Freire (2000) has been influential for educators, participatory researchers, and people focused on liberation and other critical-based theories.
Freire (2000) describes that problem-posing education offers the people opportunity for transformation. That student-teachers “develop their power to perceive critically the way they exist in the world with which and in which they find themselves; they come to see the world not as a static reality, but as a reality in process, in transformation” (p. 83). The methodology of thematic investigation is closely aligned to agreed-upon components of participatory research. He describes this as useful to help find the contextual reality of the individuals and analysis of the interactions.
Meanwhile, the significant dimensions, which in their turn are constituted of parts in interaction, should be perceived as dimensions of total reality. In this way, critical analysis of a significant existential dimension makes possible a new, critical attitude towards the limit-situations. The perception and comprehension of reality are rectified and acquire new depth. When carried out with a methodology of conscientizagdo, the investigation of the generative theme contained in the minimum thematic universe (the generative themes in interaction) thus introduces or begins to introduce women and men to a critical form of thinking about their world (p. 104).
The argument for the teachers to follow a more problem-posing educational model is directed towards implementation at a micro level but is completed in a mezzo application. In general, the argument described by Freire (2000), takes a critical and change focus. He relates that dialogics and antidialogics as opposing theories of social action. He calls for a type of liberation that can only happen at a macro level.
The theoretical framework related to participatory action research is intricate with many ways of approaching the broad methodology and conceptual framework for analysis. Freire (2000) has made a significant contribution to the theoretical framework of participatory research without having named that specifically. It seems to have been a type of groundwork for what would later come. Rodgers (1979) move towards a person-centered counseling model and Freire focus on dialogue with students. Borda (2008) describes participatory research requiring a type of scientific modesty, that there is a “struggle against academic arrogance and imperialism” (p. 360) to help bridge the gap between academics and other others involved in the participatory research process.
Contemporary Understanding of Participatory Research
Participatory research (PR) contributes to social action in co-researchers’ everyday life and generating basic knowledge in social sciences (Cochran et al., 2008). This concentration makes some of the fundamental principles of PR relevant components to consider in its implementation. First, participatory research must be understood in the context of a democratic approach in its execution and in the systems that allow it to be engaged. The very nature of a participatory evaluation is at least democratic in how it facilitated. The inclusion of participants from under-privileged demographic groups and the social commitment required by the researchers is only possible if there is a political framework that allows it. Second, the facilitators must keep the group as a safe space. Intending a prominent level of participation for the co-researchers, the facilitator must implement practices that encourage engagement and participation. Participatory research groups include a facilitator who leads the group. Paradoxically, along with the facilitator’s leadership, is that that co-researchers also must be granted opportunities to lead the group through expressing their desires or anxieties. When participants trust each other, they are willing to express their views. Third, the community needs to be able to be defined. Participatory research has gone into many different community sectors and with diverse populations. To understand the outcomes and concepts expressed by the participants. The researcher must also understand the nuances and context of the participants (Bergold & Thomas, 2012). Figure 3: Components of Participatory Research provides a graphic showing these.
Many interrelated research methodologies surround participatory research. These different methods each have nuance and difference between them. This essay is grouping all of the related participatory research subtypes. These subtypes include community-based participatory research, participatory action research, feminist participatory action research, and social action research. As a research method, PR has a particular way of viewing and seeing the world. It is essential to understand the axiology (or main ethical view), the ontology (or reality), and epistemology (knowledge and relationships with what can be known).
Participatory research often includes persons who are from vulnerable populations. Balcazar et al. (1998) discuss some special considerations that should be taken in looking at participatory action research with persons who have disabilities. They encourage researchers to have individuals with disabilities articulating the problems they are experiencing and participating directly in the process of defining, analyzing, and solving those problems. This direct involvement of persons with disabilities in the process makes for a more authentic and accurate understanding of their social reality. The engagement that the persons with disabilities show can increase their awareness of their strengths and resources. The goal of the research is to improve the quality of life for individuals with disabilities.
The considerations for doing participatory action research with persons with disabilities connect with the axiology of PR. Figure 4: The OEA of Participatory Research addresses ontology, epistemology, and axiology of participatory research methods. Axiology addresses the ethical nature of a research methodology (Killam, 2013). Lincoln et al. (2011) describe that participatory research and the postmodern research methods have the subjects’ voices present in the study. Those voices need to be understood through the three principles of deciding for others and with others and for one’s self. It is about a “practical knowing how to flourish with a balance of autonomy, co-operation and hierarchy” (p. 111).
Participatory research views and understands reality, or its ontology, through an understanding of multiple realities. Lincoln et al. (2011) describe this as a participatory reality or a subjective-objective reality. The co-researchers co-create that ontology. Each person has their worldview and way of understanding events, context, and both individual and group experiences. In the cooperative process, they determine and define the problems they are analyzing and develop the group’s worldview.
Participatory research looks at a co-created set of findings between the research facilitator and the co-researchers who are participating in the process. Killam (2013) defines epistemology related to the relationship between the researcher and the knowledge as we discover it. It is how the researcher comes to know what they know. Participatory research includes identifying both through practical and experiential applications (Lincoln et al., 2011). The co-researchers who are participating in the process of PR many times might not be previously trained in research methods. The process is designed to draw out the participants’ experiences and inner wisdom as a procedure for defining a group’s needs as a group and then using the group to analyze and address those needs. Lincoln et al. (2011) also describe it as being a critical subjectivity. Within the critical methodologies, they make understanding group power dynamics a key point. They also work to address social change or action through the research process.
Positionality of Participatory Research and Areas of Inquiry
Participatory research has been used in a variety of fields and to address a diverse set of needs. There are some fields and areas of inquiry; it is most readily able to be used to implement. The social work profession set of values connects with participatory research. School systems and classrooms serving students with EBD have also been linked to participatory research. Resilience has also benefited from participatory research methodologies.
The intersection of Social Work Values and Participatory Research
Social workers follow a code of ethics that was ratified by the National Association of Social Workers (NASW) Delegate Committee in 1996 and then revised in 2017. It is focused on identifying core values and summarizing ethical principles related to social workers. It helps social workers to identify professional obligations and address ethical concerns. The NASW Code of Ethics categorizes six core values embraced by social workers. These include service, social justice, dignity and worth of a person, importance of human relationships, integrity, and competency.
The core values of social workers align with the practice of PR. At their core, social workers are about service. The Council on Social Work Education (2015) describes both bachelor’s and master’s level social work students need to be able to demonstrate competency in what they term practice-informed research and research-informed practice. This interchange of practice and research is in line with the various action research-oriented models. One of the primary outcomes of participatory research is the change of systems for its clients through social justice. Lincoln et al. (2011) describe that one of the quality criteria for PR is that it leads to action in the service of human flourishing through a transformation in the world.
The dignity and worth of the person and the importance of human relationships also have a secure connection to PR. Bergold and Thomas (2012) explain that there is both closeness of the research participants and very personal reactions that can come from participating in the inquiry. The conditions that are conducive to participatory methods require openness and an accepting attitude from the co-researchers. The democratic nature of these types of groups also requires the ability to engage the group members with dignity and worth consistently. The identification of participants as co-researchers is an attempt to pay homage to the individuals uniting to solve their problems. Consideration of co-participants verses research subjects is in sharp contrast to the majority of research methods that separate the researcher and the participants.
In developing a safe space for the co-researchers to conduct their study, being able to reflect their viewpoints with integrity is a vital skill for the facilitator in the process. There are several competencies required to be able to facilitate a participatory group. Burdine (2010), in a discussion of community-based research partnerships, elaborates on components needed for the development of community health. These include elements such as building relationships, identifying priorities, defining problems, capacity building, knowledge transfer, information dissemination, and planning, implementing, and evaluating interventions. Each of these components requires a diverse set of skills and competencies to be performed effectively.
| Social Worker Core Values | Connections with Participatory Research |
|---|---|
| Service | Focus on both serving science (though advancing scientific understanding) and serving the co-researchers through attempting to improve |
| Social justice | Leads to action in the service of human flourishing through a transformation in the world |
| Dignity and worth of a person | Open and accepting attitudes; Democratic nature of these types of group |
| Importance of human relationships | Focus on participants as co-researchers versus typical researcher and subject relationship |
| Integrity | Reflecting co-researchers’ viewpoints accurately |
| Competency | Components needed for the development of the group |
The intersection of Participatory Research and K-12 Special Education
My initiation and first exposure to participatory research were through my experience working with students with emotional and behavioral disabilities in a K-12 educational setting. Along with participatory research’s connection with social work values and ethics, one of the reasons for my interest in it as a methodology is related to previous experience engaging with a type of PR. Within my role as a program social worker working with students with emotional and behavioral disabilities, I support a set of classrooms K-12 working with students who have the most severe behavioral difficulties in the district. This population can be an extremely challenging group to know how to best support both the students and the staff who are working with them. One activity my school district engaged was to bring in a consultant to assist with assessing program needs and provide professional development for staff. The evaluation process this consultant used is an evidence-based program evaluation, a Participatory Evaluation Expert Review for Classrooms Serving Students with Emotional and Behavioral Disabilities (PEER-EBD). Tsai et al. (2013) describe the psychometrics and process of this evaluative process.
The general format for the participatory portion of the evaluation is through a questionnaire that staff complete individually that reviews 19 practice indicators and 93 specific sub-questions related to practice behaviors that align with research-based best practices for engaging with students with emotional and behavioral disabilities. The facilitator brings together the school team, which includes district and building administrators, school psychologists, special education teachers, paraeducators, and other staff working with the students in the program. The facilitator guides the team to come to censuses each of the practice indicators. Along with the facilitated team meetings, there is an expert review of the classroom looking for the same practice indicators in practice. Finally, based on areas of strengths and weaknesses, a set of recommendations can be developed and implemented to facilitate programmatic growth.
The PEER-EBD is not the only participatory research that has been done in a K-12 educational setting. There have been many, but as program evaluation, it has been implemented in different school districts around the nation more often than other individual participation research inquiries.
The intersection of Participatory Research and Resilience
Psychological resilience is a topic that has been looked some through models related to participatory research methods but appears not to have been pursued to its fullest extent. Often it has been examined through the lens of violence. Vindevogel et al. (2015) used a participatory research method in war-affected communities in northern Uganda. Ellis and Abdi (2017) report on how community-based participatory research can assist participants in overcoming cultural barriers, prevailing against stigma, and being able to build trusting relationships with fellow participants as ways that PR can directly connect to developing resilience.
Shaw et al. (2016) encourage resilience workers to expand their methodological and analytic toolboxes to include tools such as community-based participatory research. In relationship to resilience, the PR helps to ground researchers with the partnerships they have in the community. It is also a method that allows explicit attention to be drawn on systems. Participatory research facilitates precise attention on systems understanding of psychological resilience.
Reference
Balcazar, F. E., Keys, C. B., Kaplan, D. L., & Suarez-Balcazar, Y. (1998). Participatory action research and people with disabilities: Principles and challenges. Canadian Journal of Rehabilitation, 12, 105–112.
Bergold, J., & Thomas, S. (2012). Participatory Research Methods: A Methodological Approach in Motion (Partizipative Forschungsmethoden: Ein methodischer Ansatz in Bewegung). Historical Social Research, 37 (4), 191-222. https://doi.org/10.12759/HSR.37.2012.4.191-222
Boden, R., Greenwood, D. J., Hall, B., Levin, M., Marshall, J., & Wright, S. (2015). Action research in universities and higher education worldwide. In H. Bradbury (Ed.), The SAGE Handbook of Action Research (pp. 281-290). 1 Oliver’s Yard, 55 City Road London EC1Y 1SP: SAGE Publications Ltd. https://doi.org/10.4135/9781473921290.n28
Borda, O. F. (2008). The application of the social sciences’ contemporary issues to work on participatory action research. Human Organization; Oklahoma City, 67(4), 359-361.
Burdine, J. N., McLeroy, K., Blakely, C., Wendel, M. L., & Felix, M. R. J. (2010). Community-based participatory research and community health development. The Journal of Primary Prevention, 31(1), 1-7. https://doi.org/10.1007/s10935-010-0205-9
Cavanaugh, B. (2016). Trauma-informed classrooms and schools. Beyond Behavior, 25(2), 41-46. https://doi.org/10.1177/107429561602500206
Cochran, P. A. L., Marshall, C. A., Garcia-Downing, C., Kendall, E., Cook, D., McCubbin, L., & Gover, R. M. S. (2008). Indigenous ways of knowing: Implications for participatory research and community. American Journal of Public Health, 98(1), 22-27. https://doi.org/10.2105/AJPH.2006.093641
Council on Social Work Education. (2015). Educational policy and accreditation standards for baccalaureate and master’s social work programs. Alexandria, VA. Retrieved from http://www.cswe.org/File.aspx?id=81660
Dombo, E. A., & Sabatino, C. A. (2019). Creating trauma-informed schools: A guide for school social workers and educators. New York, NY: Oxford University Press.
Ellis, B. H., & Abdi, S. (2017). Building community resilience to violent extremism through genuine partnerships. American Psychologist, 72(3), 289–300. https://doi.org/10.1037/amp0000065
Freire, P. (2000). Pedagogy of the oppressed (30th-anniversary ed). New York: Continuum (Original work published 1970).
Hoffman, S., Palladino, J. M., & Barnett, J. (2007). Compassion fatigue as a theoretical framework to help understand burnout among special education teachers. Journal of Ethnographic & Qualitative Research, 2(1), 15–22.
Individuals with Disability Education Act Amendments of 1997 [IDEA]. (1997). Retrieved from https://www.congress.gov/105/plaws/publ17/PLAW-105publ17.pdf
Killam, L. (2013). Research terminology simplified: Paradigms, axiology, ontology, epistemology, and methodology. Author.
Kirst-Ashman, K. K., & Hull, Jr., G. H. (2015). Understanding Generalist Practice (Seventh Ed). Stamford, CT: Cengage Learning.
Knight, B. A. (1999). Towards Inclusion of Students with Special Educational Needs in the Regular Classroom. Support for Learning, 14(1), 3–7. https://doi.org/10.1111/1467-9604.00091
Lane, K. L., Wehby, J. H., Little, M. A., & Cooley, C. (2005). Students educated in self-contained classrooms and self-contained schools: Part II—how do they progress over time? Behavioral Disorders, 30(4), 363–374. https://doi.org/10.1177/019874290503000408
Lincoln, Y. S., Lynham, S. A., & Guba, E. G. (2011). 6 - Paradigmatic controversies, contradictions, and emerging confluences, revisited. In N. K. Denzin & Y. S. Lincoln (Eds.), The Sage handbook of qualitative research (4th ed, pp. 97-128). Thousand Oaks: Sage.
National Association of Social Workers. (2017). NASW code of ethics. https://www.socialworkers.org/About/Ethics/Code-of-Ethics/Code-of-Ethics-English
Rogers, C. R. (1979). The foundations of the person-centered approach. Education, 100(2), 98-108.
Shaw, J., McLean, K. C., Taylor, B., Swartout, K., & Querna, K. (2016). Beyond resilience: Why we need to look at systems too. Psychology of Violence, 6(1), 34–41. https://doi.org/10.1037/vio0000020
Streck, D. R., & Holiday, O. J. (2015). Research, participation, and social transformation: Grounding systematization of experiences in Latin American perspectives. In H. Bradbury (Ed.), The SAGE Handbook of Action Research (pp. 472-480). 1 Oliver’s Yard, 55 City Road London EC1Y 1SP: SAGE Publications Ltd. https://doi.org/10.4135/9781473921290.n47
Tsai, S.-F., Cheney, D., & Walker, B. (2013). Preliminary Psychometrics of the Participatory Evaluation and Expert Review for Classrooms Serving Students with Emotional/Behavioral Disabilities (PEER-EBD). Behavioral Disorders, 38(3), p137-153. 17p. 1 Diagram, 5 Charts. https://doi.org/10.1177/019874291303800303
U.S. Department of Education. Institute of Education Science. National Center for Education Statistics. (2019) Students with disabilities, inclusion of. Digest of education statistics, 2018. https://nces.ed.gov/fastfacts/display.asp?id=59
Vindevogel, S., Ager, A., Schiltz, J., Broekaert, E., & Derluyn, I. (2015). Toward a culturally sensitive conceptualization of resilience: Participatory research with war-affected communities in northern Uganda. Transcultural Psychiatry, 52(3), 396–416. https://doi.org/10.1177/1363461514565852
Appendices
Author Note
The following essay was originally posted as a paper for TSD 8215 - Varieties of Scholarly Experience as a part of my Ph.D. Studies in Transformative Studies at the California Institute of Integral Studies.