Skip to main content

Advertisement

SpringerLink
Log in

A Multi-Level Examination of Stakeholder Perspectives of Implementation of Evidence-Based Practices in a Large Urban Publicly-Funded Mental Health System

  • Original Paper
  • Published:
Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

Our goal was to identify barriers and facilitators to the implementation of evidence-based practices from the perspectives of multiple stakeholders in a large publicly funded mental health system. We completed 56 interviews with three stakeholder groups: treatment developers (n = 7), agency administrators (n = 33), and system leadership (n = 16). The three stakeholder groups converged on the importance of inner (e.g., agency competing resources and demands, therapist educational background) and outer context (e.g., funding) factors as barriers to implementation. Potential threats to implementation and sustainability included the fiscal landscape of community mental health clinics and an evolving workforce. Intervention characteristics were rarely endorsed as barriers. Inner context, outer context, and intervention characteristics were all seen as important facilitators. All stakeholders endorsed the importance of coordinated collaboration across stakeholder groups within the system to successfully implement evidence-based practices.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Aarons, G. A., Ehrhart, M. G., Farahnak, L. R., & Hurlburt, M. S. (2015). Leadership and organizational change for implementation (LOCI): A randomized mixed method pilot study of a leadership and organization development intervention for evidence-based practice implementation. Implementation Science. doi:10.1186/s13012-014-0192-y.

    Google Scholar 

  • Aarons, G. A., Fettes, D. L., Hurlburt, M. S., Palinkas, L. A., Gunderson, L., Willging, C. E., & Chaffin, M. J. (2014). Collaboration, negotiation, and coalescence for interagency-collaborative teams to scale-up evidence-based practice. Journal of Clinical Child and Adolescent Psychology, 43(6), 915–928. doi:10.1080/15374416.2013.876642.

    Article  PubMed  PubMed Central  Google Scholar 

  • Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011a). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 4–23. doi:10.1007/s10488-010-0327-7.

    Article  PubMed  Google Scholar 

  • Aarons, G. A., Sommerfeld, D. H., & Willging, C. E. (2011b). The soft underbelly of system change: The role of leadership and organizational climate in turnover during statewide behavioral health reform. Psychological Services, 8(4), 269–281. doi:10.1037/a002619.

    Article  PubMed  PubMed Central  Google Scholar 

  • Aarons, G. A., Wells, R. S., Zagursky, K., Fettes, D. L., & Palinkas, L. A. (2009). Implementing evidence-based practice in community mental health agencies: A multiple stakeholder analysis. American Journal of Public Health, 99(11), 2087–2095. doi:10.2105/AJPH.2009.161711.

    Article  PubMed  PubMed Central  Google Scholar 

  • American Psychological Association (2009) Partnering with communication scientists, collaborating across disciplines and leveraging impact to promote children’s mental health. Report of Healthy Development: A Summit on Young Children’s Mental Health. Washington, DC: Society for Research in Child Development.

  • American Psychological Association Presidential Task Force on Evidence-Based Practic. (2006). Evidence-based practice in psychology. The American Psychologist, 61(4), 271–285. doi:10.1037/0003-066X.61.4.271.

    Article  Google Scholar 

  • Beidas, R. S., Aarons, G., Barg, F., Evans, A., Hadley, T., Hoagwood, K.,…, Mandell, D. S. (2013). Policy to implementation: Evidence-based practice in community mental health–Study protocol. Implementation Science, 8(38). doi: 10.1186/1748-5908-8-38.

  • Beidas, R. S., Marcus, S., Aarons, G. A., Hoagwood, K. E., Schoenwald, S., Evans, A. C.,…, Mandell, D. S. (2015). Predictors of community therapists’ use of therapy techniques in a large public mental health system. JAMA Pediatrics. doi:10.1001/jamapediatrics.2014.3736.

  • Beidas, R. S., Marcus, S., Benjamin Wolk, C., Powell, B., Aarons, G., Evans, C.,…, Mandell, D. S. (2015). A prospective examination of clinician and supervisor turnover within the context of implementation of evidence-based practices in a publicly-funded mental health system. Administration and Policy in Mental Health and Mental Health Services Research. doi: 10.1007/s10488-015-0673-6.

  • Bond, G. R., Drake, R. E., McHugo, G. J., Peterson, A. E., Jones, A. M., & Williams, J. (2014). Long-term sustainability of evidence-based practices in community mental health agencies. Administration and Policy in Mental Health and Mental Health Services Research, 41(2), 228–236. doi:10.1007/s10488-012-0461-5.

    Article  PubMed  Google Scholar 

  • Bradley, E. H., Curry, L. A., & Devers, K. J. (2007). Qualitative data analysis for health services research: Developing taxonomy, themes, and theory. Health Services Research Journal, 42(4), 1758–1772. doi:10.1111/j.1475-6773.2006.00684.x.

    Article  Google Scholar 

  • Casper, E. S. (2008). Using implementation intentions to teach practitioners: Changing practice behaviors via continuing education. Psychiatric Services, 59(7), 747–752. doi:10.1176/appi.ps.59.7.747.

    Article  PubMed  Google Scholar 

  • Chambers, D. A., & Azrin, S. T. (2013). Research and services partnerships: Partnership: A fundamental component of dissemination and implementation research. Psychiatric Services, 64(6), 509–511. doi:10.1176/appi.ps.201300032.

    Article  PubMed  Google Scholar 

  • Cohen, J. A., Deblinger, E., Mannarino, A. P., & Steer, R. A. (2004). A multisite, randomized controlled trial for children with sexual abuse-related PTSD symptoms. Journal of the American Academy of Child and Adolescent Psychiatry, 43(4), 393–402. doi:10.1097/00004583-200404000-00005.

    Article  PubMed  PubMed Central  Google Scholar 

  • Creed, T., Stirman, S., Evans, A., & Beck, A. (2014). A model for implementation of cognitive therapy in community mental health: The Beck Initiative. The Behavior Therapist, 37(3), 56–65.

    Google Scholar 

  • Croft, B., & Parish, S. L. (2013). Care integration in the Patient Protection and Affordable Care Act: Implications for behavioral health. Administration and Policy in Mental Health and Mental Health Services Research, 40(4), 258–263. doi:10.1007/s10488-012-0405-0.

    Article  PubMed  Google Scholar 

  • Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science. doi:10.1186/1748-5908-4-50.

    PubMed  PubMed Central  Google Scholar 

  • Fixsen, D. L., Blase, K. A., Naoom, S. F., & Wallace, F. (2009). Core implementation components. Research on Social Work Practice, 19(5), 531–540. doi:10.1177/1049731509335549.

    Article  Google Scholar 

  • Foa, E. B., Hembree, E. A., Cahill, S. P., Rauch, S. A., Riggs, D. S., Feeny, N. C., & Yadin, E. (2005). Randomized trial of prolonged exposure for posttraumatic stress disorder with and without cognitive restructuring: Outcome at academic and community clinics. Journal of Consulting and Clinical Psychology, 73(5), 953–964. doi:10.1037/0022-006X.73.5.953.

    Article  PubMed  Google Scholar 

  • Green, A. E., & Aarons, G. A. (2011). A comparison of policy and direct practice stakeholder perceptions of factors affecting evidence-based practice implementation using concept mapping. Implementation Science. doi:10.1186/1748-5908-6-104.

    Google Scholar 

  • Hoge, M. A., & Morris, J. A. (2004). Implementing best practices in behavioral health workforce education-building a change agenda. Administration and Policy in Mental Health and Mental Health Services Research, 32(2), 85–89. doi:10.1023/B:APIH.0000042741.19527.a8.

    Article  PubMed  Google Scholar 

  • Hoge, M. A., Morris, J. A., Stuart, G. W., Huey, L. Y., Bergeson, S., Flaherty, M. T.,…, Madenwald, K. (2009). A national action plan for workforce development in behavioral health. Psychiatric Services, 60(7), 883–887. doi:10.1176/appi.ps.60.7.883.

  • Hurlburt, M., Aarons, G. A., Fettes, D., Willging, C., Gunderson, L., & Chaffin, M. J. (2014). Interagency collaborative team model for capacity building to scale-up evidence-based practice. Children and Youth Services Review, 39, 160–168. doi:10.1016/j.childyouth2013.10.005.

    Article  PubMed  PubMed Central  Google Scholar 

  • Institute of Medicine Committee on the Quality of Health Care in America. (2001). Crossing the quality chasm: A new health system for the 21st Century. Washington, DC: National Academy Press.

    Google Scholar 

  • Isett, K. R., Burnam, M. A., Coleman-Beattie, B., Hyde, P. S., Morrissey, J. P., Magnabosco, J.,…, Goldman, H. H. (2007). The state policy context of implementation issues for evidence-based practices in mental health. Psychiatric Services, 58(7), 914–921. doi:10.1176/appi.ps.58.7.914.

  • Kutash, K., Acri, M., Pollock, M., Armusewicz, K., Serene Olin, S. C., & Hoagwood, K. E. (2014). Quality indicators for multidisciplinary team functioning in community-based children’s mental health services. Administration and Policy in Mental Health and Mental Health Services Research, 41(1), 55–68. doi:10.1007/s10488-013-0508-2.

    Article  PubMed  PubMed Central  Google Scholar 

  • Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.

    Article  CAS  PubMed  Google Scholar 

  • Langley, A. K., Nadeem, E., Kataoka, S. H., Stein, B. D., & Jaycox, L. H. (2010). Evidence-based mental health programs in schools: Barriers and facilitators of successful implementation. School Mental Health, 2(3), 105–113. doi:10.1007/s12310-010-9038-1.

    Article  PubMed  PubMed Central  Google Scholar 

  • Linehan, M. M., Comtois, K. A., Murray, A. M., Brown, M. Z., Gallop, R. J., Heard, H. L.,…, Lindenboim, N. (2006). Two-year randomized controlled trial and follow-up of dialectical behavior therapy vs therapy by experts for suicidal behaviors and borderline personality disorder. Archives of General Psychiatry, 63(7), 757–766. doi: 10.1001/archpsyc.63.7.757.

  • Lloyd, D. (1998). How to maximize service capacity: Nuts and bolts solutions for implementing change in behavioral healthcare organizations below the senior management level. Rockville, MD: National Council for Community Behavioral Healthcare.

    Google Scholar 

  • Lyon, A. R., Ludwig, K., Romano, E., Koltracht, J., Stoep, A. V., & McCauley, E. (2014). Using modular psychotherapy in school mental health: Provider perspectives on intervention-setting fit. Journal of Clinical Child and Adolescent Psychology, 43(6), 890–901. doi:10.1080/15374416.2013.843460.

    Article  PubMed  Google Scholar 

  • Maslow, A. H. (1943). A theory of human motivation. Psychological Review, 50, 370–396.

    Article  Google Scholar 

  • McHugh, R. K., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatments. A review of current efforts. American Psychologist, 65(2), 73–84. doi:10.1037/a0018121.

    Article  PubMed  Google Scholar 

  • Pagoto, S. L., Spring, B., Coups, E. J., Mulvaney, S., Coutu, M. F., & Ozakinci, G. (2007). Barriers and facilitators of evidence-based practice perceived by behavioral science health professionals. Journal of Clinical Psychology, 63(7), 695–705. doi:10.1002/jclp.20376.

    Article  PubMed  Google Scholar 

  • Palinkas, L. A., Holloway, I. W., Rice, E., Fuentes, D., Wu, Q., & Chamberlain, P. (2011). Social networks and implementation of evidence-based practices in public youth-serving systems: A mixed-methods study. Implementation Science. doi:10.1186/1748-5908-6-113.

    PubMed  PubMed Central  Google Scholar 

  • Palinkas, L. A., Olin, S., Chor, B., Young Um, M., Hyeon Jeong, C., O’Connor, B.,…, Hoagwood, K. (2014). Influence of organizational role, consensus and innovation status on perceived facilitators and barriers to adoption of evidence-based practices. Paper presented at the 7th Annual Conference on the Science of Dissemination and Implementation, Bethesda, MD.

  • Palinkas, L. A., Schoenwald, S. K., Hoagwood, K., Landsverk, J., Chorpita, B. F., Weisz, J. R., & The Research Network on Youth Mental Health. (2008). An ethnographic study of implementation of evidence-based treatments in child mental health: First steps. Psychiatric Services, 59(7), 738–746. doi:10.1176/appi.ps.59.7.738.

    Article  PubMed  Google Scholar 

  • Powell, B. J., McMillen, J. C., Proctor, E. K., Carpenter, C. R., Griffey, R. T., Bunger, A. C.,…, York, J. L. (2012). A compilation of strategies for implementing clinical innovations in health and mental health. Medical Care Research and Review, 69(2), 123–157. doi:10.1177/1077558711430690.

  • Powell, B.J., Beidas, R.S., Stewart, R.E., Wolk, C.B., Rubin, R.M., Matlin, S.L., Weaver, S., Hurford, M.O., Evans, A.C., Hadley, T., & Mandell, D.S. (in review). Applying the policy ecology framework to Philadlephia’s Behavioral Health Transformation Efforts.

  • Proctor, E. K., Powell, B. J., Baumann, A. A., Hamilton, A. M., & Santens, R. L. (2012). Writing implementation research grant proposals: Ten key ingredients. Implementation Science. doi:10.1186/1748-5908-7-96.

    Google Scholar 

  • Rapp, C. A., Etzel-Wise, D., Marty, D., Coffman, M., Carlson, L., Asher, D.,…, Holter, M. (2010). Barriers to evidence-based practice implementation: Results of a qualitative study. Community Mental Health Journal, 46(2), 112–118. doi: 10.1007/s10597-009-9238-z.

  • Ringle, V., Read, K., Edmunds, J., Brodman, D., Kendall, P., Barg, F., & Beidas, R. S. (2015). Barriers and facilitators to the implementation of cognitive-behavioral therapy for youth anxiety in the community. Psychiatric Services, 66(9), 938–945. doi:10.1176/appi.ps.201400134.

    Article  PubMed  PubMed Central  Google Scholar 

  • Sackett, D. L., Rosenberg, W. M., Gray, J. A., Haynes, R. B., & Richardson, W. S. (1996). Evidence based medicine: What it is and what it isn’t. BMJ, 312(7023), 71–72. doi:10.1136/bmj.312.7023.71.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Scaccia, J., Cook, B., Lamont, A., Wandersman, A., Castellow, J., Katz, J., & Beidas, R. S. (2015). A practical implementation science heuristic for organizational readiness: R = MC2. Journal of Community Psychology, 43(4), 484–501.

    Article  PubMed  PubMed Central  Google Scholar 

  • Southam-Gerow, M. A., Rodriguez, A., Chorpita, B. F., & Daleiden, E. L. (2012). Dissemination and implementation of evidence based treatments for youth: Challenges and recommendations. Professional Psychology-Research and Practice, 43(5), 527–534. doi:10.1037/a0029101.

    Article  Google Scholar 

  • Staw, B. M., Sandelands, L. E., & Dutton, J. E. (1981). Threat-rigidity effects in organizational-behavior—a multilevel analysis. Administrative Science Quarterly, 26(4), 501–524. doi:10.2307/2392337.

    Article  Google Scholar 

  • Stewart, R., Adams, D. R., Mandell, D., Hadley, T., Evans, A., Rubin, R., …, Beidas, R. S. (2015). The perfect storm: Collision of the business of mental health and the implementation of evidence-based practices. Psychiatric Services. doi:10.1176/appi.ps.201500392.

  • Stirman, S. W., Gutierrez-Colina, A., Toder, K., Esposito, G., Barg, F., Castro, F.,…, Crits-Christoph, P. (2013). Clinicians’ perspectives on Cognitive Therapy in community mental health settings: Implications for training and implementation. Administration and Policy in Mental Health and Mental Health Services Research, 40(4), 274–285. doi: 10.1007/s10488-012-0418-8.

  • Stirman, S. W., Spokas, M., Creed, T. A., Farabaugh, D. T., Bhar, S. S., Brown, G. K.,…, Beck, A. T. (2010). Training and consultation in evidence-based psychosocial treatments in public mental health settings: The ACCESS Model. Professional Psychology, Research and Practice, 41(1), 48–56. doi:10.1037/a0018099.

  • Trupin, E., Weiss, N., & Kerns, S. E. U. (2014). Social impact bonds behavioral health opportunities. JAMA Pediatrics, 168(11), 985–986. doi:10.1001/jamapediatrics.2014.1157.

    Article  PubMed  Google Scholar 

  • Willging, C. E., Green, A. E., Gunderson, L., Chaffin, M., & Aarons, G. A. (2015). From a “perfect storm” to “smooth sailing”: Policymaker perspectives on implementation and sustainment of an evidence-based practice in two states. Child Maltreatment, 20(1), 24–36. doi:10.1177/1077559514547384.

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

We are especially grateful for the support that the Department of Behavioral Health and Intellectual disAbility Services provided for this project, and for the Evidence Based Practice and Innovation (EPIC) group. Funding for this research project was supported by the following grants from NIMH: (K23MH099179, Beidas; F32MH103960, Stewart). Additionally, the preparation of this article was supported in part by the Implementation Research Institute (IRI), at the George Warren Brown School of Social Work, Washington University in St. Louis; through an award from the National Institute of Mental Health (R25 MH080916) and Quality Enhancement Research Initiative (QUERI), Department of Veterans Affairs Contract, Veterans Health Administration, Office of Research & Development, Health Services Research & Development Service. Dr. Beidas was an IRI fellow from 2012 to 2014.

Author information

Authors and Affiliations

  1. Department of Psychiatry, University of Pennsylvania Perelman School of Medicine, 3535 Market Street, 3015, Philadelphia, PA, 19104, USA

    Rinad S. Beidas, Rebecca E. Stewart, Danielle R. Adams, Tara Fernandez, Susanna Lustbader, Byron J. Powell, Arthur C. Evans, Matthew O. Hurford, Trevor Hadley & David S. Mandell

  2. Department of Psychiatry, University of California San Diego, La Jolla, USA

    Gregory A. Aarons

  3. Department of Psychiatry, New York University, New York, USA

    Kimberly E. Hoagwood

  4. Department of Behavioral Health and Intellectual disAbility Services, Philadelphia, USA

    Arthur C. Evans & Matthew O. Hurford

  5. Community Behavioral Health, Philadelphia, USA

    Matthew O. Hurford & Ronnie Rubin

  6. Department of Family Medicine and Community Health, University of Pennsylvania Perelman School of Medicine, Philadelphia, USA

    Frances K. Barg

Authors
  1. Rinad S. Beidas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rebecca E. Stewart
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Danielle R. Adams
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tara Fernandez
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Susanna Lustbader
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Byron J. Powell
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Gregory A. Aarons
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Kimberly E. Hoagwood
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Arthur C. Evans
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Matthew O. Hurford
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Ronnie Rubin
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Trevor Hadley
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. David S. Mandell
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Frances K. Barg
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rinad S. Beidas.

Appendices

Appendix 1: Stakeholder Perspectives of EBP Initiatives in Philadelphia: Qualitative Interview for Agency Stakeholders

Instructions: The Department of Behavioral Health and Intellectual disAbility Services (DBHIDS) has sponsored several Evidence-Based Training Initiatives in recent years. Evidence-based practices, or EBP, refer to integrating “the best available research with clinical expertise in the context of patient characteristics, culture and preferences.”

We are interested in understanding your experience with DBHIDS-sponsored EBP initiatives. You are on the front line and we believe you have important information to share with us due to your role in the agency. We are particularly interested in understanding agency- and system-wide factors that impact your ability to use evidence-based practices. When we say agency-wide, we mean factors specific to your agency. When we say system-wide, we mean factors specific across agencies within the Philadelphia public community mental health system.

Let’s start by discussing the initiatives your agency has participated in.

  1. 1.

    Tell us briefly about your experience with these initiatives and your responsibilities in implementing these initiatives in your agency.

When did you take on this responsibility (e.g., prior to starting, mid-way through)?

  1. 2.

    What was it about [initiative] that motivated or drove your agency to adopt it?*

  2. 3.

    Generally, what makes these initiatives successful and not successful at your agency? Compare and contrast if you have participated in more than one initiative.

What would you change? What could be done to make these initiatives more responsive, more relevant, and less burdensome?

Let’s talk specifically about your agency now.

  1. 1.

    What kind of infrastructure is needed at the agency-level to support these EBPs?

  2. 2.

    What is your perspective on the impact of these initiatives on your agency?

Is there a sense at the agency level that this EBP is clinically better (i.e., results in better outcomes) than what was being done before with clients?

  1. 3.

    Tell me about agency support at all levels around these various initiatives.

Be sure to get information about how all management levels including administrators support EBPs. Who at the agency is critical to making the EBP a success? When we say support, we mean reducing productivity requirements, problem-solving, supervision, making it easy to learn the practice.

Is the EBP integrated into regular supervision, additional supervision/meeting time documentation, intake, screening or ongoing training provided by the agency?

  1. 4.

    What is the plan in your agency to continue the use of EBPs for the long term?

What is the single greatest barrier or facilitator to long term sustainment of any EBP? What type of infrastructure would be needed to maintain these EBPs?

  1. 5.

    Tell me about the financial realities of implementing EBPs in your agency.

What costs are you incurring to implement EBPs? What is the financial value to having EBPs at your agency?

Are your therapists fee-for-service or salaried? How does that impact implementation?

How and what kind of financial incentives would help you increase use of EBPs in your agency? Does the potential for an enhanced rate or “preferred provider” designation make these initiatives more feasible?

ONLY FOR AGENCIES PARTICIPATING IN TF-CBT: Tell me about your experience with the enhanced rate provided for TF-CBT.

Tell me about the System

  1. 1.

    What kind of infrastructure is needed at the system-level to support these EBPs?

  2. 2.

    What is your perspective on the impact of these initiatives on the system (i.e., across agencies in Philadelphia)?

Across agencies, not just their agency. Are single initiatives perceived as having an overall impact of moving the system towards EBPs

  1. 3.

    Tell me about system-level support of these initiatives.

Be sure to get information about how/if the system supports EBPs. Who at the system-level is critical to making the EBP a success?

  1. 4.

    What do you think is important for policy-makers to know/understand about how to support the implementation of EBPs through these initiatives or other efforts moving forward at a system-level?

Appendix 2

See Table 3.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Beidas, R.S., Stewart, R.E., Adams, D.R. et al. A Multi-Level Examination of Stakeholder Perspectives of Implementation of Evidence-Based Practices in a Large Urban Publicly-Funded Mental Health System. Adm Policy Ment Health 43, 893–908 (2016). https://doi.org/10.1007/s10488-015-0705-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-015-0705-2

Keywords

Search

Navigation