Suicide remains a leading cause of death for young Australians. We need robust analytical tools to help us understand and respond to the dynamic complexity underlying suicide and suicidal behaviour. Participatory systems modelling leverages empirical data together with the experiential knowledge of stakeholders to inform model building, evaluation, and implementation. Efforts to evaluate collaborative deliberative processes such as participatory system modelling remain sparse. The aim of this study was to evaluate the feasibility, value and impact of adopting a participatory systems modelling approach for developing a youth suicide prevention system dynamics model in North-West Melbourne, Australia.

Three participatory workshops were conducted with an expert stakeholder group including young people with lived and living experience of self-harm/suicidal behaviour, family members, healthcare professionals, and policy-makers. Data collection was informed by an evaluation framework, including (i) briefing questionnaires, (ii) workshop recordings and field notes and (iii) evaluation questionnaires. Data from each source were analysed separately, using codebook and reflexive thematic analysis, and synthesized to identify patterns in meaning and broad concepts under the evaluation categories of feasibility, value and impact.

We found that the feasibility of undertaking an inclusive and accessible participatory systems modelling project was influenced by stakeholder cultural diversity, meaningful involvement of those with lived and living experience, accessibility of technical aspects of modelling and managing group dynamics. The value of adopting a participatory approach in developing a complex systems model was demonstrated in the collaborative learning that took place between the expert stakeholder group and the research team. Central to this was the experiential knowledge of young people who helped shape our understanding of the contextual drivers of self-harm and suicidal behaviour. Stakeholders’ knowledge of the regional health system informed the structure, logic and assumptions of the systems model.

This is one of very few evaluations of a participatory systems modelling project involving young people with lived and living experience of self-harm/suicidal behaviour and bereaved families. Our study offers novel theoretical and practical insights into the processes adopted to ensure the safe and inclusive involvement of diverse perspectives in developing a youth suicide prevention system dynamics model.

Suicide is the leading cause of death for Australians between 15 and 24 years [1]. In 2022, suicide deaths accounted for just over one third of all deaths in those aged 15–24 years (a significant increase from 2001, when suicide deaths accounted for 17% and 24% in those aged 15–17 and 18–24 years, respectively) [1]. The economic cost of youth suicide in Australia is estimated at $22 billion a year, including direct (ambulance costs, coronial inquiry, policing costs and funeral expenses), indirect costs (lost economic productivity) and intangible costs (bereavement costs) [2]. Despite increasing investment by the Australian government in suicide prevention initiatives [3], suicide remains an intractable public health problem.

We have previously argued that understanding and responding to the dynamic complexity underlying suicidal behaviour within a public health framework requires a shift in thinking [4]. System dynamics modelling (SDM) is a computer-assisted method that helps frame, test and simulate the causal processes and interactions that underlie complex behaviours or systems, such as suicide, to guide evidence-informed decision-making [5]. Using SDM, we can model multiple interacting and interrelated suicide risk and protective factors to understand likely pathways of suicidal behaviour while considering the complexity of health systems including factors such as capacity, access and workforce supply [6, 7]. We can subsequently simulate interventions, programs or policies of interest to determine what will work, for which groups of young people, over what time horizon and via which potential mechanisms [7, 8]. Policy-makers, health professionals, service planners and commissioners can use this tool to experiment with different “what-if” scenarios and ascertain which of these scenarios will deliver the best outcomes within the system before implementing decisions in the real world [9].

A unique selling point of systems modelling is that it may be guided by a participatory approach to model building, evaluation and implementation [10,11,12]. Participatory systems modelling (PSM) refers to “A purposeful learning process for action that engages the implicit and explicit knowledge of stakeholders to create formalized and shared representations of reality” ([13] p. 233). PSM brings together an interdisciplinary group of stakeholders to be actively involved in the development of complex models. Stakeholders are facilitated to work collaboratively to identify and structure the problem, map the causal mechanisms underlying the system, identify policy levers, prioritize and test policy scenarios and support the real-world implementation of those scenarios [8, 12, 14].

There is a proliferation of approaches to modelling with stakeholders guided by different tools and methods depending on the research questions asked, project needs, feasibility, timelines and model outputs [13]. The challenge is how to systematically document and evaluate collaborative deliberative processes, such as PSM, to capture how knowledge and learning is generated, the potential influence of relational and social dimensions of the decision-making process [12, 15], and those characteristics that differentiate nominal from truly transformative forms of participation [16]. In modelling projects where expert stakeholder groups include those whose voices have long been neglected due to perceived safety issues, such as young people with lived and living experience of self-harm or suicidal behaviour [17], the need for systematic evaluation of PSM process and outcomes becomes even more critical. However, despite the increasing popularity of adopting PSM to inform mental health and suicide prevention policy and practice [8, 11, 18], efforts to assess and evaluate PSM in terms of achieving its intended aims remain sparse [12, 19]. To date, there are no evaluations of PSM solely focusing on young people with lived and living experience of self-harm or suicidal behaviour and bereaved families.

A novel, comprehensive multi-scale framework guiding the evaluation of PSM has recently been published [18] consisting of four categories (including 28 criteria). These categories are feasibility (that is, Is participatory modelling feasible?), value (that is, What is the value of participatory modelling when building complex systems models?), impact/change (that is, What changed as a result of the participatory modelling?), and sustainability (that is, What are the longer-term outcomes of the participatory modelling?). Each of these categories evaluates the impacts of PSM across four levels: individual, group, project, and system. The framework (Supplementary Material 1) has previously been adopted to evaluate the process and outcomes of PSM used to develop a complex youth mental health systems model in the Australian Capital Territory [19]. The authors found that PSM enabled the involvement of a diverse group of stakeholders, including those with lived experience of mental ill-health, who worked collaboratively to develop a model that had the potential to support evidence-based policy and planning in youth mental health [19]. PSM facilitated better understanding of contextual barriers and facilitators for the implementation of the model findings [19].

The aim of the present study was to evaluate the PSM process used for developing a system dynamics model to inform which programs and policies across health and community settings have the potential to generate the most significant reduction in rates of suicide and attempted suicide among young people aged 12–25 years in North-West Melbourne in Victoria, Australia. The study adopted the Lee et al. framework [18] to address the following questions:

The study is part of a wider programme of work [System Dynamics Modelling for Suicide Prevention (SEYMOUR)] funded by the European Commission [17] aiming to inform youth suicide prevention policy and practice in Australia and the UK. In Australia, the project took place in North-West Melbourne, which includes some of Victoria’s most socioeconomically disadvantaged and culturally diverse areas [20].

The North-West Melbourne Primary Health Network (NWMPHN), an independent organization funded by the Australian Government to coordinate primary healthcare in the region, was identified as the primary partner agency for the modelling project. The research team worked closely with representatives from the Mental Health and Suicide Prevention unit of the NWMPHN to discuss local priorities, scope the model and its boundaries and facilitate engagement with local, regional and state-level expert stakeholders for the participatory workshops. The organization was in the process of developing a new suicide prevention strategy for the region, and the project’s focus was closely aligned with local needs and priorities.

The timing of the modelling project also coincided with the implementation of the recommendations from The Royal Commission into Victoria’s Mental Health System established in 2019 with the aim of reforming Victoria’s mental health and well-being system. Developing a government-wide approach to suicide prevention was one of the key recommendations of the Royal Commission [21].

A description of the study design is provided in the study protocol [17]. Three participatory workshops were conducted between August 2022 and December 2022. The research team worked collaboratively with an expert group of stakeholders, including (i) young people aged 12–25 years with lived/living experience of self-harm and/or suicidal behaviour; (ii) family members/carers of young people; (iii) primary care and allied health professionals and youth mental health clinicians; (iv) health department policy-makers; and (v) service planners and commissioners. Together, these stakeholders conceptually mapped the causal pathways for the development of suicidal behaviour in young people, and their help-seeking journey through health and non-health settings (for example, community services) including mental health service pathways.

A detailed description of participant recruitment and the participatory workshops is provided in Supplementary Material 2.

The following procedures were implemented to create a safe environment for the involvement of young people with lived and living of self-harm and suicidal behaviour [17]:

A range of data collection methods was used to systematically document and evaluate the feasibility, value and impact of PSM process before, during and after the workshops. The evaluation plan and data collection methods are presented in Table 1.

A one-to-one online briefing session took place between the first author (M.M.) and young people to explore barriers and facilitators to their involvement and measures to ensure their safe and inclusive participation. Young people were asked to complete a short questionnaire asking: (1) What are your expectations or drivers for taking part in this study; (2) are there challenges or concerns you have about your participation, and if yes, would you like to share these; and (3) what would keep you engaged in this study/are there any barriers to your continued engagement?

The research team kept summary points using post-it notes, butcher paper and a whiteboard to document and evaluate the feasibility, value and impact of PSM. Workshops were audio-recorded, and two researchers (D.K. and M.V.) kept detailed field notes including (i) descriptions of activities, observations, behaviour, verbal and non-verbal communication patterns and discursive interactions as well as potential conflicts, disagreements, decision-making or collaboration between participants; (ii) interpretations of what was observed during the workshop, for example, how different stakeholders interacted (or did not interact), how stakeholders behaved towards each other and explanations of why this was important in relation to the aim of the workshop; and (iii) reflections of the researcher’s own thoughts, insights, meanings or concerns they might have as they were conducting the observation and any impact they, as an observer, might have had on the workshop. Workshops were facilitated by M.M. and K.W., both of whom are experienced in stakeholder engagement and workshop facilitation and have received the Applied Suicide Intervention Skills Training. M.M. and K.W. held regular planning meetings to establish clear workshop objectives, tailor content to participants’ unique needs and coordinate logistical details, ensuring a safe and inclusive environment. Throughout the process, they drew on J.-A.O.’s expertise in running PSM workshops, while J.W. and J.R. offered mentoring on effectively facilitating and managing complex discussions.

The research team conducted three one-to-one consultations with participants who were not available to attend the workshops but wished to be involved and participants who, following the workshops, wished to share additional information or insights in relation to the model development. M.M. wrote a reflective diary recording thoughts and experiences from stakeholder engagement activities, including notes from online and face-to-face meetings, day-to-day procedures and personal introspections.

Following each workshop, the research team emailed each stakeholder a debrief questionnaire to collect feedback on (i) aspects of the workshop that worked well and why, (ii) aspects of the workshop that did not work so well and why, (iii) how the delivery of the next workshop could be improved, (iv) whether there were enough opportunities for stakeholders to meaningfully participate in the workshop (for example, in sharing experiences) and (v) whether any aspects of the workshop were upsetting/distressing, and if so, what helped/did not help. This last question was specifically addressed to those participants with experiential knowledge of self-harm and suicidal behaviour (that is young people and family members/carers).

Data from each source (briefing and debriefing questionnaires and field notes/consultations) were analysed separately in relation to the three research questions of feasibility, value and impact of PSM.

Field notes kept during the workshops were analysed by M.M. using thematic analysis following a latent, inductive codebook approach [22]. M.M. listened to each audio recording before familiarizing herself through repeated reading with each set of field notes. Line-by-line coding of the field notes taken during workshop 1 led to the development of an initial codebook containing code names and definitions for each code, a quote from the data attached to each code and the evaluation category (feasibility, value and impact/change) and criteria that the code might map onto. During the coding of data from workshops 2 and 3, the initial codebook was amended through a process of constant revision. No qualitative analysis software was used. Throughout the data analysis process, M.M. met regularly with J.W. and J.R. to discuss patterns in the data and their meaning, as well as reflect on themes and how these might relate to the categories in the Lee et al. [18] evaluation framework. During these discussions, occasional disagreements arose over the interpretation and categorization of emerging themes and sub-themes (for example whether specific responses should be grouped under existing themes or treated as distinct categories). These disagreements were addressed through open, iterative discussions. Preliminary themes and codes were presented to the wider research team. This collaborative review allowed for multiple perspectives to be considered, ensuring that any discrepancies were carefully examined and reconciled through discussion and reflection. Field notes from the three workshops were analysed in the same way and included in the final codebook.

Free text from the briefing questionnaires was coded by M.M. who read through all responses and subsequently identified and extracted key components relevant to the research questions guiding the analysis. These components were grouped by common themes under the categories: challenges or concerns about participation, what would facilitate young people’s continued engagement in the study and expectations or motivation for participating. The same process was followed in the analysis of the debriefing questionnaires, the findings of which are presented under the categories: aspects of the workshop that worked well, aspects of the workshop that did not work well, aspects of the workshops that might have been upsetting or distressing and suggestions for improving subsequent workshops. Insights from the reflective diary were used to contextualize the understanding of stakeholder engagement during the PSM process, including the researcher’s own influence of this context.

Findings across the three data sources were reviewed to explore patterns in meaning and broad concepts that could be clustered around the PSM evaluation categories of feasibility, value and impact and the specific evaluation questions under each category.

In total, 16 participants consented to taking part in the workshops, 3 (2 young people and 1 health professional) of whom dropped out after consent, citing competing priorities and moving on to a new job as reasons for their withdrawal. The final expert stakeholder group included 13 participants. Further demographic information and workshop attendance is presented in Table 2.

Around half (53%) of our expert stakeholders identified as having experiential knowledge of self-harm and/or suicidal behaviour: 30% identified as young people with lived and living experience of self-harm and/or suicidal behaviour and 23% as family members or carers of a young person with lived and living experience.

Pre-workshop activities

The briefing questionnaire was completed by the four young people with lived and living experience of self-harm and/or suicidal behaviour who took part in the workshops. Findings are presented under three categories: (i) challenges or concerns about participation, (ii) what would facilitate young people’s continued engagement in the study and (iii) expectations or drivers for taking part in this study. The findings from the first two categories address the question around the value of PSM highlighting the barriers and facilitators to developing systems models through participatory methods. Findings around young people’s expectations and drivers for taking part do not map onto any evaluation category but do provide important contextual information on motivations for participation.

Young people referred to practical challenges that might impede their ongoing involvement in the project, including time constraints and availability due to competing priorities. One young person also mentioned feeling apprehensive about having to participate in a face-to-face meeting following the lifting of coronavirus disease 2019 (COVID-19) restrictions. Feeling nervous among other people and fear of how their anxiety might potentially influence their involvement in the workshops was also a concern. Although young people felt passionate about drawing upon their lived experience to inform the project, one of the concerns raised in their responses was talking about their lived experience in presence of “powerful” people referring to clinicians and commissioners. Concerns were also expressed about the possibility for some stories or experiences shared during the workshop to be triggering for others.

Young people offered suggestions for processes that could be implemented to support those with lived experience in preparation for and during the workshops. These included (i) giving young people enough notice about the date and timing of the workshops, (ii) sending young people the agenda and broad topics to be discussed in advance to give them time to prepare and (iii) incorporating processes to support young people’s meaningful involvement, including having a youth advocate, creating space and time specifically for them to share their thoughts and insights, acknowledging the role of lived and living experience throughout the workshops and having a designated quiet space for them to take a break, if needed.

Young people described feeling passionate about mental health advocacy for the purpose of improving the health sector (and ultimately helping others) and identified as a key driver for their involvement alignment between the study aim and their vision “Being part of a bigger purpose and vision such as to help young people, improve services and quality of life”. Being keen to contribute to a study which had the potential to generate useful outputs for their region played an important part in young people’s decision to participate. The focus of informing suicide prevention policy and practice in North-West Melbourne, a region described by young people as an “underserved area…it does not get a lot of attention but there is increased need for youth mental health support” was identified as a driver for their involvement.

Model-building workshops

Three superordinate themes were identified: (i) ways to facilitate effective stakeholder working, (ii) benefits of stakeholder engagement and (iii) changes to the model following stakeholder feedback. Each theme is presented in detail below and supported by quotes. The thematic framework is presented in Table 3. Findings from the model-building workshops offer important insights into the feasibility, value and impact of PSM.

Improving diversity and cultural representation within expert stakeholder group

Participants spoke extensively during the first workshop about the lack of diversity and cultural representation within the expert stakeholder group, referring to certain groups of young people not represented. Given that the region where the project was taking place includes some of Victoria’s most culturally diverse areas, participants highlighted the impact of lack of cultural representation on addressing the needs of specific communities in the region.

Many people are missing from the table (for example, Indigenous community). This is a mainstream white space. System is racist and everyone’s needs are not represented by this group (CF1).

Participants identified the project and its aims as highly important and timely due to the increased need for mental health support by young people in their region. However, to ensure that the project had real world relevance, participants offered suggestions to the research team about ways of improving the cultural representation among the expert stakeholder group in future workshops.

Cultural representation is important. If you cannot bring on the table certain groups of young people, then we can talk about bringing in people who work with young people to make this representation better (AHM1).

Participants working across health and community settings in the region offered to promote the project among their networks, foster relationships with community leaders and support the research team to engage with a diverse group of young people in the community to ensure their needs were sufficiently represented in the expert stakeholder group.

Observations of participants’ behaviour and interactions highlighted the presence of dominating voices in the room. This manifested in attempts to lead or navigate the discussion towards a certain (or preferred) path; taking control of the conversation during small-group activities by asking leading questions or diverting focus from the task at hand; referring to one’s position, experience or expertise to influence conversations; and not allowing other stakeholders to share their views. We provide below an extract from the field notes to illustrate this. During the second workshop, the facilitator reiterated that the focus of the model was on mapping health and community systems and how we could maximize those systems to improve suicide prevention initiatives in the region. One of the participants (SF1), who had not joined the first workshop, disagreed with the focus of the project being on systems and insisted that the team should be focusing on individuals and intrapersonal risk factors. In the extract below, having taken over the conversation and insisting on changing the focus of the mapping exercise, SF1 abruptly interrupts another stakeholder (CF1) in their attempt to bring back the conversation to the original aim.

SF1:We are talking about systems when we should be looking at ourselves talking about the individual.

CF1:But it’s also…that sense of individualism is a very western concept… [gets interrupted by SF1 before finishing their sentence]

SF1:Yes, now you are talking about individualism versus collectivism and that is the same thing as in order to heal yourself you will be healing your community.

Observations of power dynamics were reported throughout the three workshops and involved participants across all stakeholder groups. For example, young people who had prior youth advocacy experience appeared more confident in leading group activities than young people with no prior advocacy or research experience. During the intervention mapping exercise (workshop 3), it was noted that two young people had taken over the task of listing services and interventions in the region, conversing between themselves without including other young people at the table. The researcher taking the field notes made the following observation “The remaining young people are not contributing at all to the ‘discussion’ because it is very much (name of young person) listing the list of services, and another young person writing them down. This activity was definitely overpowered by (name of young person) views, and the other young people are not represented in here, and there is some indication that they don’t share all of [their] views”. In another example, a health professional was observed interacting with carers during a group task. The health professional appeared to be leading the discussion, asking one carer many questions including about a family member the carer was caring for, and their experience of mental health services. The extensive discussion of the family member’s case contributed to the carer’s discomfort. The carer eventually became upset and left the room for a while. When the carer returned, they told the group that they were going to completely leave the workshop, as “it’s become too much for me”. Several observations of the facilitators’ efforts to manage power dynamics within the expert stakeholder group were recorded throughout the workshops. Such attempts included: (i) reminding stakeholders of the ground rules, (ii) picking up signs when someone was trying to dominate or divert discussion and using this opportunity to re-clarify workshop aims, (iii) ensuring all voices were heard by taking stakeholder feedback into consideration when building the systems model, (iv) facilitating group discussions using guided questions and prompts, (v) ensuring circular flow of knowledge that is, checking and clarifying stakeholders’ understanding, and (vi) empowering participation by validating stakeholders’ views. We provide an example supported by relevant quotes below.

The example involves a conversation between a stakeholder (SF1) and one of the facilitators (F1) during the intervention mapping exercise about whether the policies of interest to be tested in the model should focus on prevention versus intervention. SF1, who participated in the workshop as a family member, was leading the small-group task and categorically disagreed with identifying suicide-related policies that would focus both on intervention and prevention. Despite repeated calls by other stakeholders for SF1 to be inclusive of everyone’s views on the table, SF1 became increasingly frustrated by the fact that others did not agree with their views. SF1 referred to their experiential knowledge of having lost a loved one to suicide to highlight the need to focus solely on prevention:

I don’t know how many people in the room have actually lost someone so close to suicide (which is relevant due to the relative amount of grief it causes)… I think people need to ask themselves: do you want suicides to stop, or just reduce? (SF1).

One of the facilitators (F1), having picked up signs of irritability in the group and some stakeholders being quiet and withdrawn, approached the group to ask about progress with the task.

SF1: Talking about [name of service] is irrelevant to be honest. We need to be talking about risk factors, what causes someone to develop suicidal ideation in the first place.

F1: We need to be talking about both because there will be people who are becoming increasingly unwell and for whom preventative efforts at that early stage might not work…and they might end up needing specialist help.

SF1: I guess what you are talking about is how much you want to focus on prevention versus intervention. Is it 50–50?

F1: We will focus on both. As agreed, we will go over the model now and at the second part of the workshop we will talk about interventions you will identify and where the gaps are. So, if you, as a group in your table, want to focus on prevention that is absolutely fine.

In the example above, as a way of managing the dynamics of power between SF1 and other stakeholders, the facilitator (i) acknowledges the importance of the stakeholder’s views (that is, focus on prevention) but also validates the opposite view (that is, focus on intervention), explaining why there is need for both; (ii) repeats and clarifies the aims of the task; (iii) explains the process the group needs to follow; and (iv) emphasizes the need to identify policies of interest “as a group”.

Facilitators had to communicate technical information and use modelling language that stakeholders were unfamiliar with. To ensure this information was accessible, a variety of methods were employed. For instance, when explaining how dynamic systems operate, a simple illustration – the flow of water in and out of a bathtub – was used. Instead of presenting intricate diagrams, simplified flowcharts and mind maps were provided. Charts and tables were used to present complex data, while concise summaries offered an overview of progress. Personal accounts and case studies (for example, challenges experienced by a bereaved parent accessing support for their child when feeling suicidal) were incorporated to illustrate system weaknesses that could serve as leverage points for intervention aimed at optimizing system behaviour. Facilitators ensured a circular flow of knowledge by continuously checking and clarifying stakeholders’ understanding and highlighting outstanding issues. Although these methods helped distil the technical aspects of modelling, some participants experienced difficulties in meaningfully engaging with certain activities referring to lack of familiarity with specific terms or difficulty in processing how information was presented. For example, a group-based activity during workshop 3 required stakeholders to define and describe “peer-led interventions”, as one of the prioritized interventions for testing, by working through a list of questions (Supplementary Material 3). Observations of the group’s interaction extracted from the field notes demonstrates their difficulty in relating to the technical language used in the list provided to them “The definition of the term components was too complex. The group asked for a definition but was confused with the explanation offered by the facilitator. Stakeholders had difficulty to define any component and what exactly they would want from the peer worker. The group could not complete the task and left this question for later. The group also appears very confused with the terms reach and adoption. The instructions need to be simplified and perhaps offer examples”.

In a similar example, a group of stakeholders were discussing the fact that there are young people whose presentation and needs will be too complex for primary care but who are not unwell enough for tertiary services, including hospital admission or state-run specialist mental health services. One stakeholder referred to the term “secondary services” as a way of bridging the divide between primary and tertiary services in the region. Everyone in the table appeared to be familiar with the term “secondary services” and engaged in conversation about what this would look like for the region of North-West Melbourne, except for one stakeholder:

I feel I don’t know enough about the service landscape to be able to decide on what secondary services should look like…I am just unfamiliar with the concept. (CF2).

Observations of this stakeholder’s non-verbal behaviour during this task mention “…[name of stakeholder] was feeling overwhelmed by the exercise and kept saying she doesn’t have the expertise to suggest things. Eventually, [name of stakeholder] asked for a break and left the room”.

The complexity of information and technical concepts of the workshop might have led to some voices being marginalized during the workshops.

The role of lived experience in shaping and informing the participatory modelling approach

Young people drew upon their personal experience of navigating the region’s health system to identify contextual factors, such as availability and accessibility of services, for understanding how the system does (or does not) work. The process of drawing upon personal stories to inform the systems model was identified by stakeholders as invaluable, offering credibility to the participatory approach and overall project.

A young person’s experience or story is different from collecting data at a single point in time with very specific questions. Stories give context to the data and help see how it is different from another person’s story/experience. Stories add dimension. Data (that you’d get from a questionnaire or other type of psychometric measure) from two people might be the same, but there are two different stories and contexts. (AHM1).

Participants spoke extensively about the value of adopting a bottom-up approach to developing a systems model by bringing together an interdisciplinary group of local experts and placing those with lived and living experience front and centre. Validation of young people’s experiential knowledge and transparency about how this knowledge provided context and learning during the model building process helped to close the gap between researchers and the community.

Researchers will spend a lot of time building an evidence base for treatment effectiveness, but this rationale is not available to the public, therefore there is no trust. (AHM1).

The concept of trust was often mentioned in participants’ narratives when referring to how the participatory modelling approach helped build trust in the relationship between the research team and the expert stakeholder group and, by doing so, increased confidence in stakeholders’ involvement in the model building process.

Sharing and exchanging of knowledge

Throughout the workshops stakeholders engaged with each other through dialogue and debate with some activities (for example, small-group discussions) encouraging more active and productive interaction manifested in sharing and exchanging knowledge, validating concerns but also challenging each other. Stakeholders often had different views on how regional services operated or different experiences of delivering and receiving services. It was through the process of dialogue and debate that stakeholders had the opportunity to discuss disagreements, ask questions and find resolution collectively. This process of collaborative learning helped to connect knowledge across the expert stakeholder team and foster in-depth understanding and awareness of the system.

The following extract involves four stakeholders – two health professionals (AHM1, CL_F_2), one commissioner (SP/C_F_1) and one carer (CF2) – debating the pros and cons of introducing a central coordinating system to manage mental health referrals, wait lists and pathways to care in the region. While three of the stakeholders drew upon their professional and experiential knowledge to argue for and against adding the central coordinating system as a hypothetical intervention to be tested in the model, another stakeholder informed the group that such an initiative [referred to as “client management interface” (CMI)] already exists in the state of Victoria.

AHM1:There are too many narrowly specialist services so there is a lot of siloing. We need to have a shared waitlist between the different services, some kind of a centralized coordination centre that manages it all.

SP/C_F_1:In the housing services, they have a central coordinating system; they have entry points but also an online data referral pathway where services will advertise what their services and capacity is. For example, in theory you’d have someone present to an access point, they would then be able to look at what available resources are e.g., a crisis bed, and then they hold that client until they are referred into the service; the service then engages with that client; they might support or find what else might be available after they’ve addressed their immediate needs. The challenge would be in this space…there are so many different players within the mental health system that requires coordination. So, I think that would be the greatest challenge.

CF2:We just need to be aware of the pros and cons of centralization and decentralization. I worked in the homeless services for 14 years and there are so many problems with it (referring to centralization of services). If it was working well, it would be great. I also want to talk about relationships between services…e.g., not knowing what [referring to name of service] criteria are…I find that odd. We all know that there are liaison roles, there are roles that are set up to do these things to help the system be more coordinated, but relationships are so important…good quality relationships.

CL_F_2:Are people familiar with the CMI [client management interface]? It’s statewide and you can see what services the person has accessed etc. There’s also a different system to see how many beds there are available statewide.

The sharing by CL_F_2 of this piece of information about the existence of CMI changed the course of the conversation from developing and modelling the effects of a central coordinating system, as a hypothetical scenario, to the potential for sourcing existing data to support the research team in evaluating its effectiveness. Additionally, CL_F_2’s input raised awareness among the expert stakeholder group about the existence of an important resource for the mental health sector facilitating the sharing of statutory data with the Victorian Department of Health and other services.

Stakeholder insight informing model structure and logic

Stakeholders provided unique knowledge on how the (mental) health system operates in their region, and this knowledge was used to inform, review and revise both the structure of the computer simulation model and the logic and assumptions underlying the model. The example below illustrates how stakeholder feedback led to the addition of an important source of help-seeking in the model, labelled “online services”, which the team had not previously considered.

CF1:I’ve just been thinking about means of suicide and we know that it’s different for young men and women; we know it’s different for First Nations communities; we know it’s different for LGBTQIA+ young people who have a harder time. So’ I’m just wondering how we…online services…young people could be looking online for services to seek help and that’s not captured there.

F1:So, perhaps what we need is an arrow from seeking help to online sources of help for example, helplines?

CF1:Yeah, helplines… all sorts of stuff

Stakeholders highlighted causal pathways between variables in the model and helped the research team understand dynamic behaviours that drive some of the vicious cycles identified in the model.

SP/C_F_1: Care in Mind and Head to Health are also available services in our region. However, just because someone “completes” a treatment, doesn’t mean they’re actually recovered and now mentally well. They end up returning to the [mental health] system.

F2:Like an “vicious loop”? So, the times someone goes through the system, the more it adds to their distress?

SP/C_F_1:Yes, and this then makes the system a contributing factor to a young person’s distress…

Stakeholders worked together to identify weaknesses in the system that could be used as leverage points where one could intervene to optimize system behaviour. In the example below, one participant raised the issue of long waiting times experienced by young people at different points in their journey (for example, between referral acceptance and first appointment) causing unnecessary delays in the system at a time when they are most vulnerable.

There are many points in the system where young people spend a lot of time waiting…not just [when moving] between services but also within the same service; for example, waiting between headspace appointments. Just because you are in a service that doesn’t mean you don’t have to wait for each appointment. (SP/C_F_1).

Stakeholders reflected on how the system could be improved through different policies, interventions or service planning decisions. The outcomes of the voting exercise (workshop 3) helped the team identify the top three interventions prioritized by stakeholders as the most locally appropriate and relevant for inclusion in the model [1]:

Post-workshop evaluation

Debrief questionnaires following the first, second and third workshops were completed by 7/9, 7/10, and 3/7 participants, respectively. The main themes from the debrief data are presented and supported by relevant quotes; these offer insights relating to the feasibility, value and impact of PSM.

Facilities and workshop format including delivery methods

All participants commented positively on the chosen setting, location and organization of the workshop. The format of the workshops, including the variety of delivery methods adopted was highlighted as key to accommodating the needs of a diverse group of stakeholders while meeting the project objectives. Being flexible about how information was communicated and the different ways in which stakeholders could share their insights (during and after the workshops) fostered inclusivity and meaningful engagement.

I thought the format was good too… The discussions, in smaller groups, before we committed to writing some of our ideas down on the sticky notes worked very well, the colour coding was clever and the full group discussion after bringing it all together was also really good. The range of participants also good. (CF1).

Small-group discussions gave participants the opportunity to share their views in a more informal way and have deeper conversations in a safe space while building up their confidence to speak in front of the wider group.

The small group discussions feeding into the larger, facilitated discussions also worked well in terms of feeling listened to and having opportunities to contribute – it’s easier to do so in a small group, and then can provide confidence to speak up in a larger group. (YP1).

Workshop facilitation

Workshop facilitation was highlighted in participants’ responses as one of the key factors for the success of the first workshop. Examples of facilitative leadership included the provision of a clear explanation of the project aims and process, offering multiple opportunities to stakeholders to be involved in different ways and being respectful, inclusive and validating of stakeholders’ views.

I am not a great fan of a lot of powerpoint but I thought you explained the process well and I understood the task. (CF1).

The facilitators successfully directed the conversations and allowed targeted but also diverse discussions. I definitely felt listened and had enough opportunities to contribute throughout the day. (AHM1).

Participants identified two key issues related to (i) the duration of the workshop (full-day), which was perceived as demanding and might have led to the number of participants slowly dropping during the afternoon session, and (ii) lack of adequate representation of young people with lived and living experience of self-harm and/or suicidal behaviour.

The fact that for the majority of the day I was the only young person didn’t really work too well – I think I managed fine, but it really wasn’t ideal. (YP1).

Participants identified three ways in which the second workshop could be improved. Firstly, there was a clear need to improve diversity and representation within the expert stakeholder group in relation to the involvement of those with lived and living experience.

I think there is a “human element” missing from the session/s. I would like to suggest that at some time in the day we hear a statement or a share from a young person or someone with lived experience to remember the seriousness of the impacts of everything to do with suicide and what helps in recovery – not to distress participants or bring the group down, but to remember the real human element in our modelling journey. (CF2).

It’s generally best practice to have at least 2 young people in any meeting such as this one, especially in an environment with mental health professionals. (YP1).

Secondly, participants recommended that the duration of future workshops should be shortened to a half-day to prevent attrition and ensure “continuity of contributions and more collaboration across the range of participants throughout the day”. (CF2).

Thirdly, as the development of the conceptual map became more complex, a suggestion was to allow more time for deeper thinking about the bigger picture in relation to improving the mental health systems in the region.

Only challenging part for me was that figuring out how to deal with the growing complexity of the model building process, as you may remember that the map became very complex by the end of the workshop. I felt like I lost my connection with the big picture a bit, as I focused more on the details. I may just suggest adding a section or allowing more time for attendees to further workshop specifically on how to fit all these detailed pathways in the big picture and encourage them to focus more on the ideas around how these services and pathways could be placed in a way that the map looks a little easier to understand and less overwhelming. (AHM1).

No adverse incidents^{Footnote 1} were reported during the first workshop. A certain level of apprehension and nervousness was noted by one carer who said: “I was a bit nervous and shy and worried about crying but everyone was lovely and welcoming and seemed to value my contribution” (CF2). Feelings of validation and the sense of having a shared goal was one of the reasons the workshop was described as “hope instilling”.

Not particularly [referring to feeling upset during the workshop]. I think it was a really great discussion and being surrounded by others looking for solutions to the problem was really hope-instilling. (YP1).

Having reviewed the feedback from workshop 1, the research team recruited two more youth stakeholders, shortened the workshop to half a day and allowed more time for discussion and reflection.

Workshop facilitation

Participants highlighted many examples of good practice, including facilitators encouraging participation but not putting pressure on stakeholders to share, supporting stakeholders in unpacking the complexity behind some of the topics discussed and clearly showing how stakeholders’ views and feedback were used to inform the content of the workshop.

In terms of the aspects of the workshop that worked well for me. Well, it’s more or less the fact that at no point was I pressured into speaking or anything like that. The relevant information was presented in a way that would help bridge the gap of not really being as informed as most people there. I was in-fact asked at multiple points if I had anything to share, and as such I feel that I was given ample opportunity to share my views and story. (YP2)

Workshop format and delivery methods

Participants commented positively on the revised organization and delivery of the second workshop. These changes, implemented in response to participants’ feedback, facilitated stakeholders’ involvement in a more meaningful way during the second workshop.

I think the reduced time was better for this second workshop and I like your mix of big picture and detailed small group work. I felt able to contribute. (CF1).

Four out of the seven participants who provided feedback on the second workshop highlighted the presence of “stronger” voices in the room as problematic. Participants referred to dominating voices which overpowered conversations, tried to navigate the discussion away from the aims of the workshop and repeatedly referred to their experience or expertise as a way of influencing conversations and invalidating or dismissing others’ contributions.

I’m aware that this isn’t necessarily something you can control, but at times some of the stronger voices in the room seemed to overpower and even invalidate the experiences of other young people, healthcare providers, etc. in the room, which may have made it difficult, particularly for young people, to speak up and voice their opinions on things. In the same vein, some of these views seemed to fall outside the aim of the model and led to the group being sidetracked and not being able to focus completely on the topic at hand. (YP1).

The impact that this had on group dynamics is evident in the quote below:

[Participant’s name] seemed to shut down other people mid-sentence if [they] did not agree with what they were saying. It made myself and at least one other person feel uncomfortable. There were times where I didn’t share what I wanted to say because I feared I’d be bluntly interrupted and shut down if this [person] did not like what I had to say. (YP3).

Going forward, participants highlighted the need for a more detailed debrief at the end of the workshop due to the sensitive nature of some of the conversations that took place.

Young people and carers reported feeling somewhat upset due to the nature of the topic of suicide and/or suicide bereavement. One young person reported feeling upset by the fact that there were stakeholders who overpowered the conversation and invalidated their lived experience. Despite these concerns, participants reported that things which helped manage their feelings were good facilitation (for example, managing group dynamics) and, for young people specifically, the use of strategies recorded in their wellness plan.

There were occasions that I felt upset or distressed, but not to any excessive degree. More that the topic of conversation was sometimes uncomfortable, and as such I tended to zone out on occasion. There isn’t as far as we know anything that can be done to reduce this in the future, as the subject of discussion is more or less the cause, and it is more than tolerable at those levels. (YP2).

Only three out of seven participants provided feedback following the completion of the third workshop. We provide a narrative summary of participants’ views of things that worked (or did not work) and how the PSM process could be improved in the future. No adverse incidents were reported. Participants highlighted the fact that having a smaller expert stakeholder group worked very well, as people were able to have more meaningful conversations and felt confident sharing their views without worrying about being judged. The quality of facilitative leadership was once again highlighted in relation to meaningfully incorporating lived experience and professional knowledge thereby ensuring inclusivity and transparency in the PSM.

[Name of facilitator 1] and [name of facilitator 2] facilitated the workshop very well and was inclusive (and validating) of all feedback and suggestions made. It was obvious that you both were appreciative of the insights provided and would make a concerted to utilise the feedback in a constructive way that reflects the intention of the feedback. Managing a space where you are incorporating lived experience and professional knowledge can always been challenging as both contributions are unique and valid, and at times, can be contrasting – you were able to navigate this space through validation and the through small group discussions. (SP/C_F_19).

In terms of improving future PSM workshops, participants had two suggestions: (i) delivering workshops in the community, as opposed at a mental health service, as this would improve inclusivity and accessibility and (ii) ensuring that small groups are mixed and incorporate both lived experience and professional knowledge.

Findings from each data source are summarized and tabulated (Table 4) to demonstrate patterns in meaning and identify broad concepts that clustered around the PSM evaluation categories. Data tabulation ensured that key findings from each data source were not overlooked, thereby enhancing the trustworthiness of the data analysis process, synthesis and interpretation of conclusions [23].

This study offers new insights into the feasibility, value and impact of adopting PSM to develop a complex systems model for youth suicide prevention in North-West Melbourne. Drawing upon the extant literature, we present a critical analysis of the study findings, which has informed the development of a theoretical framework of the key ingredients for the successful delivery of a PSM project.

The findings of the evaluation identified several factors that influenced the feasibility of undertaking an inclusive, accessible and transparent PSM project. These factors included appropriate representation of those with lived and living experience, stakeholder cultural diversity, translation of technical information and effective management of group dynamics.

Those with experiential knowledge of self-harm or suicidal behaviour were identified as key beneficiaries and were appropriately represented in the expert stakeholder group (53% of the sample identified as having lived and living experience, including young people and carers). The meaningful involvement of young people with lived and living experience was facilitated through the appointment of a youth advocate, ensuring that the views of young people were adequately represented and valued. The role of a youth advocate has been previously recommended as a key strategy for facilitating youth involvement in research, ensuring a positive working synergy between researchers and those with experiential knowledge [24].

Although lived and living experience of self-harm and/or suicidal behaviour was appropriately represented in the PSM, cultural diversity was not achieved despite the research team’s efforts.^{Footnote 2} Difficulties in recruiting all necessary stakeholder perspectives in the PSM process may have implications for the cultural appropriateness of the model we developed, as this might not accurately reflect the needs and experiences of all young people in the region. Involving culturally diverse communities in participatory youth mental health systems modelling entails adapting participatory processes to acknowledge the presence of cultural differences [25]. In our case, for example, hosting the participatory workshops in the community (for example youth club), as opposed in a clinical service, could have made it more feasible and accessible for young people from diverse cultural backgrounds to be involved.

In line with previous findings [13, 15, 19, 26, 27], effective facilitation was consistently highlighted by stakeholders as key to managing power dynamics, empowering participation and ensuring transparency in how professional and experiential knowledge were incorporated in the PSM process. Challenges in successfully incorporating the technical aspects of modelling and the experiential knowledge of stakeholders could have inadvertently compromised accessibility, thereby exacerbating power dynamics between the research team and stakeholder group [27]. The value of clear, transparent and non-technical communication, as a way of ensuring inclusivity in the PSM process, has been highlighted in a previous PSM evaluation study [19]. The lack of optimal management of the technical aspects of modelling highlights the need for reflective practice in PSM facilitation, allowing researchers to learn from successes and failures and apply those learnings to future projects [13]. Voinov et al. [13] highlight the importance of keeping detailed documentation of facilitation processes to encourage critical reflection and learning. The use of field notes and a reflective diary in our study significantly helped researchers in this respect by identifying examples of good practice in facilitation but also examples where practice could be improved in the future.

The value of adopting a participatory approach in developing the North-West Melbourne systems model is demonstrated in the collaborative learning that took place during the PSM. Despite the presence of epistemological differences among stakeholders, engaging in dialogue and debate – within the context of effective facilitation – was invaluable in fostering mutual learning and trust in the PSM. Bringing together different but complementary types of knowledge promoted a sense of shared ownership described by participants as “hope instilling”. Previous research [28] exploring stakeholders’ experiences of engaging in participatory modelling projects has also identified collaboration and co-production as highly valued elements of the participatory process enabling interdisciplinary communication and knowledge exchange.

Central to the process of fostering collaborative learning was the role of lived and living experience. Creating a safe space for young people to draw upon their own experience shaped our understanding of how the system works (and does not work) while at the same time affirming young people’s capacity to act as creators of knowledge. This has important implications for addressing epistemic injustice [29] in suicide prevention research whereby the voices of those with experiential knowledge, and specifically those of young people, are excluded or silenced due to concerns about inflicting harm and causing distress [17]. The present study, however, demonstrates that it is feasible, valuable and safe to include young people with lived and living experience of self-harm and/or suicidal behaviour in participatory systems modelling projects. Consulting with young people about potential safety issues and ways of facilitating their continued engagement, co-creating safety protocols tailored to their needs and ensuring the involvement of a youth advocate were key strategies which facilitated the creation of a safe and inclusive space for lived experience involvement. Within the context of repeated calls for addressing epistemic and structural injustice in mental health research [30, 31], we highlight the great potential of participatory systems modelling in legitimizing lived experience knowledge through the creation of inclusive and ethical research practices.

Stakeholders’ unique knowledge of the regional mental health system informed the structure, logic and assumptions of the systems model, thereby increasing its face validity. Contextual knowledge (professional and experiential) of service capacity issues, service access delays, resources and affordability helped identify weaknesses in the system and ways of intervening to improve system behaviour through the identification of locally relevant interventions and policies. Stakeholders also helped the research team identify data gaps and successfully negotiate data access for model calibration and validation.

Stakeholder feedback was considered throughout the project to enhance the PSM process. Young people’s responses to the briefing questionnaires provided useful insights into factors that might affect their participation but also suggestions for ensuring their continued involvement, which the research team implemented (for example, having a youth advocate). Post-workshop feedback enabled the iterative improvement of the PSM process by reviewing the format and delivery of the workshops (for example, shorter duration and increasing the number of youth stakeholders) while at the same time reflecting on aspects of the participatory approach that were considered effective (for example, facilitation) in enhancing stakeholder engagement. Lee et al. [19], in their evaluation of a youth mental health PSM process in the Australian Capital Territory (ACT), also highlight the importance of reflective practice in improving stakeholder experience of and engagement in PSM.

There is no gold standard in how PSM can best be implemented when developing complex systems models [16]. However, there are mechanisms, processes and principles which interact to create the optimal circumstances for conducting PSM in a way that is feasible, valuable and safe. Accurate documentation and reporting of these mechanisms taking place prior to, during and after the modelling process is key in understanding how we can move from nominal to transformative forms of PSM [16].

In their systematic assessment of group modelling building interventions, Rouwette et al. [32] adopt Pawson and Tilley’s [33] framework of context–mechanism–outcome (CMO) configurations to capture a comprehensive set of variables that could determine the success of group modelling building interventions. Using this framework and focusing on those variables most relevant and transferable to PSM, we present a theoretical framework of the SEYMOUR PSM approach informed by the findings of the present study (Table 5). This framework explores the relationship between the context, mechanisms and outcomes of the PSM, offering novel insights into the processes adopted to ensure the safe and inclusive involvement of young people with lived and living experience of self-harm and/or suicidal behaviour.

Context refers to all those pre-existing factors that facilitated the implementation of the PSM including: (i) the wider policy context supporting the reform of the mental health system in Victoria; (ii) alignment between the project aims and the primary partner agency’s vision for improved suicide prevention for young people in the region; (iii) stakeholder characteristics, experience and roles; and (iv) organization characteristics

Mechanisms refer to the components, resources or processes in the study which underlie or support the effective delivery of PSM approach [32] and include:

Outcomes, referring to the intended and unintended effects of the PSM, are based on the context and mechanisms interaction and are categorized into individual (for example, building of trust between research team and expert stakeholder group), group (for example, shared vision of the problem) and project (for example, prioritization of interventions for testing, ensuring model face validity).

This framework will form the basis for future testing and refinement through in-depth qualitative interviews with members of the expert stakeholder group who participated in this study (ongoing work). This framework, which serves as the architectural blueprint of the PSM approach we adopted, has the potential to delineate the key ingredients for the successful delivery of a PSM project which could be transferable and applicable to other system modelling projects in the field of mental health and suicide prevention.

The sample size of the participatory model development workshops could be considered relatively low compared with similar studies [19]. While the aim was to achieve experiential and demographic variation, our sample allowed for meaningful engagement with key stakeholders, ensuring that conversations were carried out in a safe, respectful and manageable way [34]. Indeed, stakeholder feedback highlighted the small-group format as suitable for having meaningful conversations.

In line with previous studies in the field of youth mental health and/or suicide prevention [11], our priority was to meaningfully embed local and contextual knowledge into the participant recruitment process. We acknowledge the lack of cultural diversity within our expert stakeholder group and highlight the importance of working closely with local youth organizations and community leaders to diversify recruitment and ensure young people’s needs are sufficiently represented in expert stakeholder groups. Holding workshops in community settings (for example, youth clubs) could improve inclusivity and accessibility. The debrief questionnaires were not completed by all participants despite several reminders; as a result, the feedback might not reflect everyone’s experience. Requesting that stakeholders complete evaluation questionnaires on-site upon completion of each workshop might improve completion rates.

CMO:

Context–mechanism–outcome

CMI :

Client Management Interface

NWMPHN:

North-West Melbourne Primary Health Network

PSM:

Participatory systems modelling

SDM:

System dynamics modelling

SEYMOUR :

System dynamics modelling for suicide prevention

Download references

M.M., J.R. and J.W. conceived the study idea. M.M., J.R., J.W. and J.O. designed the study. M.M., K.W., D.K., M.V. and M.L. supported data collection, stakeholder engagement and participant recruitment. M.M. analysed the data. J.W. and J.R. provided methodological input and advice on data analysis. A.S. supported the interpretation of the technical aspects of modelling. M.M. drafted the manuscript, and all authors contributed to the edit of the manuscript. All authors read and approved the final version.

Correspondence to Maria Michail.

The project has received ethical approval from the University of Melbourne Human Research Ethics Committee (2022-22885-25971-4). All participants provided informed consent to the study.

J.O. is both head of Systems Modelling, Simulation and Data Science at the University of Sydney’s Brain and Mind Centre and managing director of Computer Simulation and Advanced Research Technologies (CSART).

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.

Reprints and permissions