At the outbreak of the pandemic in Australia, which directly followed unprecedented bushfires, the PATH Through Life Study had just commenced data collection of its 5th wave for the 40s age cohort. Continuation of fieldwork required dynamic transition from an in-person structured assessment protocol, to remote assessment methods with adaptation of established measures. We aim to describe the methods used to adapt the longitudinal study to these events, and the implications of data collection methodology for analysis. Reflections on these experiences are essential for transparent reporting of protocol change, and for informing future study design.

Evaluation of the data collection process for the fifth follow-up assessment of the PATH Through Life Study, a population-based cohort of Australians aged 58–64 years (n = 2530). Evaluation metrics include response rates for interviews done in-person and remotely, observations from data collection modifications, and participant feedback. Additional online survey items measuring the impact of exposure to bushfires and COVID-19 were developed and deployed as part of the study.

Of 2147 contacted for follow-up, 1558 participants completed wave 5 (data collection commenced September 2019). By November 2019 (onset of the bushfires), 585 (37%) participants had completed face-to-face interviews and by March 2020, a further 1057 (68%) participants had completed their interviews at the onset of government restrictions relating to COVID-19. The shift to remote assessments resulted in 30% of assessments being completed by telephone. Challenges of the data protocol approach are discussed including the impact of multiple data collection pathways on analysis, and limitations of telephone interviews. Participant completion rates for the additional bushfire and COVID-19 surveys for the wave were 60% and 69% respectively; anecdotal participant feedback was positive regarding the inclusion of these measures.

Dynamic capture of exposure to unexpected events within the context of an established longitudinal study requires rapid design and protocol adaptations, and careful documentation of participation timing and pathways. Given the heightened public interest, participant response was encouraging, and the data captured not only enhances the value of the whole dataset, but is uniquely placed to address questions on population-level vulnerabilities and ongoing impacts of the exposures.

The end of 2019 and 2020 was a tumultuous period as parts of Australia experienced unprecedented severe summer bushfires, shortly followed by the first domestic outbreak from the global coronavirus disease (COVID-19) pandemic, caused by the SARS-CoV-2 virus. Both events had significant public health impacts. In the 2019-20 Australian bushfires, 33 people died, 3,094 houses were destroyed, and over 17 million hectares of land was burned across six of the seven states and territories of the country (Australian Capital Territory (ACT), New South Wales (NSW), Victoria, Queensland, Western Australia and South Australia) [1]. As the bushfires eased, the COVID-19 pandemic intensified, becoming a significant public health issue. From March 2020, Australian states were under ‘stay at home’ orders including the closure of interstate and international borders and non-essential businesses, and strict social distancing and hygiene measures were commonplace in an attempt to slow the spread of infection [2].

During global emergencies and natural disasters, individual changes in psychological and physical health are expected due to the perceived threat and instability that these events pose [3]. These events can also result in broader structural and societal impacts, such as reduced health service access, displacement or enforced isolation. It is therefore impossible to ignore the significant impact these external events have on participant health and well-being outcomes in health research. Longitudinal studies provide an excellent opportunity to capture such change, particularly during early life [4] or late life, due to the transitional nature of these life periods and the vulnerability to disruption from external crises. However, there are practical difficulties related to capturing change in a way that maintains methodological rigour [particularly in already established studies), while also rapidly adapting to accommodate such unforeseen but significant exposures of interest.

As a result of the COVID-19 pandemic, many research studies were forced to amend face-to-face data collection methods to remote methods such as assessment via telephone, video conferencing, or mobile applications [5,6,7,8,9]. Changes to data collection methods also required innovative analytic approaches for data processing due to the wealth of digital health data generated during the pandemic, as opposed to conventional methods [10]. Open science principles and experiences from other research projects indicate the need to be transparent about subsequent changes to methods of data collection, statistical analysis, and dissemination of findings in order to fully understand the impact of a pandemic (or other external events) on study results [11,12,13,14,15]. This presents a challenge for public health research studies to balance capturing the effect of external events, such as the bushfires and COVID-19 pandemic, whilst maintaining study design and measurement fidelity, particularly for longitudinal studies.

Although the literature on approaches to adapting fieldwork methodology during a pandemic is still emerging [5, 16,17,18,19], few reports have focused on dynamically capturing an event of public interest outside of the existing protocol or responding to multiple such events [16, 17, 20]. Most of the current research on these crises focuses on the impact on participants caused by the COVID-19 pandemic [3, 7], Australian bushfires [21], or both [22,23,24,25,26,27]; rather than the impact on study methodology and data collection caused by these events. For affected studies, many of the protocol changes were retrospective [22, 24, 28], whereby data collection was planned and modified following the significant event/s, rather than concurrent at the time of the event. This was the case for the National Social Life, Health, & Ageing (NSLHA) Project [29] where the fourth wave of data collection was modified from in-person to remote interviews. However, unlike the PATH study, the NSLHA study was not active in a wave of data collection as the COVID-19 pandemic began, and could therefore systematically plan for the upcoming protocol changes. Emerging data from the NSLHA study suggests that the mode of data collection (i.e. in-person versus remote) did not affect the study’s various physical and mental self-report data [30].

The Personality & Total Health (PATH) Through Life project was in a unique position to capture data from both the 2019 Australian bushfires and the COVID-19 pandemic as the project was in the process of data collection in the 2019-20 period. In addition, the fact that the PATH study was an established longitudinal study, meant it was particularly important to collect data in a timely fashion, in order to capitalise on the wealth of previous data and support ongoing engagement of participants. This called for immediate and significant changes to the study protocol while in an active wave of data collection. Few long-term studies have reported on how these methodological challenges were managed during these crises. Indeed, a review of methodological challenges for health research by Hensen et al. [31] stated that ‘Lessons learnt in designing and implementing remote data collection methods in a COVID-19 era are critical to inform future execution of these methods’. This case study aims to meet this need by describing the methods used to adapt the study to these emerging historical events, discussing the process and its implications for analysis, and providing directions for future research.

The PATH project commenced in 1999 and involves 7,485 young (aged 20–24 at baseline), midlife (aged 40–44 at baseline) and older (aged 60–64 at baseline) adults randomly sampled from the electoral roll of the Australian Capital Territory and the nearby city of Queanbeyan (cohort profile described in detail in Anstey et al. [32], and [33]). The present report relates to the 5th wave of data collection (2019/2020) for the midlife cohort (aged 60–64 at the time of data collection). A total of 2,147 active participants were approached for wave 5 and were invited to the follow-up. 1,558 participants took part, comprising 86.3% of the preceding wave’s sample size.

The PATH project aims to track and define the lifespan course of depression, anxiety, substance use and cognitive ability; identify environmental risk and protective factors within these domains; and examine the relationships between depression, anxiety and substance use with cognitive ability and dementia. Each cohort is followed up approximately every four years and invited to participate in structured interviews, comprising of surveys and cognitive and physical tasks.

Due to the significant impact of the Australian bushfires and the COVID-19 pandemic, this wave of data collection required a flexible approach including a full transition to remote telephone interviews and modifications to include questionnaires on concurrent external events. Wave 5 for the midlife cohort of the PATH Through Life study commenced in September 2019. Participants were invited to complete (1) an online self-complete questionnaire via Qualtrics and (2) to participate in a 1.5-hour face-to-face interview at their home (which comprised of physical and cognitive tests). From March 2020, face-to-face interviews were transitioned to telephone interviews due to COVID-19 restrictions (see Fig. 1 for overview of project timeline). Data collection for the wave was completed in May 2020.

Timeline of data collection for the Personality and Total Health (PATH) study against external events in Australia (from September 2019 to May 2020). Note. WHO: World Health Organisation, COVID-19: Coronavirus disease, F2F: face to face, ACT: Australian Capital Territory, NSW: New South Wales

Full size image

Bushfire questionnaire

Given the significance of the bushfires and unprecedented conditions in Australia over this period, the study was adapted to assess the impact of this natural disaster on PATH participants living in bushfire-affected areas. While the PATH study began in Canberra/Queanbeyan, many participants inevitably moved interstate since the study began in 1999, and many were living in other geographical areas across Australia that were affected by the 2019–2020 bushfires. Those still living in Canberra and surrounding regions (over two-thirds of the sample) were also significantly affected by heavy smoke, poor air quality and bushfires within the area and the surrounding regions (1).

To address the impact of the bushfires on participants, in January 2020 an additional subset of questions was added to the online self-administered survey completed by participants. These questions focused on participant exposure to the bushfires, and subsequent potential mental trauma. This bushfire questionnaire concluded with a qualitative question asking participants if they would like to add anything else about their bushfire experiences. For participants who had completed their online survey before the bushfires occurred, a separate link to the additional survey was emailed to them via Qualtrics (n = 621). For these participants, the generalised anxiety questions from the main questionnaire were also repeated, as responses to these questions are likely to be impacted by potential stress and anxiety surrounding the bushfires. Consequently, these participants represent a sub-sample for whom we obtained pre and post-exposure responses to the stress and anxiety measures within the same wave of data collection.

COVID-19 questionnaire

A 6-item scale measuring exposure to and impact of the COVID-19 pandemic was developed and added to the online survey (or sent as a separate questionnaire via Qualtrics, similar to the additional bushfire questionnaire). As a result of participant feedback, a qualitative response question was also subsequently added, to allow participants to provide a rationale for their responses, if they chose to do so. It is anticipated that the responses to the COVID-19 questionnaire will assist researchers with interpreting changes on other measures in the survey.

Transition from in-person assessments to remote assessments

The impact of the COVID-19 pandemic significantly affected the data collection methods of the wave. In early March 2020 when the pandemic began, face-to-face interviews were able to continue with minor protocol modifications including strict hygiene protocols, use of personal protective equipment (PPE; face masks and plastic gloves), an additional COVID-19 screening questionnaire prior to home visits, and the exclusion of high-risk physical tests (i.e. lung capacity tests). However, by late March 2020, more restrictive public health orders meant that face-to-face interviews were no longer allowed and the remaining face-to-face interviews for the wave had to be cancelled. At this point, approximately one-third of participants were yet to complete the face-to-face interview. Therefore, the decision was made to amend the testing protocol to include a modified remote telephone interview for the remaining participants (n = 460).

Adapted measures for remote assessments

Due to the transition to telephone interviews, all physical tests (e.g. blood pressure, hand grip, balance tests etc.) had to be excluded as these could not be administered over the telephone. Many of the cognitive tests used in PATH have a validated oral version that could be used for telephone administration instead. The advantage of this is it allows for investigations into the reliability and validity of the cognitive tests in regards to face-to-face versus telephone administration. These tests included the California Verbal Learning Test (CVLT) [34], Trail Making Test (TMT) [35], Digit Span Backwards [36], and Symbol Digit Modalities Test (SDMT [37]. The oral TMT has been validated against the written version (Part B [38] and there is evidence to support the validity of telephone administered CVLT, Digit Span Backwards [39, 40]. The testing battery also included the Spot the Word test (STW [41], however the validity for telephone administration for this test is not yet established.

The SDMT and STW tasks both required modifications and the use of electronic visual stimuli in order to be administered remotely. A third-party research services provider, ORIMA Research, was engaged to set up a customised online participant database and interview booking system whereby participants could login and complete the consent form, and book their face-to-face interview at a suitable time. The online system allowed live monitoring of bookings by the interviewers and project management team. As part of the transition to remote telephone interviews, ORIMA Research developed a custom-built secure webpage to display the visual stimuli to participants for the above cognitive tests. Participants were emailed a link to the password-protected webpage prior to the telephone interview, and were only given the password to the webpage by the interviewer at the time of the interview. This was done to prevent unsolicited prior practice of the tasks.

Project management implications

Project management for the wave required a dynamic approach due to the continuously-changing landscape of the bushfires and the global pandemic. Ethics modifications were required to be submitted to the university Human Research Ethics Committee for both additional sub-study questionnaires (bushfire and COVID-19) and the change to remote telephone interviews before commencing the new protocols. This meant that data collection for the wave had to be placed on hold while the new protocols were developed and reviewed and while the electronic stimuli platform was built. Once the team received approval for remote participation, the PATH project manager conducted a supplementary training course via Zoom to train the research interviewers in the new remote standard operating procedures. Training covered the revised booking process, administration of the telephone interview via Qualtrics, and how to share the electronic stimuli platform with participants at the time of interview. The changes to the study protocol had significant research funding implications as a result of the additional staff time spent amending ethics protocols, documenting new procedures, training research staff and building new resources (i.e., the online visual stimuli platform).

A total of 1,558 participants participated in this wave of the PATH study. 1,440 participants completed the online survey component (92% of the wave sample), this administration format remained consistent throughout the wave. 1,057 participants (68% of sample) completed face-to-face interviews prior to the switch to telephone administration as a result of COVID-19 restrictions; 460 participants (26% of sample) completed the modified telephone interview. For the small sample of participants who did not complete the online survey component, a greater number of participants completed the telephone interview (n = 97) rather than the face-to-face interview (n = 21). Participant completion rates for the additional bushfire and COVID-19 surveys were 60% and 69% respectively. Participant response to the inclusion of the sub-study questionnaires was overwhelmingly positive. Anecdotally, the team received a high level of participant-initiated contact expressing positive feedback and interest in the results of the bushfire and COVID-19 questionnaires.

An unanticipated consequence of the sub-study bushfire/COVID-19 questionnaires and remote participation was that this resulted in multiple data collection pathways for participants (see Fig. 2). For participants who completed their online survey prior to the bushfires and the pandemic, each supplementary questionnaire was emailed separately which therefore resulted in different dates of assessment for different components of the survey. Depending on the timing of the interview, participants either completed a face-to-face interview (pre-COVID-19) or a telephone interview, resulting in differences in the type of collected data. For example, participants who completed telephone interviews were not able to participate in physical assessments and completed a reduced and modified battery of cognitive assessments. The dynamic nature of COVID-19 infection rates and public health response led to bi-directional interactions between the time of completion of the surveys and their interpretation during data analysis. This poses additional challenges for data analysis due to the multiple pathways and dates of assessment for each group of participants.

Data collection flow chart with participant statistics for Wave 5 (2019/2020) of the Australian Personality and Total Health (PATH) study. Note. N: Number, w1-5: Wave number, F2F: face to face, COVID-19: Coronavirus disease

Full size image

We have described how we adapted protocols from a longitudinal study during the COVID-19 pandemic and Australian bushfires. Rapid adaptation was essential as these events disrupted an ongoing wave of data collection and we needed to maintain data collection and participant engagement. Since the time of data collection for this study, other researchers have considered how to best manage data collection, ethical considerations, and participant engagement through the pandemic [14, 41]. Our experiences address these more theoretical approaches, by showing how such changes can be enacted. We encountered many of the challenges discussed in these studies; these challenges along with the strengths and limitations of our methodological adaptations are discussed in more detail below.

It is anticipated that participant experiences of the bushfires and COVID-19 pandemic may vary greatly depending on the date their sub-study survey was completed and where they live. Community alerts and restrictions were continuously updated and revised in response to the bushfires and pandemic. The immediate impact of this is likely to be reflected in participants’ responses in relation to general physical health and mental well-being measures in the survey, as well as in relation to the specific sub-study questions. Therefore, the date of completion for the survey, bushfire questionnaire, COVID-19 questionnaire and face-to-face/telephone interview will become important variables in analysis. Categorical variables will help to determine whether the main survey and/or interview were completed before, during or after the bushfires, and before or during the pandemic (as recommended by Stiles-Shields [12]). It is worth noting that data collection for this wave only covered the initial few months of the pandemic in its early phase. The next wave for the older PATH cohort (aged 60–64 at initial recruitment) was scheduled to follow this wave and spanned a greater period of the pandemic (November 2020 to July 2021). We were able to apply the lessons learnt from this case study to this wave of data collection as this was all completed via telephone interviews. This will provide additional data on the validity of the telephone interviews and assessments.

The nature of PATH being a longitudinal study affected the decision-making for data collection and the immediacy of required modifications. The majority of measures used in PATH are repeat-measures which are used across waves to allow investigation of changes over time. Given this, the PATH research team was constrained when it came to modifying the telephone interviews and the sub-study questionnaires. For example, PATH has previously collected data relating to the 2003 Canberra bushfires [31, 42]; therefore, the decision was made to repeat the same bushfire exposure measures and post-traumatic stress scale in order to allow comparison with previous data (despite the fact that new measures have since been developed; e.g [43]). Similarly, the decision was made to modify previously used cognitive tests rather than use new cognitive batteries that have been developed specifically for telephone administration. Future papers will aim to investigate the reliability and validity of these specific cognitive tests when comparing face-to-face versus telephone administration versions from this study.

The positive response from participants regarding the addition of the bushfire and COVID-19 sub-study questionnaires is likely due to the high-perceived relevance at the time of data collection. This highlights the value of continuing and extending research in periods of chaos, particularly in longitudinal studies such as PATH with an established and engaged sample. Likewise, participants valued the inclusion of qualitative questions (for both the bushfire and COVID-19 questionnaires) in order to provide further information about their responses and personal experiences. From preliminary analysis, this data is incredibly valuable in terms of investigating diverse participant impact [21].

An unintended benefit of transitioning to remote interviews was the increased participation of out-of-area participants in this wave. In previous waves, this has often been achieved by having interviewers travel to other capital cities of Australia. However regional and rural participants are often excluded due to budgeting constraints. Telephone interviews allowed out-of-area participants (comprising approximately one-third of the wave) to be easily sampled and contributed to the relatively high response rate for the wave. Similarly, it resulted in greater efficiency for research interviewers as they were able to complete participant interviews from their homes, rather than travelling around the ACT and surrounding areas to complete home visits. An additional benefit is that telephone interviews may be less burdensome for disengaged participants; a higher number of participants who did not complete the online survey (suggesting they were less engaged) completed a phone interview, compared to pre-lockdown numbers for face-to-face interviews.

PATH is in a unique position due to the wealth of longitudinal data from previous waves that allow for valuable comparisons between baseline and present. This is especially true in relation to changes in mental health and well-being as a result of external events such as the Australian bushfires and COVID-19 pandemic. In addition to the longitudinal data, some measures were also repeated within this wave, before and after potentially traumatic events (e.g. the anxiety scales completed both before and after the bushfire for a sub-sample). For participants who completed both, this provides valuable short-term data on the immediate mental health impact during and after the bushfires. Collecting this data is also likely to enhance the value of the PATH dataset as a source of evidence for contemporary public health issues.

Telephone interviews come with their own challenges and considerations. The interviewer lacks important contextual information from the participant such as non-verbal cues (potentially signalling boredom, fatigue, inattention etc.) and it can be difficult to assess how the participant is managing the interview. Furthermore, hearing impairments and comprehension difficulties can create further challenges. The lack of visual feedback for the interviewer is particularly disadvantageous for the administration of cognitive tests, such as the CVLT. In this memory task, participants are read out a list of 10 words and required to recall as many words as possible in an immediate and delayed recall task. The interviewer provides specific instructions to the participant, including that pens and paper are not to be used. However, it is impossible to monitor adherence to such instructions over the telephone. PATH interviewers are highly experienced and flagged a note if it was suspected the participant may have been using written aids so that this could be considered during analysis. In hindsight, interviews over a video conferencing platform would have been better suited for cognitive tests to ensure standardisation. However, this decision-making occurred relatively early on in the pandemic and video conferencing was at the time not as widely used as it is today. Due to time constraints and ease of administration, the decision was made to progress with telephone interviews.

A further limitation of the transition to remote data collection is that the physical assessment data (e.g. blood pressure, hand grip, balance tests etc.) was not able to be collected this wave for select participants (approximately 30% of the sample). The logistics, cost, and time constraints of this study meant that remote physical testing was not possible, however it would be of great benefit to consider alternative options for this component for future studies such as potential wearable technologies (e.g. smart watches) [10].

Lessons learned during this period of data collection demonstrate the value of a dynamic framework of data collection during periods of significant volatility, in order to balance capturing the impact of external events, whilst preserving the aims of the study. The dynamic framework of data collection utilised in the PATH project has shaped the way our research team has managed other studies during the continued period of uncertainty in 2021. As a result of ongoing lockdowns in Australia in mid- to late-2021, the majority of research studies and clinical trials were also placed on hold due to ongoing restrictions. As these studies recommenced, lessons learned during data collection for the PATH project were applied to these studies as a means of building resilience in the face of possible future emergencies. Other studies also adopted dynamic approaches to study protocols that can be easily adapted based on current restrictions in relation to PPE requirements, requirements of COVID-19 vaccination status, and methods of delivery (whereby studies have sought approval for both face-to-face and video conferencing modes). Through anticipation of the challenges that are likely to arise based on ongoing external events, ethics approval and procedures for modifications can be sought in advance creating greater flexibility and efficiency for project administration.

The shift to remote methods of data collection is likely to be utilised in future research due to the advantages of increased recruitment and efficiency. From our experience in the PATH project, remote assessments allowed greater participant involvement due to the lack of geographical barriers. Furthermore, research staff experienced greater efficiency with telephone interviews due to removing the need to travel and therefore also reducing travel costs. While it is clear there are many advantages to remote assessments, it is important to acknowledge this may also come with novel barriers, especially in the field of dementia research. Whilst this was not a significant issue for the midlife cohort of the PATH project, this is relevant to the older PATH cohort. Other studies have suggested that these methods may inadvertently exclude participants due to technological and access barriers [6, 44] which may have further implications for sampling and representation. This is an important consideration to address in future studies, even more so when collecting data on the pandemic, to ensure sufficient representation in the sample of the experiences of older adults with cognitive impairment, or those living in care homes, for example [45]. Specific strategies for minimising barriers to participation for these individuals will need to be put in place such as collaboration with aged care staff who can assist with the logistics and technology of remote participation. Advancements in video conferencing platforms may also help to minimise barriers for those with hearing impairments or non-native English speakers (or other participants who benefit from written information) due to the introduction of closed captioning software on these platforms. In line with a dynamic framework to data collection, access barriers for the specific population of interest should be considered prior to commencing, in an effort to promote inclusivity for all research participants.

COVID-19:

Coronavirus disease 2019

PATH:

Personality and total health through life study

ACT:

Australian Capital Territory

NSW:

New South Wales

PPE:

Personal Protective Equipment

CVLT:

California Verbal Learning Test

TMT:

Trail Making Test

SDMT:

Symbol Digit Modalities Test

STW:

Spot the Word Test

Download references

KJA, TL, RE, MEM, CS and IG were involved in planning and set-up of the study follow-up. TL project managed the follow-up and drafted the manuscript; IG was data manager. All authors reviewed the draft versions of this manuscript and read and approved the final version.

Correspondence to Kaarin J. Anstey.

The Wave 5 follow-up of the PATH 40s cohort was conducted with approval from the UNSW HREC (Protocols: HC180505; HC180518). All participants provided informed written consent to take part in the study.

Not applicable.

The authors declare no competing interests.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.

Reprints and permissions