"Never doubt that a small group of thoughtful, committed citizens can change the world. Indeed, it is the only thing that ever has."

Margaret Mead

Review article
peer-reviewed

Emerging Needs and Viability of Telepsychiatry During and Post COVID-19 Era: A Literature Review



Abstract

The coronavirus disease 2019 (COVID-19) pandemic has resulted in nationwide stay-at-home orders in an effort to slow the spread severely impacting the healthcare sector. Telepsychiatry provides a platform bridging the gap through advanced technologies connecting mental health providers and patients who need their services, overcoming previous barriers of great distances, lack of transportation, and even time constraints. The most obvious benefit is increased accessibility to mental healthcare, especially in underserved and remote areas where there is no easy access for in-person care. It is important to note that benefits are not limited to patients, but also allow clinicians greater flexibility in scheduling and reduced practice overhead costs, both of which aid with physician burnout and burden. Telepsychiatry during COVID-19 provides its own unique advantages over in-person visits. The risk of exposure to healthcare workers and patients receiving care is reduced, allowing immunocompromised patients to receive much-needed psychiatric care. Without the need to meet in person, self-isolating psychiatrists can still provide care, decreasing strain on their co-workers. Although telepsychiatry is relatively new, it has already exhibited considerable success in its effectiveness at treating psychiatric conditions and widespread corollary benefits. Telepsychiatric consults may be carried out synchronously and asynchronously, each having benefits and setbacks. Different mobile application interventions have been explored, which are available for the purpose of both monitoring/assessing patients and/or providing treatment. The scope of conditions these applications address is broad, from anxiety disorders to schizophrenia to depression. As promising and beneficial telepsychiatry may seem, it is necessary to recognize that building the program can be challenging. It involves adapting to new methods in medicine. We highlighted barriers to general telepsychiatry, the most prominent being technological literacy of both physician and patient, and possible negative effects of eliminating the in-person patient-doctor interaction.

Introduction & Background

The recent coronavirus disease 2019 (COVID-19) pandemic has resulted in nationwide stay-at-home orders in an effort to slow the spread of the disease severely impacting many industries, including the healthcare sector [1]. As hospitals and healthcare providers pause non-emergency services and elective procedures, many patients are unable to receive care. In fact, 45% of U.S. adults reported they or someone in their household had deferred medical care due to the coronavirus pandemic [2]. Although the lack of these services may seem insignificant given their non-emergent status, they can still prove detrimental to an individual’s well-being, especially when it comes to mental health. With patients being unable to see their psychiatrists for in-person sessions, it is imperative that alternative methods be explored in an attempt to bypass this obstacle.

As the prevalence of mental health disorders continues to increase among young adults, there is an imminent need for new and innovative ways to meet the populations' demands of effective delivery of psychiatric care [3]. In addition to the challenges introduced by COVID-19, there is an existing disparity regarding access to healthcare; 3.6 million Americans are unable to receive medical care due to a lack of transportation [4]. Telepsychiatry offers a prospective solution to all the aforementioned problems, promising a delivery method that is not only much easier to access but can adapt to match the demand whilst abiding by social distancing policies in place [5].

Telepsychiatry provides a virtual platform for the matching and communication between mental health providers and the patients who need their services, overcoming previous barriers of great distances, lack of transportation, and even time constraints. Many mental health conditions require outpatient follow-up and medical management. Given that, telepsychiatry has the potential to increase access to care while improving waiting times and reducing costs and the need for hospitalization [6]

This review examines the methods and limitations of telepsychiatry that are currently available and provides valuable insight into the outcomes of populations that are currently being implemented in telepsychiatry. It is important to note that telepsychiatry is not a cure-all solution. Despite the encouraging benefits, there are many questions to be considered, such as: Will the patient feel as connected with the provider as in-person visits? Can the same trust and emotional bond be forged over a virtual session? Psychiatry is different from other medical specialities where treatment is often a prescription or a one-time procedure. Ninety percent of the diagnosis and fifty percent of treatment involve direct communication between patient and psychiatrist, so it is imperative to address how telepsychiatry may modify or change the physician-patient relationship.

Review

With the COVID-19 pandemic came a dramatic shift of the U.S. healthcare system as traditional medicine transitioned from in-person visits to telemedicine instead. Without a vaccine or standardized effective treatment options, social distancing procedures and stay-at-home orders have become the primary forms of intervention. Given that, clinicians, healthcare workers, and insurers alike have been forced to adapt rapidly in order to provide an alternative to in-person care, turning to telemedicine. This surge in telemedicine has largely been focused on triage screening patients, much of which has been related to physical symptoms pertaining to COVID-19 [7,8]. What has been neglected amidst this pandemic, though, is the delivery and availability of mental healthcare. Whether it be the detrimental effects of social isolating, the panic that comes from losing one’s job in a period of economic instability, or simply the stress that comes from living during such uncertain times, it is clear that mental health is more important than ever. During this viral outbreak, there has been a huge surge in symptoms of depression, anxiety, insomnia, and acute stress, whether it be healthcare providers or the general public [9-11]. As the prevalence of mental health disorders continues to increase at a startling rate, it is pertinent to turn to telepsychiatry as a means of increasing access to mental healthcare.

Before diving into the benefits, limitations, and methods of implementation of telepsychiatry, it is important to gain a thorough understanding and definition of telepsychiatry (Table 1). Specifically, what makes it different from telemedicine? Telemedicine is described as the remote assessment, diagnosis, and treatment of patients through the use of telecommunication audiovisual technology. This includes, but is not limited to, consultations over the phone, video calls or text messaging, digital transmission of medical imaging, and remote diagnoses. Telepsychiatry is a specific application of telemedicine within the speciality of psychiatry and involves making psychiatric diagnoses and administering psychiatric treatment or therapy through technological means. The history of telemedicine is long and extensive, with the first use of telemedicine being described in The Lancet in 1879 as the transmission of medical knowledge and conducting medical appointments through the telephone [12]. However, telepsychiatry was not documented until nearly a century later, in the 1950s, when the Nebraska Psychiatric Institute utilized videoconferencing for consultations, educational endeavours, and training for patients [13]

Telemedicine Telepsychiatry
Remote assessment, diagnosis, and treatment of patients through the use of telecommunication audiovisual technology.  Application of telemedicine within the specialty of psychiatry. 
Use of telemedicine was first described in The Lancet in 1879. Related terms are telemental health and telepsychology. 
  Administration of psychiatric treatment or therapy via technological communication. 
  First use of videoconferencing in psychiatry was in the 1950s at the Nebraska Psychiatric Institute. 

Although telepsychiatry is a newer branch of telemedicine, the potential benefits of implementation are extensive and far-reaching (Table 2). The increased accessibility to mental healthcare especially for those in underserved and remote areas seems to benefit maximally; additionally, transportation times and expenses are reduced along with base patient costs. These benefits result in lower attrition rates of patients, along with increased patient satisfaction with care [14-16]. It is important to note that benefits are not limited to patients, but also allow clinicians greater flexibility of scheduling and reduces practice overhead costs, both of which can aid with physician burnout and burden. Telepsychiatry during COVID-19 provides its own unique advantages over in-person visits. The risk of exposure to both healthcare workers and patients receiving care is greatly reduced, allowing immunocompromised patients to receive much-needed psychiatric care. Without the need to meet in person, self-isolating psychiatrists can still provide care, decreasing strain on their co-workers. Patients who have tested positive for coronavirus will still be able to meet with their psychiatrists.

Telepsychiatry Implementation and Potential Barriers and Benefits [14-16]
Barriers     Benefits    
Adapting to novel practice model and technology Accessibility to specialized mental health care in underserved and remote areas
Interruption of the in-person patient-doctor interaction Expand outreach, delivery, and efficiency of mental health practice
Liabilities and malpractice coverage concerns Reduction in transportation time and expenses
Technology costs and security Low attrition rate of patients 
Billing and reimbursement constraints Relieve physician burnout and increase well-being
Out-of-state physician licensing and credentialing issues Improved patient satisfaction
Obtaining vital signs and necessary physical examinations, although the physical examination is less of a barrier in telepsychiatry compared to traditional telemedicine except for the evaluation of movement disorder or extrapyramidal side effects (EPS) Improve clinical workflow and increase practice efficiency 
Concerns about patient confidentiality and security  Reduce practice overhead
Need for psychiatrists as well as patients to have a basic level of technological literacy; both parties will need to adapt to the technology  Reduce patient costs 
Patients with moderate to severe mental health illness might be reluctant to discuss their problems over teleconsultation services which can be a huge barrier. Reduce emergency room visits
  Early detection allowing for preventative measures 
  Flexibility of scheduling 
  During COVID-19: 
  Minimize risk of exposure to healthcare workers, as well as patients receiving care
  Immunocompromised patients will still be able to receive psychiatric care 
  Self-isolating psychiatrists can still provide care; prevents increasing strain on remaining co-workers 

Within the branch of telepsychiatry, one can dissociate between synchronous and asynchronous telepsychiatry, each having its own benefits and setbacks. Synchronous telepsychiatry requires the presence of both the patient and the psychiatrist at the same time, engaging in real-time communication. Asynchronous telepsychiatry, on the other hand, can be thought of as the store-and-forward transmission of medical data, as well as clinical diagnosis, treatment, and recommendations [17,18]. For an extensive comparison of the relative benefits and setbacks, refer to Table 3. Given the varying advantages, we recommend that the use of either method be situational; it is a decision that should be made largely depending on the patient-provider relationship and what would optimize individual patient care. 

Synchronous Telepsychiatry [17-24] Asynchronous Telepsychiatry [25-28]
Consists of live videoconferencing and direct communication link between the psychiatrist and patient. Consists of pre-recorded videos of psychiatrist that are then sent to the patient to be viewed on their own time.
Allows for real-time live interaction and assessment. Involves acquiring medical data and transmitting it to the psychiatrist at a convenient time for assessment offline.
Can gain additional details essential for care during the session, which is more efficient in making a clinical decision and providing advice during the interaction. Patients can get timely care without needing to travel beyond the location of their primary care providers.
Real-time telepsychiatry provides timely care, especially in emergencies or urgent situations, and maintains the concept of the doctor-patient relationship by enabling a face-to-face analysis and treatment. Wait times for specialty care are lessened, especially in areas with shortages of psychiatrists.
The doctor can see the patient and engage in conversation, much as they would in an office visit. The store-and-forward process can overcome language and cultural barriers.
  Patients who are traveling or live in a different timezone can receive care without inconvenience. 
  The turnaround time to receive answers to patient concerns/questions may be longer than synchronous telepsychiatry. 

Although telepsychiatry is relatively new, it has already exhibited considerable success in both its effectiveness at treating psychiatric conditions as well as widespread corollary benefits. Although there are a few cases in which no significant improvement was seen between in-person psychiatry and telepsychiatry (five out of the 16 studies listed in Table 4), peripheral consequences such as decreased cost, better patient-doctor interactions, and higher satisfaction make telepsychiatry a viable, if not preferable choice to traditional in-person medicine. This, of course, only holds true for visits and disorders that do not necessitate physical examinations, which would be much more difficult to conduct virtually. Table 4 contains a comprehensive list of studies examining the effectiveness of telepsychiatry across multiple conditions, from depression and anxiety to dementia and post-traumatic illness, among others. 

Author, Country, Year Type of Study Psychiatric condition studied Sample Size Outcomes Measured Results Interpretation of study
Yellowlees et al, USA 2018 [27] RCT (Asynchronous or ATP vs Synchronous or STP Telepsychiatry) Depression, anxiety and substance abuse ATP=77, STP= 81   1.Clinical outcome using Clinical global impressions (CGI) 2. Satisfaction 3. economic data 1.Significant improvement in CGI from baseline 2.Less average wait times (all for ATP) 3.More efficient use of psychiatrist time 4.Good teamwork along with PCP 1.Improved patient satisfaction 2.Relieve Physician burnout and well-being 3.Decrease time to essential mental health care
Farabee et al, USA, 2016 [29] RCT (TP vs TAU)     N= 104   1.Satisfaction with treatment 2. Therapeutic alliance 3. Medication adherences 4. Psychological functioning. 1.High satisfaction with telepsychiatry 2.No significant group differences in medication adherence or psychological functioning. 3.Lower levels of the therapeutic alliance at follow-up. 1.Improved satisfaction with telepsychiatry
Xiong et al, USA 2018 [30] RCT (STP vs ATP)   Dementia, depression Total = 43, ATP=22, STP=21   Clinical global impressions (CGI) severity at 6month and 12 month Significant improvement in CGI from baseline to 6-month follow-up, regardless of group assignment, Demonstrating the acceptability, feasibility, and impact of both forms of telepsychiatry in the Small nursing facility (SNF) setting. 1.Telepsychiatry is effective 2.Improved satisfaction rates with telepsychiatry
Shulman et al, USA, 2017 [31] RCT (TP vs TAU)     N=22   Visit adherence Not much difference in visit adherence between the telepsychiatry and treatment-as-usual groups (14%, compared with 15%). A greater number of participants in the telepsychiatry group reported less subjective difficulty in keeping appointments. 1.Increased compliance with telepsychiatry
O’reilly et al, Canada, 2007 [32] RCT (TP vs face to face)     254 (face to face) 241(TP)   Distress from psychiatric symptoms using the Brief Symptom Inventory. Psychiatric consultation and follow-up delivered by telepsychiatry produced clinical outcomes that were equivalent to those achieved when the service was provided face to face Not much difference observed between telepsychiatry and in-person visits.
Zou et al, USA, 2016 [33] RCT (T-CSCT vs TAU) MDD (N = 190), T-CSCT (n = 97), TAU (n = 93) 17-item Hamilton Depression Rating Scale (HDRS₁₇); the positive response was defined as a ≥ 50% decrease in HDRS₁₇ score 1. Response and remission were significantly greater for the T-CSCT group compared to the control group (odds ratio [OR] = 3.9 [95% CI, 1.9 to 7.8] and 4.4 [95% CI, 1.9 to 9.9], respectively). 2.T-CSCT group had significantly greater improvement over time in HDRS₁₇ (F4,95 = 4.59, P = .002), CGI-S (F4,95 = 4.22, P = .003), and CGI-I (F4,95 = 2.95, P = .02) scores. Improved response rates with telepsychiatry
Chong et al, USA, 2012 [34] RCT (WEB vs TAU) Depression N=167, WEB=80, TAU= 87 1.Acceptability 2.visit satisfaction 3. Antidepressant use. 4.Feasibility 1.WEB patients did not differ in the proportion of completed primary care versus telepsychiatry appointments and rated their working alliance with the psychiatrist and their visit satisfaction significantly higher than the TAU patients with their provider. Significantly more WEB than TAU patients used antidepressants. 1. Improved acceptability, feasibility, and satisfaction rates with telepsychiatry
De Las Cuevas et al, Spain, 2006 [35] RCT (VCTP vs FTF)   N= 140 Clinical Global Impressions-Severity of Illness (CGI-S) and -Improvement (CGI-I) scales 1.Significant improvements were found on the CGI and SCL-90- R Global Indexes scores of both treatment groups, showing clear clinical state improvement. 2. statistically significant differences were observed when the efficacy of VCTP treatment was compared to FTF psychiatric treatment efficacy Higher response rates and satisfaction rates in the telepsychiatry group.
Frueh et al, USA, 2007 [36] RCT (TP vs SR) Post-Traumatic Stress disorder N=38, 21-SR, 17-TP 12-item SER-TACP (Therapist adherence and competence of treatment) was used to assess adherence and competence. Findings suggest that therapist competence and adherence to cognitive-behavioural therapy is similar whether the treatment is delivered via TP or by traditional means, and TP does not compromise therapists’ ability to effectively structure sessions or build rapport with patients 1.Treatment Adherence and competence is similar in telepsychiatry vs in-person visits. 2. Improved patient-doctor interactions.
Ruskin et al, USA, 2004 [37] RCT (TP vs in person) Depression N=118 1.Treatment response. 2. Treatment adherence. 3. Patient satisfaction.  4. Psychiatrist satisfaction. 5. Resource consumption or cost effects. 1. Hamilton Depression Rating Scale and Beck Depression Inventory scores improved over the treatment period and did not differ between treatment groups. 2. The two groups were equally adherent to appointments and medication treatment. No between-group differences in dropout rates or patients’ ratings of satisfaction with treatment were found. 3. Telepsychiatry was more expensive per treatment session, but this difference disappeared if the costs of psychiatrists’ travel to remote clinics more than 22 miles away from the medical center were considered. 1. Not much significant between treatment and in person groups. 2. Cost effective for clinics far away from the medical centers.
Moreno et al, USA, 2012 [38] RCT (WEB vs TAU) Depression N=167 1. Montgomery-Åsberg Depression Rating Scale by clinician’s blind to treatment group and 2. Self-ratings on the nine-item Patient Health Questionnaire. 3. Quality of Life Enjoyment and Satisfaction Questionnaire. 4. Sheehan Disability Scale. 1. On all four measures, a significant interaction of time by intervention favoring the Webcam group was noted. 2. Results suggest that telepsychiatry delivered through the internet utilizing commercially available domestic Webcams and standard internet and computer equipment is effective and acceptable Telepsychiatry was effective and acceptable.
Bishop et al, Canada, 2002 [39] RCT (VCTP vs FTF)   N=24 Patient satisfaction with the services was assessed using the Client Satisfaction Questionnaire (CSQ-8), completed four months after the initial consultation The mean scores were 25.3 in the FTF group and 21.6 in the videoconferencing group. Although there was a trend in favor of the FTF service, the difference was not significant Not much significance in the telepsychiatry vs face to face visits groups.
Hulsbosch et al, Netherlands, 2017 [40] RCT (VCF vs care as usual) Severe mental illness N=93, VCF 47, care as usual (CAU)= 46 Patient satisfaction was the primary outcome of the study. The ‘GGZ Thermometer’, a validated Dutch instrument for assessing patient satisfaction within the mental health care setting,11 was used A statistically significant time by treatment interaction effect was found, where higher degree of satisfaction was associated with the patients in the VCF-group. Improves patient satisfaction in the telepsychiatry group.
Maieritsch et al, USA, 2016 [41] RCT (VCTP vs in person) Post-Traumatic stress disorder (PTSD) N=90 Clinician-administered PTSD scale (CAPS) scores A trend was observed which suggested that Cognitive processing therapy (CPT) over VTC may be equivalent to the treatment delivered in person Not much significance in the telepsychiatry vs in person group.
Myers et al, USA, 2015 [42] RCT (VCF vs in-person) Attention deficit hyperactivity disorder N=223 Outcomes were diagnostic criteria for ADHD and ODD and role performance on the VADRS completed by caregivers (VADRS-Caregivers) and teachers (VADRS-Teachers) and impairment on the Columbia Impairment Scale-Parent Version (CIS-P) Children assigned to the telehealth service model improved significantly more than children in the augmented primary care arm for VADRS-Caregiver criteria for inattention (χ(2)[4] = 19.47, p < .001), hyperactivity (χ(2)[4] = 11.91, p = .02), combined ADHD (χ(2)[4] = 14.90, p = .005), ODD (χ(2)[4] = 10.05, p = .04), and VADRS-Caregiver role performance (χ(2) [4] = 12.40, p = .01) and CIS-P impairment (χ(2)[4] = 20.52, p < .001).  Improved response rates in telepsychiatry group.
Hur et al, Republic of Korea, 2017 [43] RCT - scenario-based CBT mobile app group, the Todac Todac group (TT group) and an app-based mood charting group (control group) Depressive disorder N=34 Outcomes measured using Dysfunctional attitude score (DAS), Beck’s depression inventory Scores(BDI-II) and Situation dependent trait version of state trait Anxiety inventory  (STAI- X2) 1.After 3 weeks TT group showed lower DAS scores and significantly lower STAI-X2 scores compared to the control group. 2.Both groups showed reduced BDI-II scores. Mobile based apps are helpful in delivering effective and feasible cognitive therapy.
 

Up until this point, we have discussed telepsychiatry in the context of virtual appointments and interactions that presumably take over phone or video calls, but with the advent of smartphones, mobile applications (apps) have become an area of interest. In Table 5, though, we have explored different mobile application interventions that are available for the sole purpose of both monitoring/assessing patients and/or providing treatment. The scope of conditions that these apps address is broad, ranging from anxiety disorders to schizophrenia to depression, etc. All studies reported high user satisfaction and a few even saw significant improvement in symptom reduction [44-51]. These findings, although preliminary, suggest a promising future for mobile applications that patients can use on their own time either independently or in conjunction with psychiatrist appointments and monitoring. 

Author, Country, Year Tools Study Design  Psychiatric Condition Studied Description Usefulness of Tool
De Sa & Carrico, USA, 2012 [44] Mobile App Device: PDA  Description of app design process and the results of a qualitative exploratory trial study, utilizing direct observation, interviews, and questionnaires. Participants underwent 1 session.   Children with anxiety disorders  The existence of therapeutic treatment tools for young patients is low. Paper presents mobile apps that aim to help children (5-14 years of age) better quantify their fears through the use of drawing and components that improve patient intractability and understanding. The app was also designed to better support therapy procedures. Out of 8 children interviewed: 3 were pleased with the experience, 5 found it easy to use, 7 found it easier than the paper equivalent.    Both patients and therapists were satisfied and excited about the app. 
Iiboshi et al., Japan, 2020 [45]  VTC-administered Montreal Cognitive Assessment Tool (MoCA)  Participants older than 60 years of age with and without cognitive impairment (mild cognitive impairment [MCI], dementia, and healthy controls [HC]) assessed using VTC and face-to-face (FTF)-administered MoCA  at an interval of >2 weeks and <3 months.  Mild cognitive impairment (MCI) and dementia  Despite the various potential benefits of utilizing VTC-administered MoCA to detect and diagnose cognitive decline, not many studies have examined the reliability of such a tool, which was the aim of this study.  Intraclass correlation coefficient (ICC) for the MoCA was high in the entire sample overall, and moderate to high for the subgroups, indicating good agreement between VTC- and FTF-administration.    Participants also indicated overall satisfaction with VTC-administered MoCA. 
Graham et al.,  USA, 2020 [46]  IntelliCare: a suite of mobile apps providing a library of psychoeducation, and administering weekly symptom assessment  A two-arm randomized clinical trial with 146 patients, evaluating the efficacy of a mobile platform in providing treatment for depression and anxiety Depression (positive on Patient Health Questionnaire-8; score>=10) and anxiety (Generalized Anxiety Disorder-7; scores >= 8)  Participants in intervention conditions began IntelliCare through onboarding calls with their coach. Coaches managed participants using an online dashboard, containing info on patient app use and their symptom assessments.    Primary outcomes were assessed with PHQ-9 and GAD-7 at baseline and the follow-up assessments.  Individuals in the intervention condition had a greater reduction in depression and anxiety symptoms compared with treatment-as-usual participants. These changes were also sustained over 2 months.   
Donker et al.,  Netherlands, 2019 [47] ZeroPhobia, a fully self-guided virtual reality equipment for CBT Single-blind randomized clinical trial with 193 participants with acrophobia symptoms to examine the effectiveness of ZeroPhobia  Acrophobia (score of at least 45.45 on Acrophobia Questionnaire (AQ)-Anxiety)   Participants received 6 animated CBT-based modules that took between 5-40 minutes and were asked to complete it within 3 weeks. The primary outcome was AQ scores.  Intervention through ZeroPhobia showed a significant reduction in acrophobia symptoms compared with control on AQ scores.    The VR-CBT app was also rated as user friendly, showing the ease of using technology-based interventions. 
Espie et al.,  United Kingdom, 2019 [48]  dCBT for insomnia  An online, 2-arm, parallel-group randomized trial comparing dCBT for insomnia with sleep hygiene education (SHE) evaluated 1711 participants with self-reported insomnia symptoms  Insomnia  dCBT was delivered to participants using the web and/or mobile channels along with treatment as usual. SHE was a website and a downloadable booklet along with treatment as usual. Primary outcomes were scores on the Patient-Reported Outcomes Measurement Information System: Global Health Scale, Warwick-Edinburgh Mental Well-being Scale, and Glasgow Sleep Impact Index. The use of dCBT was associated with a small improvement in functional health, and psychological well-being, and a large improvement in sleep-related quality of life. A large improvement in insomnia was the mediator of these outcomes.    Thus, we see that dCBT is effective in reducing insomnia symptoms, which mediates the aforementioned effects. 
Ainsworth et al.,  United Kingdom, 2013 [49] Android app “ClinTouch” for monitoring schizophrenic patients   Randomized control trial with 24 participants comparing two monitoring systems for schizophrenic patients — one that is app-based and one that utilizes is short messaging service-based.  Schizophrenia (DSM-4)   Schizophrenic participants were randomly allocated to completing 6 days of assessment, involving four sets of questions a day, using the smartphone app or SMS text-only implementation. A 1-week break was given before the alternative method was used for 6 more days. Feedback was obtained at the end of each period of sampling.  A significantly larger proportion of data was completed through the smartphone application compared to the short-messaging implementation. Most participants also reported a preference for the app, citing its ease of use.
Macias et al., USA, 2015 [50] iPhone and Android app “WellWave” designed to facilitate self-directed healthy living and psychiatric recovery. Pilot study that tested the usability and acceptability of WellWave for adults diagnosed with psychiatric conditions, with the focus on walking as a physical exercise.  A wide variety of psychiatric disorders (schizophrenia, bipolar disorder, or MDD)  Eleven volunteers, each with at least one of the mentioned psychiatric (4 schizophrenics) completed an individual WellWave training session before completing a 4-week pilot. 9 participants provided pretest and posttest questionnaire data.  Participants used the app an average of 94% of the days and 70% of participants went on a walk at least 2x a week.  All participants reported being satisfied with the app, reporting feelings of well-being as well as practice benefits. 
Ben-Zeev et al., USA, 2014 [51] Android app “FOCUS” designed for real-time illness management of schizophrenic patients Single-arm feasibility trial examining the efficacy and acceptability of real-time/real-place illness management to support schizophrenic patients  Schizophrenia or schizoaffective disorder  33 individuals w/schizophrenia or schizoaffective disorders used the FOCUS app for 1 month, with 32 users completing the trial fully.  Completers used FOCUS on 86.5 % of days for an average of 5.2x a day.    90% of participants rated the app as highly acceptable and usable.    Study demonstrated the feasibility, acceptability, and preliminary efficacy of the FOCUS intervention. 

Unfortunately, there is very limited literature on the current implementation of telepsychiatry during the coronavirus pandemic (Table 6). This is not a reflection of the inefficacy of telepsychiatry, though, but rather a reflection of its underutilization. This is a field of untapped potential that needs to be further explored and expanded upon, especially with the current climate of the pandemic. As promising and beneficial telepsychiatry may seem, though, it is necessary to recognize that no telemedicine program can be created overnight. There are challenges involved in adapting to this new method of medicine. Table 2 above highlights barriers to general telepsychiatry-the most prominent ones being the need for technological literacy on both the physician and patient end, as well the possible negative effects of eliminating the in-person patient-doctor interaction that is often used to build rapport. 

Higher utilization rate of Telepsychiatry  
Author, Country, Year Type of Study Psychiatric condition studied Sample Size Outcomes Measured Results Interpretation of study
Zarghami et al., Iran, 2020 [52] Report in which video chat was used Insomnia seen in 24 (29.3%) patients and adjustment disorder in 13 (15.9%) patients Total(82) 32 inpatient, 50 outpatient, 32 male and 50 female Patient Health Questionnaire-9 (PHQ-9), Generalized Anxiety Disorder Assessment (GAD-7), and Perceived Stress Scale-14 (PSS 14) questionnaires Female and hospitalized patients presented significantly more frequent comorbidities than males and outpatients Telepsychiatry in the early stages of mental problems during a catastrophic event like the coronavirus pandemic, can be an efficient instrument for the screening of psychosomatic comorbidities, so that pharmacological treatment (considering possible drug interactions with COVID-19 medications) and psychotherapeutic intervention can be optimized by psychiatrists.
Chen et al, USA, 2020 [53]  A review of key changes that were implemented during COVID-19 with regards to psychiatric care, as well as observations of advantages and limitations to telepsychiatry N/A N/A N/A   Biggest advantage of telepsychiatry during COVID-19 pandemic is limiting viral transmission and reducing risk of exposure. Clinical benefits include decreased no-show rates, increased ease of scheduling and access to care, among other advantages.    The largest limitations included the lack of a physical examination and mental status markers, loss of intimacy, and exacerbated disparities for those who lack access to technology. 
Johns Hopkins Medicine, 2020 [54] Description of transitions to telepsychiatry at Johns Hopkins Hospital during the COVID-19 pandemic  N/A N/A N/A Patients with opioid use disorder have been able to benefit from at-home video consultation and management through telepsychiatry.  The efficacy and benefits of telepsychiatry during the coronavirus pandemic have been preliminarily demonstrated at a large scale hospital (i.e., Johns Hopkins), implicating the clinical benefits of transitioning to telepsychiatry. 

Specific to COVID-19, structured interviews with clinicians revealed problems that they ran into when transitioning to telepsychiatry (Table 7). These included decreased clinical data available to make an assessment or diagnosis and technological challenges (lack of reliable access to a computer, smartphone, or the internet) among other issues.

Author/Year Type of Study Psychiatric condition studied Sample Sizes Outcomes Measured Results Interpretation of study
Uscher-Pines et al., USA, 2020 [55] Qualitative Semi structured interview N/A N/A 20 outpatient psychiatrists from 5 US states semi structured interviews N/A Interview summary Matrix analysis 1. Challenges affecting the quality of provider-patient interactions          2. Decreased clinical data for assessment,       3. Diminished patient privacy  4. Increased distractions in the patient's home setting. 5.Disadvantaged patients lacked reliable access to a computer, smartphone, and the internet.
Uscher-Pines et al., USA, 2020 [56] Semi structured interview OUD 18 clinicians from 10 states Changes included waiving urine toxicology screening, sending patients home with a larger supply of OUD medications, and requiring fewer visits Interview summary 1. Less structure and accountability,   2. Less information to inform clinical decision-making, 3. Challenges in establishing a connection,  4. Technological challenges, and shorter visits.
 
 

Given this, it is necessary to implement some advanced directives that will smoothly aid the shift toward telepsychiatry. We provide a few recommendations below. First and foremost, it will be crucial to evaluate telehealth competency among psychiatrists before establishing a standard for telepsychiatry training, in addition to creating frameworks for virtual psychiatric practice. This first step will provide a stable foundation for the entire psychiatric field to promote quality and regulated care.

Advance Directives

The following recommendations may be beneficial in improving telepsychiatry and telehealth services. Conducting and evaluating current telehealth competency among psychiatrists and physicians to promote quality care across practices [57]. Evaluate and improve current frameworks for technological competency and use in psychiatry [58]. Establish standardized training for psychiatrists who are currently practicing in the field extend telepsychiatry training to include undergraduate medical education for early training and introduction to telepsychiatry [59]. Provide patients with technological training to ensure that they have the digital literacy to use these virtual services and apps [60, 61]. Work with social workers to ensure that patients in remote areas have consistent access to technological resources.

Conclusions

Overall, the use of telemedicine to provide medical care is not novel, but the COVID-19 pandemic has forced the US healthcare system to promptly shift to telehealth, whether or not it was ready to do so. When looking at this shift to telemedicine, some specialities have been focused on greater than others, and psychiatry is one field that has been overlooked. This current review examined the feasibility, efficacy, and likelihood of successful implementation of telepsychiatry both during the pandemic and beyond. The advantageous benefits of telepsychiatry, whether it be used in conjunction with mobile smartphone applications or synchronous/asynchronous, are widespread and many. The potential of using virtual medicine to change the way the psychiatric field is structured is unbounded, although this transition is not without barriers or difficulties. To address these issues, preemptive steps must be taken to standardize practice and thoroughly prepare healthcare providers. With careful planning and collaborative teamwork, telepsychiatry implementation can provide practical and satisfactory delivery of mental healthcare, benefitting both patients and clinicians.


References

  1. Cutler D: How will COVID-19 affect the health care economy? . JAMA Health Forum. 2020, 1:e200419. 10.1001/jamahealthforum.2020.0419
  2. KFF Health Tracking Poll - May 2020. (2020). Accessed: May 27th, 2020: https://www.kff.org/report-section/kff-health-tracking-poll-may-2020-health-and-economic-impacts/.
  3. Mental health issues increased significantly in young adults over last decade. (2019). Accessed: March 15, 2019: https://www.sciencedaily.com/releases/2019/03/190315110908.htm..
  4. Social determinants of health series: Transportation and the role of hospitals. Health Research & Educational Trust, Chicago, IL; 2017. https://www.aha.org/system/files/hpoe/Reports-HPOE/2017/sdoh-transportation-role-of-hospitals.pdf.
  5. Patel UK, Malik P, DeMasi M, Lunagariya A, Jani VB: Multidisciplinary approach and outcomes of tele-neurology: a review. Cureus. 2019, 11:e4410. 10.7759/cureus.4410
  6. Salmoiraghi A, Hussain S: A systematic review of the use of telepsychiatry in acute settings. J Psychiatr Pract. 2015, 21:389-93. 10.1097/PRA.0000000000000103
  7. Lee MR, Ruehmann LL, Dively-White MT, Mathsen RL, Johnston JM, Jensen TB: Telehealth triage nursing and telemedicine: joining forces for improved care delivery in a primary care setting. Austin J health care. 2017, 4(1):1036.
  8. Korber B, Fischer WM, Gnanakaran S, et al.: Tracking Changes in SARS-CoV-2 Spike: Evidence that D614G Increases Infectivity of the COVID-19 Virus. Cell. 2020, 182:812-27. 10.1016/j.cell.2020.06.043
  9. Rossi R, Socci V, Pacitti F, Di Lorenzo G, Di Marco A, Siracusano A, Rossi A: Mental health outcomes among frontline and second-line health care workers during the coronavirus disease 2019 (COVID-19) pandemic in Italy. JAMA Netw Open. 2020, 3:e2010185. 10.1001/jamanetworkopen.2020.10185
  10. Ettman CK, Abdalla SM, Cohen GH, Sampson L, Vivier PM, Galea S: Prevalence of depression symptoms in US adults before and during the COVID-19 pandemic. JAMA Netw Open. 2020, 3:e2019686. 10.1001/jamanetworkopen.2020.19686
  11. Shi L, Lu ZA, Que JY, et al.: Prevalence of and risk factors associated with mental health symptoms among the general population in China during the coronavirus disease 2019 pandemic. JAMA Netw Open. 2020, 3:e2014053. 10.1001/jamanetworkopen.2020.14053
  12. Aronson SH: The Lancet on the telephone 1876-1975. Medical History. 1977, 21(1):69-87. 10.1017%2Fs0025727300037182
  13. Doarn CR: Telemedicine and psychiatry-a natural match. Mhealth. 2018, 4:60. 10.21037/mhealth.2018.12.04
  14. Bashshur RL, Shannon GW, Bashshur N, Yellowlees PM: The empirical evidence for telemedicine interventions in mental disorders. Telemed J E Health. 2016, 22:87-113. 10.1089/tmj.2015.0206
  15. Polinski JM, Barker T, Gagliano N, Sussman A, Brennan TA, Shrank WH: Patients' satisfaction with and preference for telehealth visits. J Gen Intern Med. 2016, 31:269-75. 10.1007/s11606-015-3489-x
  16. Patient experience - ortholive telehealth application. (2018). Accessed: September 27, 2020: https://www.ortholive.com/blog/patient-experience-ortholive-telehealth-application/.
  17. Telemedicine: 5 questions asked and answered. (2016). Accessed: June 13th, 2016: https://blogs.va.gov/VAntage/28397/telemedicine/.
  18. Telemedicine and Telehealth. (2020). Accessed: September 24, 2020: https://www.healthit.gov/topic/health-it-health-care-settings/telemedicine-and-telehealth.
  19. Nesbitt TS: The evolution of telehealth: where have we been and where are we going?. Institute of Medicine. The National Academies Press, Washington (DC); 2012.
  20. Preston J, Brown FW, Hartley B: Using telemedicine to improve health care in distant areas. Hosp Community Psychiatry. 1992, 43:25-32. 10.1176/ps.43.1.25
  21. Estimated cost savings. (2020). Accessed: September 27, 2020: https://www.ohsu.edu/telemedicine/estimated-cost-savings.
  22. Who bears the costs and receives the benefits of telehealth?. (2017). Accessed: September 27, 2020: https://prognocis.com/who-bears-the-costs-and-receives-the-benefits-of-telehealth/.
  23. The two forms of telemedicine: synchronous vs. asynchronous. (2019). Accessed: September 10, 2019: https://blog.cureatr.com/two-forms-telemedicine-synchronous-vs-asynchronous.
  24. Wilson LS, Maeder AJ: Recent directions in telemedicine: review of trends in research and practice. Healthc Inform Res. 2015, 21:213-22. 10.4258/hir.2015.21.4.213
  25. 8 Advantages and benefits of telehealth. (2019). Accessed: August 22, 2019: https://blog.cureatr.com/8-advantages-benefits-telehealth.
  26. Store-and-forward (asynchronous). (2020). Accessed: December 26, 2020: https://www.cchpca.org/what-is-telehealth/?category=store-and-forward.
  27. Yellowlees P, Burke Parish M, González Á, et al.: Asynchronous telepsychiatry: a component of stepped integrated care. Telemed J E Health. 2018, 24:375-8. 10.1089/tmj.2017.0103
  28. Store-and-forward telemedicine services expand connected health. (2018). Accessed: December 26, 2020: https://mhealthintelligence.com/features/store-and-forward-telemedicine-services-expand-connected-health.
  29. Farabee D, Calhoun S, Veliz R: An experimental comparison of telepsychiatry and conventional psychiatry for parolees. Psychiatr Serv. 2016, 67:562-5. 10.1176/appi.ps.201500025
  30. Xiong GL, Iosif AM, Godwin HT, Khan M, Parish MB, Yellowlees P, Kahn D: A pilot randomized trial of asynchronous and synchronous telepsychiatry in skilled nursing facilities. J Am Med Dir Assoc. 2018, 19:461-2. 10.1016/j.jamda.2018.02.007
  31. Shulman M, John M, Kane JM: Home-based outpatient telepsychiatry to improve adherence with treatment appointments: a pilot study. Psychiatr Serv. 2017, 68:743-6. 10.1176/appi.ps.201600244
  32. O’Reilly R, Bishop J, Maddox K, Hutchinson L, Fisman M, Takhar J: Is telepsychiatry equivalent to face-to-face psychiatry? Results from a randomized controlled equivalence trial. Psychiatr Serv. 2007, 58:836-43. 10.1176/ps.2007.58.6.836
  33. Zou L, Yeung A, Li C, et al.: Effects of meditative movements on major depressive disorder: a systematic review and meta-analysis of randomized controlled trials. J Clin Med. 2018, 7:195. 10.3390/jcm7080195
  34. Chong J, Moreno F: Feasibility and acceptability of clinic-based telepsychiatry for low-income Hispanic primary care patients. Telemed J E Health. 2012, 18:297-304. 10.1089/tmj.2011.0126
  35. De Las Cuevas C, Arredondo MT, Cabrera MF, Sulzenbacher H, Meise U: Randomized clinical trial of telepsychiatry through videoconference versus face-to-face conventional psychiatric treatment. Telemed J E Health. 2006, 12:341-50. 10.1089/tmj.2006.12.341
  36. Frueh BC, Monnier J, Grubaugh AL, Elhai JD, Yim E, Knapp R: Therapist adherence and competence with manualized cognitive-behavioral therapy for PTSD delivered via videoconferencing technology. Behav Modif. 2007, 31:856-66. 10.1177/0145445507302125
  37. Ruskin PE, Silver-Aylaian M, Kling MA, et al.: Treatment outcomes in depression: comparison of remote treatment through telepsychiatry to in-person treatment. Am J Psychiatry. 2004, 161:1471-6. 10.1176/appi.ajp.161.8.1471
  38. Moreno FA, Chong J, Dumbauld J, Humke M, Byreddy S: Use of standard webcam and internet equipment for telepsychiatry treatment of depression among underserved Hispanics. Psychiatr Serv. 2012, 63:1213-7. 10.1176/appi.ps.201100274
  39. Bishop JE, O'Reilly RL, Maddox K, Hutchinson LJ: Client satisfaction in a feasibility study comparing face-to-face interviews with telepsychiatry. J Telemed Telecare. 2002, 8:217-21. 10.1258/135763302320272185
  40. Hulsbosch AM, Nugter MA, Tamis P, Kroon H: Videoconferencing in a mental health service in The Netherlands: a randomized controlled trial on patient satisfaction and clinical outcomes for outpatients with severe mental illness. J Telemed Telecare. 2017, 23:513-20. 10.1177/1357633X16650096
  41. Maieritsch KP, Smith TL, Hessinger JD, Ahearn EP, Eickhoff JC, Zhao Q: Randomized controlled equivalence trial comparing videoconference and in person delivery of cognitive processing therapy for PTSD. J Telemed Telecare. 2016, 22:238-43. 10.1177/1357633X15596109
  42. Myers K, Vander Stoep A, Zhou C, McCarty CA, Katon W: Effectiveness of a telehealth service delivery model for treating attention-deficit/hyperactivity disorder: a community-based randomized controlled trial. J Am Acad Child Adolesc Psychiatry. 2015, 54:263-74. 10.1016/j.jaac.2015.01.009
  43. Hur JW, Kim B, Park D, Choi SW: A scenario-based cognitive behavioral therapy mobile app to reduce dysfunctional beliefs in individuals with depression: a randomized controlled trial. Telemed J E Health. 2018, 24:710-6. 10.1089/tmj.2017.0214
  44. de Sá M, Carriço L: Fear therapy for children: a mobile approach.. Proceedings of the 4th ACM SIGCHI symposium on engineering interactive computing systems. 2012, EICS '12:Pages 237-46. 10.1145/2305484.2305524
  45. Iiboshi K, Yoshida K, Yamaoka Y, et al.: A validation study of the remotely administered montreal cognitive assessment tool in the elderly Japanese population. Telemed J E Health. 2020, 26:920-8. 10.1089/tmj.2019.0134
  46. Graham AK, Greene CJ, Kwasny MJ, Kaiser SM, Lieponis P, Powell T, Mohr DC: Coached mobile app platform for the treatment of depression and anxiety among primary care patients: a randomized clinical trial. JAMA Psychiatry. 2020, 77:906-14. 10.1001/jamapsychiatry.2020.1011
  47. Donker T, Cornelisz I, van Klaveren C, van Straten A, Carlbring P, Cuijpers P, van Gelder JL: Effectiveness of self-guided app-based virtual reality cognitive behavior therapy for acrophobia: a randomized clinical trial. JAMA Psychiatry. 2019, 76:682-90. 10.1001/jamapsychiatry.2019.0219
  48. Espie CA, Emsley R, Kyle SD, et al.: Effect of digital cognitive behavioral therapy for insomnia on health, psychological well-being, and sleep-related quality of life: a randomized clinical trial. JAMA Psychiatry. 2019, 76:21-30. 10.1001/jamapsychiatry.2018.2745
  49. Ainsworth J, Palmier-Claus JE, Machin M, et al.: A comparison of two delivery modalities of a mobile phone-based assessment for serious mental illness: native smartphone application vs text-messaging only implementations. J Med Internet Res. 2013, 15:e60. 10.2196/jmir.2328
  50. Macias C, Panch T, Hicks YM, Scolnick JS, Weene DL, Öngür D, Cohen BM: Using smartphone apps to promote psychiatric and physical well-being. Psychiatr Q. 2015, 86:505-19. 10.1007/s11126-015-9337-7
  51. Ben-Zeev D, Brenner CJ, Begale M, Duffecy J, Mohr DC, Mueser KT: Feasibility, acceptability, and preliminary efficacy of a smartphone intervention for schizophrenia. Schizophr Bull. 2014, 40:1244-53. 10.1093/schbul/sbu033
  52. Zarghami A, Farjam M, Fakhraei B, Hashemzadeh K, Yazdanpanah MH: A report of the telepsychiatric evaluation of SARS-CoV-2 patients. Telemed J E Health. 2020, 26:1461-5. 10.1089/tmj.2020.0125
  53. Chen JA, Chung WJ, Young SK, et al.: COVID-19 and telepsychiatry: early outpatient experiences and implications for the future. Gen Hosp Psychiatry. 2020, 66:89-95. 10.1016/j.genhosppsych.2020.07.002
  54. COVID-19 tip sheet: story ideas from Johns Hopkins. (2020). Accessed: October 11, 2020: https://www.hopkinsmedicine.org/news/newsroom/news-releases/covid-19-tip-sheet-story-ideas-from-johns-hopkins-4.
  55. Uscher-Pines L, Sousa J, Raja P, Mehrotra A, Barnett ML, Huskamp HA: Suddenly becoming a "Virtual Doctor": experiences of psychiatrists transitioning to telemedicine during the COVID-19 pandemic. Psychiatr Serv. 2020, 71:1143-50. 10.1176/appi.ps.202000250
  56. Uscher-Pines L, Sousa J, Raja P, Mehrotra A, Barnett M, Huskamp HA: Treatment of opioid use disorder during COVID-19: experiences of clinicians transitioning to telemedicine. J Subst Abuse Treat. 2020, 118:108124. 10.1016/j.jsat.2020.108124
  57. Hilty DM, Maheu MM, Drude KP, Hertlein KM: The need to implement and evaluate telehealth competency frameworks to ensure quality care across behavioral health professions. Acad Psychiatry. 2018, 42:818-24. 10.1007/s40596-018-0992-5
  58. Hilty D, Chan S, Torous J, Luo J, Boland R: A framework for competencies for the use of mobile technologies in psychiatry and medicine: scoping review. JMIR Mhealth Uhealth. 2020, 8:e12229. 10.2196/12229
  59. Waseh S, Dicker AP: Telemedicine training in undergraduate medical education: mixed-methods review. JMIR Med Educ. 2019, 5:e12515. 10.2196/12515
  60. Hoffman L, Wisniewski H, Hays R, et al.: Digital opportunities for outcomes in recovery services (DOORS): a pragmatic hands-on group approach toward increasing digital health and smartphone competencies, autonomy, relatedness, and alliance for those with serious mental illness. J Psychiatr Pract. 2020, 26:80-8. 10.1097/PRA.0000000000000450
  61. Kichloo A, Albosta M, Dettloff K, et al.: Telemedicine, the current COVID-19 pandemic and the future: a narrative review and perspectives moving forward in the USA. Fam Med Community Health. 2020, 8:e000530. 10.1136/fmch-2020-000530

Appendices

Jayasudha Gude and Rashmi V. Subhedar work as equal authors.

Authors' contributions: Conceptualization: Jayasudha Gude and Rashmi V. Subhedar; Acquisition of literature: Michelle H. Zhang, Pratik Jain, Jatminderpal Bhela; Formal investigation: Jayasudha Gude, Rashmi V. Subhedar; Writing - original draft preparation: Jayasudha Gude, Rashmi V. Subhedar, Fariha Bangash, Nikhila Veluri, Ya-Ching Hsieh; Writing - review, critical feedback, and editing: Batool Z. Sheikh, Mansi R. Shah, Kapil Aedma, Urvish K. Patel, Tapan Parikh; Supervision: Kapil Aedma,Tapan Parikh.


Review article
peer-reviewed

Emerging Needs and Viability of Telepsychiatry During and Post COVID-19 Era: A Literature Review


Author Information

Jayasudha Gude Corresponding Author

Psychiatry, Northwell Health, Zucker Hillside Hospital, New York, USA

Rashmi V. Subhedar

Family Medicine, Mc Master University, Hamilton, CAN

Michelle H. Zhang

Psychological & Brain Sciences and Biology, Johns Hopkins University, Baltimore, USA

Pratik Jain

Psychiatry, State University of New York Upstate Medical University, Syracuse, USA

Jatminderpal Bhela

Psychiatry, Case Western Reserve University/Metrohealth system, Cleveland, USA

Fariha Bangash

Psychiatry, State University of New York Upstate Medical University, Syracuse, USA

Nikhila Veluri

Psychiatry, American University of Integrative Science School of Medicine, St. Michael, BRB

Ya-Ching Hsieh

Public Health, Icahn School of Medicine at Mount Sinai, New York, USA

Batool Z. Sheikh

Psychiatry, Dow University of Health Sciences, Karachi, PAK

Psychiatry, Brookdale University Hospital Medical Center, New York, USA

Mansi R. Shah

Psychiatry, The Wright Center for Graduate Medical Education, Scranton, USA

Zeeshan Mansuri

Psychiatry, Boston Children’s Hospital/Harvard Medical School, Boston, USA

Kapil Aedma

Psychiatry, Unity Point, Peoria, USA

Urvish K. Patel

Public Health and Neurology, Icahn School of Medicine at Mount Sinai, New York, USA

Tapan Parikh

Psychiatry, Northwestern University Feinberg School of Medicine, Chicago, USA


Ethics Statement and Conflict of Interest Disclosures

Conflicts of interest: In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work. Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work. Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.



Review article
peer-reviewed

Emerging Needs and Viability of Telepsychiatry During and Post COVID-19 Era: A Literature Review


Figures etc.

SIQ
8.1
RATED BY 2 READERS
CONTRIBUTE RATING

Scholarly Impact Quotient™ (SIQ™) is our unique post-publication peer review rating process. Learn more here.