Toward Understanding Impacts of Virtual Visits on Accreditation Review Effectiveness, Quality, and Best Practices

Purpose

ATMAE required ad hoc transitions to conduct accreditation reviews virtually during the spring of 2020 without full knowledge of the impact that virtual site visits would have on the effectiveness and quality of accreditation reviews against its 2019 Accreditation Standards (Board of Accreditation, 2019). It is reasonable to expect similar modes of evaluation to continue into the near future, even though the unprecedented events of the spring of 2020 were unique. Therefore, a critical need existed to evaluate the efficacy of this mode of accreditation review. ATMAE and other higher education institutions, accrediting agencies, and governing bodies (e.g., Council for Higher Education Accreditation [CHEA]) will be able to make data-informed decisions regarding virtual site visits or proactively prepare for similar global scenarios with this evaluation. Therefore, to begin to understand the impacts of virtual site visits on the accreditation review process, this study posed two research questions:

• 1.

  What were the perceived effectiveness and quality of virtual accreditation site visits?

• 2.

  What lessons were learned that provide preliminary best practices for future virtual accreditation site visits?

Background

The COVID-19 pandemic of 2020 and 2021 put significant pressure on higher education institutions in several ways (Priddy & Pelletier, 2020). One such pressure point was accreditation site visits. Accreditation bodies such as ATMAE include criteria that evaluate “curriculum,” “facilities,” and “institutional support,” suggesting the importance of the onsite visit. To date, limited research has been published evaluating virtual approaches to accreditation visits. Kinzie (2020) suggested that disruptions resulting from COVID-19 are an opportunity for improvement. However, much of the initial discussion has focused on learning outcomes and equity challenges in student resiliency and effective learning. Although policies and logistical guidance for accreditation visits during the COVID-19 pandemic have been published, most address medical disciplines, including nursing and medical transport. Cobourne & Shellenbarger (2021) discussed adaptations in preparation and planning for a virtual accreditation visit in nursing but did not address the quality of the visits related to accreditation. Frazer (2021), speaking on behalf of the Commission on Accreditation of Medical Transport Systems (CAMTS), advised programs to do the best they could to meet standards related to quality and safety and serve the patient. Potts et al. (2021), speaking for the Accreditation Council for Graduate Medical Education and its Review Committee, characterized the virtual accreditation visit as “forced” on them due to the COVID-19 pandemic. They described the tools used in accreditation, disruptions in the accreditation system, and the impact on resulting accreditation decisions. However, surgical, nursing, and medical transport education have a different educational context than the conventional classroom and laboratory learning typical of STEM disciplines.

Eaton (2020), President Emeritus of CHEA, noted that as higher education institutions change and innovate, so must accreditation. Eaton asserted that the most significant shift in accreditation has been how institutions and programs are reviewed and examined. The move from in-person visits to virtual visits through Zoom™ or other video conferencing platforms changed engagement. What is less clear is the impact this change has imposed. The pandemic has been a force for change in many parts of society—It was not expected to change the higher education environment. Virtual accreditation site visits potentially address many social and parity concerns and broaden the scope of reviews to include alternative providers and alternative credentials. However, limited research has examined how STEM-focused accreditation bodies can leverage this review modality.

Research design

This study used a mixed-method approach to understand the impacts and lessons learned from ATMAE virtual site visits completed during the spring of 2020. Specifically, qualitative and quantitative survey data were collected from a representative sample from three groups: 1) ATMAE Board of Accreditation (BoA) members, 2) ATMAE visiting team members, and 3) institution contact persons. Study participants in each of the three survey groups were given 4 weeks to complete the electronic survey, with weekly reminders to participate, as recommended by the Tailored Design Method (Dillman et al., 2014). The BoA survey group was responsible for reviewing all visiting teams’ and institutions’ self-study reports and making final accreditation decisions (Board of Accreditation, 2022). The visiting team group was responsible for virtually visiting and reviewing each institution’s program(s) and facility(ies) and meeting with administrator, faculty, staff, and students to collect data to inform a team accreditation report that was submitted to the BoA for final review (Board of Accreditation, 2022). The institutional contact group was composed of the point people at each institution applying for accreditation, who were responsible for developing a program self-study report, coordinating logistics for the accreditation review, and attending accreditation hearings with the BoA (Board of Accreditation, 2022).

Research setting

Traditionally, ATMAE visits take place on the college campus, submitting for accreditation over 2 days. The accreditation team interviews faculty, staff, administration, students, and the advisory board to verify the program’s submitted self-study report, tour laboratories and classrooms, and inspect the equipment utilized to deliver the program to ensure that proper facilities are used and maintained. The team addresses any shortcomings or concerns and provides feedback to the faculty, staff, and administrators on the end of the second day.

Due to the COVID-19 pandemic, ATMAE moved its spring 2020 site visits to a virtual modality. These virtual site visits largely followed the same structure as the in-person visits, minus in-person observations and conversations (i.e., lab tours, discussions with students, etc.). The institutions under review utilized various video conferencing platforms and provided the visiting teams with meeting links to complete the required interviews.

Measures

The study characterized effectiveness, quality, advantage/disadvantage, and leverage measures using the survey questions and data types indicated in Table 1. This survey instrument presented specific definitions for effectiveness and quality to survey participants as follows: “Effectiveness was the extent to which planned activities (e.g., accreditation self-study reports and accompanying site visits) are realized and planned results (e.g., program accreditation decisions) are achieved”; “Quality was the inherent perception of the user (e.g., customer’s perspective), which in this case was the accreditation team and/or institution being reviewed.” Measures of advantage/disadvantage were defined during the analysis based on positive (advantage) or negative (disadvantage) themes in the data set. In contrast, measures of leverage were defined based on “Yes/Maybe/No” responses coupled with themes found in the data set.

Data analysis

Quantitative methods were used to provide a descriptive analysis of the demographic characteristics of the survey sample. Specifically, counts, percentages, and medians were calculated to characterize each survey group based on size, response rate, age, and experience level. Age was specifically evaluated due to the nature of the research setting, in which younger individuals were assumed to adapt to the virtual modality more readily than older individuals. Similarly, the experience level of the BoA and team members was another important characteristic that framed the results of the study.

To answer the study’s research questions, a mix of data analysis methods were used to evaluate qualitative and quantitative data collected on each of the research measures (see Table 1). Specifically, non-parametric binomial exact tests (α = 0.05) were used to analyze differences in proportions of responses to effectiveness and quality measures collected using 5-point Likert scale survey questions as well as differences in proportions of categorical responses (e.g., “Yes,” “Maybe,” “No”) to qualitative open-ended questions. These tests were not used to answer study hypotheses but to statistically support whether significant perspectives were evident in the data set. Next, a qualitative content analysis was conducted that assigned sentiment codes for each of the open-ended responses from the N = 86 respondents. Responses that included affirmative phrasing received positive-sentiment codes (e.g., “We can deal with issue such as virus travel restrictions” was given a “positive” code) whereas responses that included non-affirmative phrasing received negative-sentiment codes. Furthermore, the authors conducted a thematic analysis of the qualitative results to form a rich description of collective meaning for each of the study’s measures, similar to Haughery & Raman (2016). Themes were developed and defined using a grounded theory approach that formed a common language of meaning for each measure across all respondents (Gough et al., 2012). These themes were then defined and reported to support a synthesis of the research results (Borrego et al., 2014). From this mix of analysis methods, answers to the study’s research questions were triangulated.

Descriptive

This study collected self-reported survey responses of perceptions of effectiveness, quality, advantage/disadvantage, and leverage from the spring 2020 ATMAE accreditation virtual site visits. As indicated by Table 2, the overall response rate of the study was 41% (n = 35 responses from a survey population of N = 86). Not surprisingly, the BoA group had the highest response rate (65%) possibly because they were the most invested in accreditation efforts as they were already the individuals volunteering their time to serve on this board. Interestingly, the institutional contact group also had a response rate above 50%. This could be attributed to the timing of data collection. The survey was distributed before final decisions had been made by the BoA, and the institutional contacts may have been more willing to comply with accreditation-related requests (i.e., responding to the survey).

Table 3 shows the age (years) of respondents and institutional type they were employed by. Respondent age across the entire data set was normally distributed, based on a Shapiro-Wilk test of normality (p = 0.8304), and the mean age was 50–59 years. Not surprisingly, most respondents (n = 24) were employed at 4-year institutions offering Bachelor of Science and/or Master of Science degree programs. It is important to note that n = 4 did not provide age data and n = 7 did not provide employment data.

Respondents’ experience level related to ATMAE accreditation was collected and is illustrated in Table 4. These data allowed for a fuller interpretation of the results. BoA member experience level was measured in years served on the Board, accreditation team member experience as number of appointments as a team chair or team member, and institutional contact experience as the number of accreditation self-studies prepared. The median years of experience as a BoA member among respondents was Median_BoA = 2.0, and the median number of appointments to a visiting team was Median_Team = 1.5 (Median_Member = 3.0; Median_Chair = 0.0). These results illustrate level of experience. For institution contact experience, the median number of self-studies prepared was Median_Inst = 3.0, indicating more experience for this survey group.

Effectiveness and quality

The first research question asked what the perceived effectiveness and quality of virtual accreditation site visits were. To help answer this question, the survey instrument asked the BoA members whether virtual visits were effective at upholding the rigor of accreditation reviews. Categorizing BoA members’ sentiments, it was found that n = 3 (27%) felt that virtual site visits were not effective (“no”), n = 1 (9%) were undecided (“unknown”), and n = 7 (64%) felt positive (“yes”), as indicated in Table 5. Furthermore, when a thematic analysis of rationales provided for a sentiment was performed, a strong theme of efficacy became evident in BoA members’ open-ended responses, as illustrated in Table 6. This theme of efficacy was defined as “the intended accreditation results were met, but there was a lack of in-person interaction that posed challenges to the review process.” While the observed theme of efficacy came from the entire pool of responses, regardless of sentiment, a statistically higher proportion of responses (p = 0.049) felt positive toward the effectiveness of virtual site visits to uphold the rigor of accreditation review. These results illustrate the confidence that BoA members appeared to have in the virtual modality, albeit tempered by the practical void of physical interaction that was reported to detract from the review process.

Team members and institutional contacts were asked to rate the virtual modality for each standard to further develop the assessment of effectiveness. All standards except two were rated as “effective” to “very effective” (all p < 0.050; n = 24) on a 5-point Likert scale, as indicated in Table 7. These results did not change between team member and institutional contact responses (per accreditation standard comparison: all p < 0.050). Not surprisingly, Standard 11 (Facilities, Equipment & Technical Support) was not found to be more “effective” to “very effective” compared with “very ineffective” to “neutral” (p = 0.054). The second standard to receive a higher proportion of “very ineffective” to “neutral” ratings was Standard 17 (Advisory Committee Approval of Overall Program) (p = 0.114). This result is more surprising, because the ability to meet with an institution’s advisory committee was still technically available to visiting teams, although this was more logistically challenging due to the newness of a virtual video conference format during the spring of 2020.

Based on the data collected from BoA members, team members, and institutional contacts, it was apparent that spring 2020 virtual site visits were perceived to be effective at upholding the rigor of accreditation review. To assess the impact on quality, survey data from BoA members, team members, and institutional contacts were evaluated and are presented in Table 8. It was found that n = 3 (27%) felt the impact on quality was negative, n = 6 (55%) felt it was neutral, and n = 2 (18%) felt the impact was positive. Statistically, there was no difference in the proportion of negative versus not negative sentiment (p = 0.273) toward an impact on quality. These results lack a distinctive sentiment or, more importantly, do not illustrate a distinctive negative sentiment.

To further unpack BoA members’ rationale for sentiments as to quality, the study used thematic analysis to identify a theme of outcomes across all negative, neutral, and positive results. As illustrated in Table 9, this theme was defined as “some standards were challenging, some were easy, and some were the same to assess using a virtual modality.” Even though BoA members did not agree in their sentiment regarding the impact that the virtual modality had on quality, there was a strong focus on outcomes among responses, point to a status quo impact on the quality of the review process. This inference is supported by the statistically insignificant proportion of results among sentiment categories (p = 0.273), as shown in Table 8.

To offer a fuller perspective, the survey asked team members and institutional contacts to rate the perceived impact on quality for each standard. As Table 10 indicates, no statistical difference was found between the proportion of “high quality” to “very high quality” and “neutral” to “very low quality” ratings for all standards, except three. The three standards that did exhibit statistically different ratings (i.e., “high quality” to “very high quality” vs. “neutral” to “very low quality”) were Standard 1 (Preparation of Self-Study; all p = 0.022), Standard 2 (Program Definition; all p = 0.022), and Standard 4 (Program Goals; all p = 0.022). Statistical differences in ratings are not surprising, as these standards inherently lend themselves to a virtual review modality. Moreover, no statistical differences were found between team members and institutional contacts per standard ratings (all p < 0.050; n = 24).

Implications

The study revealed several results that hint at how future virtual accreditation visits may be improved. Among the inferences noted is that virtual accreditation worked better than expected, regardless of a strong preference of about one-third of respondents for face-to-face visits. Virtual accreditation was overwhelmingly perceived as effective in assessing program rigor across all standards except two. Considering the effectiveness of virtual site visits, Standard 11 (Facilities, Equipment & Technical Support) and Standard 17 (Advisory Committee Approval of Overall Program) were the only standards perceived to have negative impacts. Non-negative impacts for all other standards indicated that they are candidates for future virtual assessment. The quality of the accreditation review process was not perceived to be affected by the virtual visit, with the assumption being that the ATMAE standards addressed by self-studies provide a better indicator of program excellence than personal contact by the visiting team.

Some level of a virtual accreditation process could replace traditional accreditation visits. This was supported by the responses regarding quality impact of the virtual visits. However, a strong preference remains for the validation of Standard 11 and Standard 17 using some form of site visitation. The type and availability of facilities and equipment and the level of technical support is a crucial component for programs that promote project-based learning and hands-on experiences. Furthermore, the support of industry and advice of working alumni continues to be a hallmark of the ATMAE program’s relevancy. Advisor board level of commitment and support is not easily determined via phone call or teleconference.

Virtual accreditation was efficient in eliminating non-value activities. The monetary cost and physical toll of travel on individuals encourage the increased use of virtual methods. It is becoming increasingly difficult to justify the value of travel and face-to-face communications as viable, particularly with ever-changing requirements for masks and social distancing. Virtual accreditation, if organized well, has the potential for reducing the amount of institutional site preparation, participant stress, and overall time spent.

Socializing and face-to-face communication continue to be highly valued as a means of assessing and validating certain aspects of programs. As noted previously, a site visit seems to be perceived as highly valuable for properly assessing the institutional program resources and support. The conventional wisdom that trust is based on eye contact and a handshake is still true. The sudden need for virtual accreditation methods was stressful for those individuals who value face-to-face communication. However, the more the virtual world can be enhanced to replicate the personal touch, the more likely the acceptance of electronic technology for accreditation.

Limitations

The factors limiting the validity and generalizability of this study follow. First, at the time of this publication, very few sources of literature existed related to factors influencing the effectiveness, quality, or lessons learned from in-person or virtual accreditation site visits. Even so, the authors made every effort to fully review all available literature to inform this study. To that end, a review of literature started with a Web of Science search to locate published academic articles. A second index, Dissertations and Theses Global, was used to locate dissertations and theses on the topic, followed by a search of Google Scholar to locate other accreditation reports from non-academic sources. In all searches, keyword search terms included “accreditation,” “accreditation visit,” “virtual accreditation,” “site visit,” “assessment,” and “engineering accreditation.” Furthermore, the sample size was too small to conclude with certainty that virtual accreditation can completely replace the traditional site visit. However, aspects of the virtual accreditation process are already in place and will continue to operate in this fashion. Second, the themes of effectiveness and quality are highly subjective, even with definitions provided. Respondent bias was likely present; individual perspectives about the level to which planned activities were realized and results were achieved varied greatly. Each accreditation team and institution were unique. Even though the study considered the results from a limited period and small survey sample, every effort was taken to rigorously and judiciously evaluate the data to begin to understand impact of virtual site visits on accreditation review effectiveness, quality, and best practices.

Future evaluation

Opportunities for future evaluation exist and should be longitudinal in nature to capture the rhythms of accreditation cycles for institutions not previously accredited or for those seeking re-accreditation. The interpersonal nuances and cooperation between accreditation visiting teams and institutions not previously surveyed will add depth to the analysis. This will also increase the sample size and generalizability of the findings over a wider range of institutions and visiting teams.

Future research could include additional questions that explore themes beyond accreditation effectiveness and quality. An ongoing concern is consistency of accreditation standards interpretation by visiting teams and the way in which institutions demonstrate compliance with the standards. Virtual methods of accreditation could offer opportunities for greater process oversight by directly involving accreditation board members or their proxies.