A Comparison of Consumer Responses Using Paper and Digital Ballots for Eating Quality Assessment of Beef Steaks

Production selection and collection

Carcasses (n = 39) of beef (USDA, 2017) were selected from a commercial beef processing facility in Friona, Texas. Thirteen carcasses per quality grade category (USDA Prime, USDA Choice [lower 1/3], and USDA Select) were selected. Paired strip loins (Institutional Meat Purchase Specification [IMPS] 180) and paired eye of rounds (IMPS 171C) were collected from each USDA Select carcass (North American Meat Institute [NAMI], 2014). Paired tenderloins (IMPS 189A) were collected from each USDA Choice carcass (NAMI, 2014). Paired strip loins (IMPS 180) were collected from each USDA Prime carcass (NAMI, 2014).

Carcasses were selected and verified by trained Texas Tech University personnel through visual appraisal of marbling and maturity (USDA, 2017) of the carcass at the time of selection. Carcass data, including beef carcass yield and quality grade traits and longissimus muscle pH (TPS Model WP-90 with pH sensor part #111227; TPS Pty Ltd., Brendale, QLD, Australia), were collected and recorded by trained personnel. All subprimals were transported by refrigerated truck to the Gordon W. Davis Meat Science Laboratory in Lubbock, Texas. Subprimals were stored in vacuum bags at 2°C to 4°C in the absence of light until steak fabrication on day 7 postmortem.

Steak fabrication

All external fat and connective tissue were removed from all subprimals prior to steak fabrication. In addition, the gluteus medius was removed from the strip loin, leaving only the longissimus lumborum. The psoas minor and iliacus were removed from the tenderloin, leaving only the psoas major. Subprimals were then fabricated into 2.5-cm steaks, maintaining position number for each steak. Steaks were further portioned into 2 equal pieces for consumer testing, and the 2 pieces were assigned randomly to half A and half B. Steak halves were individually vacuum packaged, maintaining the identity of the subprimal, steak position, and half (A or B). All samples were held at 2°C to 4°C and frozen on the respective postmortem aging day: Select longissimus lumborum steaks–7 d (SE-LL); Select semitendinosus steaks–7 d (SE-ST); Choice psoas major steaks–21 d (CH-PM); and Prime longissimus lumborum steaks–21 d (PR-LL). Variation in postmortem aging was deliberate to promote variation in eating quality, specifically tenderness.

Compositional analysis

Proximate analysis was conducted using the anterior most steak from each subprimal. External fat and connective tissue were trimmed from the steak, and the remaining muscle was ground (ProSeries DC Meat Grinder, Cabela’s L.L.C., Sidney, NE) in triplicate through a 4-mm plate for proximate analysis. Proximate analysis of raw steaks was conducted by an AOAC official method (Anderson, 2007) using a near infrared spectrophotometer (FoodScan; FOSS NIRsystems, Inc., Laurel, MD). Chemical percentages of fat, moisture, and protein were determined for each subprimal.

Consumer panels

Consumer panels were conducted in the Texas Tech University Animal and Food Sciences Building. Consumer panelists (n = 360) were recruited from Lubbock, Texas, and the surrounding local communities. Each consumer was monetarily compensated for being a participant and was only allowed to participate one time. Fifteen sessions, each consisting of 24 people, were conducted over 8 evenings. Each session lasted approximately 60 min.

Cooking procedures followed MSA protocols (Watson et al., 2008) with modification for number of samples cooked per round. The grill was preheated 45 min prior to cooking, and 12 steak pieces (unrelated to the trial) were cooked to condition the grill and ensure stable temperatures throughout all heating elements (Gee, 2006). An exact time schedule was followed to ensure all steaks were prepared identically and facilitate continued consistency of the heating elements. Each cooking round consisted of 12 samples (as opposed to the traditional 10 samples for standard MSA sessions) that were cooked on the grill at one time. Twelve samples were cooked to accommodate 24 panelists per session, rather than the traditional 20 panelists. Each session consisted of 8 cooking/serving rounds, corresponding to the 8 samples that would be served to each consumer. For each round of sample evaluation, steak samples were cooked following a precise time schedule. Each cooking round lasted 5 min and 45 s, to target a medium degree of doneness. Samples were rested for 3 min and cut into 2 equal halves and served to 2 separate predetermined consumer panelists. Panelists received their samples 7 min apart. By employing this fabrication and cooking method, 2 consumers were able to evaluate the same sample using both ballot formats and limit the variation in responses between the 2 ballot formats that can be attributed to the samples themselves.

Each consumer evaluated 8 samples, with 4 samples per section or block. USDA SE-LL steaks aged 7 d were included in the cooking order and served in the first position of each block of 4 samples—one for the paper ballot and one for the digital ballot block. These warm-up samples were always and only served in the first and fifth rounds as described by Watson et al. (2008), followed by 3 test samples served in a predetermined, balanced order.

Six test samples were served representing muscle/quality grade combinations described earlier and were evaluated using 2 ballot formats (paper and digital). Following the warm-up samples (round 1 and round 5), muscle/quality grade presentation order were randomized within each ballot format (in advance). Half of the consumers in a session (consumers 1–12) scored their first 4 samples using the digital ballot, while the other half (consumers 13–24) scored their first four samples using paper ballots. In adherence to MSA cooking style, 2 consumers evaluated 1 steak half. For samples 6 to 8, muscle/quality grade presentation order was re-randomized (in advance), and consumers used the alternative ballot format to evaluate the opposite half of the steaks that were evaluated in rounds 1 to 4, as shown by Figure 1.

Figure 1.

Serving order for samples.

Consumers rated tenderness, juiciness, flavor liking, and overall liking on 100-mm line scales following MSA protocol (Watson et al., 2008) on a paper or digital ballot. Specifically, consumers would mark the line on the paper ballot by drawing a single vertical mark using a pencil on the horizontal line scale for each attribute (Figure 2a). Consumer responses recorded on paper ballots were digitized on a GTCO Calcomp Drawing Board connected to a corded click tip pen (The Logic Group, Austin, TX). The Logic Group’s digitizing software was used to set up parameters for data collection and export all data into an electronic format. Each paper ballot was digitized independently in duplicate. Data were checked for accuracy before averaging the 2 responses. When starting the digital evaluation for each sample, the slider was always aligned to the left and set at zero by default (Figure 2b). Participants had to press and drag the marker to the desired location on the scale. Numerical scores were not displayed on the tablet during evaluation. That feature was disabled for this study. The zero anchors were labeled as not tender, not juicy, and dislike extremely of flavor and overall. The 100 anchors were labeled as very tender, very juicy, and like extremely of flavor and overall. Vertical hash marks were present on the lines, regardless of ballot format, at 20, 40, 60, and 80 mm. In addition, a composite score to predict meat quality using 4 traits, referred to as MQ4, was calculated as follows: Tenderness(0.3) + Juiciness(0.1) + Flavor Liking(0.3) + Overall Liking(0.3) = MQ4. Consumers were also asked to check one of 4 boxes for eating quality level to indicate whether they considered each sample “unsatisfactory,” “good everyday quality,” “better than everyday quality,” or “premium quality,” as is customary with MSA consumer testing protocols.

Figure 2.

(a) The top line illutrates the Tenderness line scale on the paper ballot. The (gray) mark shows how a consumer would score the tenderness trait for a given sample—with a single vertical mark. (b) The bottom line represents the Tenderness line scale on the digital ballot. When starting the digital evaluation for each sample, the slider is always aligned to the left at zero. Participants must press and slide the marker to the desired location on the scale. No numbers appear when sliding the marker.

Each panelist was seated at a numbered booth and was provided with an electronic tablet, 4 paper ballots, plastic utensils, a toothpick, unsalted crackers, a napkin, an empty cup, a water cup, and a cup with diluted apple juice (10% apple juice and 90% water). Each ballot consisted of a demographic questionnaire (digital), 4 paper ballots, 1 iPad (Apple Inc., Cupertino, CA) preloaded with 4 corresponding digital ballots, and a post-panel survey regarding beef purchasing habits (digital). For samples evaluated digitally, samples were rated on digital ballots designed through the Qualtrics survey software (Qualtrics, Provo, UT). Before beginning each session, consumers were given verbal instructions by Texas Tech personnel about the ballot and the process of testing samples. Panels were conducted in a large classroom under fluorescent lighting with tables that were divided into individual consumer booths.

For each session, panelists were selected at random (10 per session) and were timed to determine the length of evaluation time for a sample from the time they received the sample to the time the evaluation of the sample was complete (i.e., pencil down or advanced to the next ballot on the tablet). Times were recorded by 2 individuals who were observing each session.

Statistical analysis

All data were analyzed using the GLIMMIX procedure (unless noted otherwise) of SAS (version 9.4; SAS Institute Inc., Cary, NC). Carcass and compositional data were analyzed with treatment as the fixed effect. Homogeneity of variance was tested for the ballot type treatment means for the consumer data using Levene’s Test in PROC GLM of SAS. From the analysis, the P values were 0.26, 0.07, 0.69, and 0.22 for tenderness, juiciness, flavor liking, and overall liking, respectively. Therefore, homogeneity of variance assumptions were met (P > 0.05). Consumer data were analyzed as a 2 × 4 factorial design using ballot format (paper, digital), treatment (SE-LL, SE-ST, CH-PM, PR-LL), and their interaction as fixed effects. Consumer nested within testing day was included as a random effect. Ballot completion time data were analyzed as a factorial design using ballot format (paper, digital), round (1–8), and their interaction as fixed effects. A subsequent analysis of consumer data was performed to investigate any interactive effects between ballot type and demographic traits (age, gender, consumption level, education, income, and heritage) and preferred ballot type. The interaction between ballot type and the aforementioned traits were all considered fixed effects. Treatment least-squares means were separated with the PDIFF option of SAS using a significance level of P < 0.05. A chi-square analysis was conducted to determine whether the distribution of responses into the 4 eating quality categories differed between paper and digital ballots (P < 0.05). Demographic data were summarized using PROC FREQ.

Carcass data and compositional analysis

Table 1 shows the differences in carcass traits between the 3 different quality grades used in this study. Quality grade influenced (P < 0.05) marbling, ossification, and pH of the beef carcasses. Fat thickness, ribeye area, and lean maturity were similar (P > 0.05) between quality grades. As expected, Prime had the most (P < 0.05) marbling followed by Choice and Select, with a significant difference between each grade where Prime > Choice > Select. Ossification scores were greater (P < 0.05) for Prime carcasses compared with Choice and Select carcasses, which were similar (P > 0.05). However, it should be noted that all carcasses were considered “A” maturity. Choice and Select carcasses had similar (P > 0.05) pH values in the longissimus muscle, but both had greater (P < 0.05) pH values than Prime carcasses. However, it is important to note that the average pH value for all carcasses was below 5.6. Despite statistical differences in pH between quality grades, this would likely not translate to relevant differences in eating quality, especially because muscles other than the longissimus were used for half of the treatments.

Results for the compositional analysis are shown in Table 2. As expected, treatment (USDA quality grade and muscle) influenced (P < 0.01) fat, moisture, and protein percentage. PR-LL had the greatest percentage of fat, followed by CH-PM, SE-LL, and SE-ST, with each treatment differing (P < 0.05). SE-LL had greater (P < 0.05) protein percentage than all other treatments except CH-PM; CH-PM had similar protein as SE-ST, and PR-LL had lower protein percentage than all other treatments. Moisture percentage was greatest (P < 0.05) in SE-ST, followed by SE-LL and CH-PM, which were similar (P > 0.05) but were greater (P < 0.05) than PR-LL, which had the lowest moisture percentage of all the treatments. Chemical percentages of the PR-LL were similar to previous percentages (Corbin et al., 2015; Legako et al., 2015; McKillip et al., 2018). The CH-PM and the SE-LL had similar chemical percentages to previously reported literature (Legako et al., 2015).

Demographic profile

Table 3 outlines the panelist demographics for this study. Most of the panelists (67.5%) were aged 20 to 49 years old. There were slightly more female than male participants (55.0% vs. 45.0%, respectively). Most participants were either employed full-time (79.6%) or were students (10.0%). A majority (55.3%) of panelists lived in a 2-adult household, with no children being the most common (46.4%) number of children in the household. Over half of consumers had an annual income ranging from $20,000 to $75,000. A vast majority (95.6%) of panelists were at least a high school or college graduate. Caucasian/white was the primary ethnic group followed closely by Hispanic. When asked about ballot preference between the digital ballot and the paper ballot, a clear majority (79.9%) preferred digital over paper. A large proportion (81.7%) of consumers ate beef multiple times a week (2–3 times or more). When asked about how they like their beef prepared, a majority (86.4%) of consumers preferred their beef cooked to medium-rare to a medium-well done degree of doneness.

Ballot type and treatment effects on palatability ratings

As shown in Table 4, ballot type and treatment influenced (P ≤ 0.02) tenderness, juiciness, flavor liking, overall liking, and MQ4. Despite significant differences for both ballot type and treatment, the magnitude of difference between ballot types, was generally much smaller than the magnitude of differences between muscle × quality grade treatments. Eating quality classification was affected by treatment (P < 0.01) but was not impacted (P > 0.05) by ballot type. No interactions between ballot and treatment were observed for any palatability traits (P > 0.05). Consumers scored tenderness, juiciness, flavor liking, and overall liking greater (P < 0.05) on paper ballots compared with digital ballots, regardless of treatment. Because MQ4 is a composite trait of 4 four traits, it followed a similar trend. The smallest margin between paper and digital ballots was observed for tenderness, where scores only differed by 1.8 points when traits were scored on 100-mm lines. However, juiciness, flavor liking, and overall liking all differed by 3.4 points in favor of paper ballots. When weightings were applied to determine the MQ4 score, the composite score differed by 2.9 points. Despite differences in all palatability traits, eating quality classification was similar between ballots. This is further illustrated in Figure 3, which shows the distribution of eating quality classification by ballot type. A chi-square analysis revealed that the frequency of distribution of the various categories did not differ between paper and digital ballots (P > 0.05). It is of particular note, however, that despite similar percentages in each category, there were instances when consumers would designate the same steak sample as a different category on the 2 ballot types. As an example, a consumer might indicate that Steak A was “unsatisfactory” on the digital ballot, but they would classify the corresponding Steak B as “good everyday quality” on the paper ballot. Specifically, from the 1,440 paired comparisons (4 comparisons per consumer), 51% were classified the same on both ballots, 23% were placed in a higher category on the digital ballot, and 26% were designated as a higher category on the paper ballot (data not shown in tabular form).

Figure 3.

Distribution of eating quality classification by ballot type (n = 360 consumers; 4 samples/ballot type). Chi-square analysis indicated the frequency of distribution of the 4 eating quality categories did not differ between paper and digital ballots (P = 0.79). A similar percentage appeared in each category.

Tenderness, overall liking, MQ4, and quality level were greatest (P < 0.05) for CH-PM compared with all other treatments, with a significant difference between the remaining treatments where CH-PM > PR-LL > SE-LL > SE-ST. Consumers scored juiciness greater (P < 0.05) in PR-LL compared with all other treatments, with a significant difference between the remaining treatments where PR-LL > CH-PM > SE-LL > SE-ST. Flavor liking was similar (P > 0.05) and greater (P < 0.05) for CH-PM and PR-LL than SE-LL, which was intermediate, and SE-ST had the lowest flavor liking scores.

Muscles that were considered support muscles (PR-LL, CH-PM, SE-LL) were scored as more tender than the locomotive muscle (SE-ST). The CH-PM were scored most tender, and the SE-ST were scored as least tender, which was consistent with reports by Rhee et al. (2004), who evaluated the palatability traits of 11 beef muscles. Juiciness scores were related to fat percentages, with consumers scoring the PR-LL as most juicy and scoring the SE-ST as least juicy. Overall liking mirrored that of tenderness scores; this finding was not surprising given the documented importance of tenderness to overall palatability (Boleman et al., 1997; Miller et al., 2001; O’Quinn et al., 2018).

Consumers were able to identify differences in treatments and rate accordingly, regardless of ballot. However, on average consumers scored samples higher on paper ballots compared with digital ballots. The magnitude of difference between ballot types, however, was generally much smaller than the magnitude of difference between cut treatments. The reason for the difference between the 2 ballots is unclear, as it was expected that samples would be scored similarly on both ballots. In part, the adequate sample size (n = 360 consumers) allows for easier detection of differences because it reduces the standard error, but the adequate sample size also improves the validity of the results because it increases power. It is possible that, in a smaller test sample (i.e., fewer consumer panelists), the difference in tenderness between ballot type could be inconsequential. Even so, despite what seems to be a relatively small difference (1.8- to 3.4-point difference scored on 100-mm line scales) could have significant bearing on a consumer’s final classification of overall eating quality. A 1-point difference on this scale can equate to a difference between “good everyday quality” and “premium quality.” It only takes a 1-point shift, but it depends where on the scale that 1-point shift occurs.

On the digital ballots, the marker is always initially placed at zero by default for each trait, and consumers must drag the marker to the desired score on the line. They cannot simply point to the precise spot on the line where they want to score the trait but must drag the marker to that point. Perhaps consumers become fatigued with this dragging exercise and just get to a point that is “close enough” in their mind. Conversely, on the paper ballot, the consumers simply mark the exact spot where they wanted to score a particular trait for a given sample. Essentially, the consumer could be suffering from what we have coined as “lazy finger” when they score on a digital ballot as opposed to the paper ballot.

It was also noted that, when comparing the 4 palatability traits on a paper ballot versus a digital ballot, tenderness scores had the lowest difference at 1.8 points. One reason could be that this was the first trait listed and scored on the ballot, and perhaps consumers were more critical of this trait and then relaxed as they observed the next 3 traits. However, there were no clear reasons as to why these discrepancies existed.

Ballot completion times

Tables 5 and 6 show that ballot type and ballot round affected (P < 0.01) time of sample evaluation. No interaction was observed between ballot type and round (P = 0.11). Consumers used more time to record responses (P < 0.05) on digital ballots compared with paper ballots regardless of ballot round. Consumers used the most time (P < 0.05) for ballot completion in round 1, followed by round 5, compared with all other rounds, regardless of ballot type. Consumers were likely taking longer to evaluate those samples as they were getting acclimated to the ballot. However, once the consumers became accustomed to the ballot they were using, consumers completed their assessments more rapidly for the remaining samples in that ballot block.

Demographic traits and ballot preference effects on palatability ratings

To determine whether demographic characteristics (age, gender, consumption, income, education, and heritage) or preferred ballot type influenced palatability scores, we evaluated the interactive effects of those traits with the actual ballot type. As seen in Table 7, preferred ballot influenced (P = 0.01) juiciness scores when comparing paper and digital ballots. Gender contributed to variation in flavor and overall liking scores (P < 0.01), and education level resulted in variation between ballot type for flavor liking scores (P < 0.01). Age, consumption level, income level, and heritage did not contribute to the variation between ballot type for any of the palatability traits (P > 0.05). When considering the demographic characteristics, we suspected that age could influence palatability ratings between ballot types, as younger generations might be more familiar with technology. However, according to Kakulla (2020), tablet adoption among adults aged 50+ years has risen from 30% in 2014 to 52% in 2019, actually surpassing tablet adoption among 18- to 49-year-olds (49%). As such, age is not one of the demographic characteristics that influenced scores between ballot types.

As previously noted in Table 3, when asked about ballot preference between the digital ballot and the paper ballot, a clear majority (79.9%) preferred digital over paper. Figure 4 illustrates the interaction between actual and preferred ballot type for juiciness scores. This was the only palatability trait for which this interaction was observed. When consumers were using a digital ballot for evaluation and preferred the digital ballot, juiciness scores were lower (P < 0.05) than when scored on a paper ballot with a preference for a paper ballot. However, within each tested ballot type, there were no differences in juiciness scores owing to the preferred ballot type (P > 0.05). Overall, there does not appear to be any consistent pattern or advantage in scores among all palatability traits when using the preferred ballot as indicated by the consumer, but we did want to investigate and report any differences that were observed.

Figure 5 illustrates the interaction between gender and ballot type for flavor liking (Figure 5a) and overall liking (Figure 5b). Regardless of the ballot type used, males scored flavor liking and overall liking greater (P < 0.05) than females. Additionally, when scoring samples on paper ballots, males scored traits greater than females on both paper and digital ballots.

Figure 4.

Influence of ballot preference on juiciness scores. Consumer (n = 360) juiciness recorded on anchored 100-mm line scale: 0 = very dry; 100 = very juicy. ^a,bBars without a common superscript differ (P < 0.05).

Finally, Figure 6 shows the variation in flavor liking scores due to education level across ballot type (P < 0.01). Participants with at least some college education similarly liked the flavor more than non–high school and high school graduates, regardless of ballot type. Participants within the 3 higher levels of education scored flavor liking similarly (P > 0.05), regardless of the ballot type. When assessing the potential influence of these demographic factors, there does not appear to be any consistent pattern or advantage in scores across all palatability traits, but we did want to investigate and report any differences that were observed.

Figure 5.

(a) Influence of gender on flavor liking scores. (b) Influence of gender on overall liking scores. Consumer (n = 360) flavor and overall liking recorded on anchored 100-mm line scale: 0 = dislike extremely of flavor and overall; 100 = like extremely of flavor and overall. ^a,bBars without a common superscript differ (P < 0.05).

Figure 6.

Influence of education level on flavor liking scores. Consumer (n = 360) flavor liking recorded on anchored 100-mm line scale: 0 = dislike flavor extremely; 100 = like flavor extremely. Bars without a common superscript differ (P < 0.05).