CDC-Data-2025/attachments/JFP_Kim_restaurant_inspections_7-22_djvu.txt

1000 


Journal of Food Protection, Vol. 85, No. 7, 2022, Pages 1000-1007 
https://doi.org/10.43 15/JFP-22-007 


Published 2022 by the International Association for Food Protection 
This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) 


Research Paper 


Foodborne Outbreak Rates Associated with Restaurant Inspection 
Grading and Posting at the Point of Service: Evaluation Using 
National Foodborne Outbreak Surveillance Data 


THUY N. KIM@htps:/orcid.org/0000-0001-6470-7046,'* LAURA WILDEY,? BRIGETTE GLEASON,* JULIA BLESER,* 
MELANIE J. FIRESTONE@ hiips://orcid.org/0000-0003-2244-3729,! GINA BARE,? JESSE BLISS,* DANIEL DEWEY-MATTIA,? 
HARLAN STUEVEN,° LAURA BROWN,° DAVID DYJACK,? AnD CRAIG W. HEDBERG! 


!University of Minnesota School of Public Health, Division of Environmental Health Sciences, 420 Delaware Street S.E., MMC 807, Minneapolis, 
Minnesota 55455; *National Environmental Health Association, 720 South Colorado Boulevard, 1000N, Denver, Colorado 80246; *National Center for 
Emerging and Zoonotic Infectious Diseases, Centers for Disease Control and Prevention, 1600 Clifton Road, Atlanta, Georgia 30329; National Network 
of Public Health Institutes, 1300 Connecticut Avenue N.W., no. 150, Washington, DC 20036; *Dining Safety Alliance, 200 Union Boulevard, Suite 200, 
Lakewood, Colorado 80228; and °National Center for Environmental Health, Centers for Disease Control and Prevention, 4770 Buford Highway N.E., 

MS F58, Atlanta, Georgia 30341, USA 


MS 22-007: Received 11 January 2022/Accepted 14 February 2022/Published Online 17 February 2022 


ABSTRACT 


A previously conducted national survey of restaurant inspection programs associated the practice of disclosing inspection 
results to consumers at the restaurant point of service (POS) with fewer foodborne outbreaks. We used data from the national 
Foodborne Disease Outbreak Surveillance System (FDOSS) to assess the reproducibility of the survey results. Programs that 
participated in the survey accounted for approximately 23% of the single-state foodborne illness outbreaks in restaurant settings 
reported to FDOSS during 2016 to 2018. Agencies that disclosed inspection results at the POS reported fewer outbreaks (mean = 
0.29 outbreaks per 1,000 establishments) than those that disclosed results online (0.7) or not at all (1.0). Having any grading 
method for inspections was associated with fewer reported outbreaks than having no grading method. Agencies that used letter 
grades had the lowest numbers of outbreaks per 1,000 establishments. There was a positive association (correlation coefficient, 
R* = 0.29) between the mean number of foodborne illness complaints per 1,000 establishments, per the survey, and the mean 
number of restaurant outbreaks reported to FDOSS (R? = 0.29). This association was stronger for bacterial toxin-mediated 
outbreaks (R* = 0.35) than for norovirus (R? = 0.10) or Salmonella (R* = 0.01) outbreaks. Our cross-sectional study findings are 
consistent with previous observations that linked the practice of posting graded inspection results at the POS with reduced 
occurrence of foodborne illnesses and outbreaks associated with restaurants. Support for foodborne illness surveillance 
programs and food regulatory activities at local health agencies is foundational for food safety systems coordinated at state and 
federal levels. 


HIGHLIGHTS 


¢ Jurisdictions with point-of-service disclosure reported fewer outbreaks. 
* Grading used in inspections was associated with fewer outbreaks than no grading. 
¢ Foodborne illness complaints may lead to increased outbreak detection and reporting. 


Key words: Foodborne illness; Foodborne outbreak; Inspection results; Public disclosure, Restaurant inspection; Restaurant 
inspection grading 


It is estimated that known foodborne pathogens are incident in which two or more people become ill from the 
responsible for 9.4 million illnesses annually in the United same contaminated food or drink (J); sporadic cases are 


States (2, 19). Depending on the pathogen, <1 to 10% of — jJInesses that have not been identified to be part of an 
cases are known to be associated with a recognized outbreak 


(3). Nevertheless, outbreak investigations provide key 
information on the food, pathogens, and settings associated 
with foodborne illness. An outbreak is defined by the 
Centers for Disease Control and Prevention (CDC) as an 


outbreak. Restaurants are an important setting for both 

outbreak-associated and sporadic (non—outbreak-associated) 

foodborne illness in the United States (2, 1/3). The 

percentage of foodborne illness outbreaks attributed to 

restaurant settings increased from a mean of 41% for the 

* Author for correspondence. Tel: 612-503-9277; E-mail: period 1967 to 1997 (/4) to a mean of 61% for the period 
kim00977@umn.edu. 2009 to 2015 (8). 


2202 Ainr 6} Uo Jasn UOqUaAaLd pUe JOUOD eseasiq 40) s1ajUBD Aq JPd'Q001-2-S8-2606-P76 L/6S78Z0E/000 L/Z/Se/yPd-a|on1e/dy[/Woo'ssasdualje"ueIpuaW/:dyY Woy papeojuMog 


J. Food Prot., Vol. 85, No. 7 


In recognition of the important role that restaurants 
play in prevention of foodborne illness and outbreaks, 
studies have identified model practices for agencies that 
inspect restaurants for compliance with food safety 
regulations. Study findings suggest that disclosing inspec- 
tion results at the point of service (POS) (1.e., at the 
establishment) using some form of grading (letter grade, 
color, numerical score, emoji, etc.) is associated with 
improved public health outcomes (5, 9, 20, 23). The 
evidence gathered by these efforts suggests that such 
disclosure yielded improved inspection scores (5), improved 
sanitary conditions (23), decreased incidence of Salmonella 
infection (9), and decreased hospitalizations due to 
foodborne illness (20). The results of these studies strongly 
suggest that the actions of restaurant inspection programs 
play an important role in reducing foodborne illness 
transmitted in restaurant settings. 

In 2021, a national survey of restaurant inspection 
programs found that disclosure at the POS was associated 
with fewer foodborne illness outbreaks reported per 1,000 
licensed food establishments. Survey methods were previ- 
ously described (/5). Briefly, the survey was disseminated to 
a total of 790 restaurant inspection agencies at two times: 7 
January 2020 and 3 March 2020 (/5). A third dissemination 
of the same survey occurred on 2 November 2020. Although 
not included in the original study results, these data were 
included in the analysis for this study. The net total number 
of agencies responding to the survey was 165. Of these, 140 
respondents represented local agencies, whereas the remain- 
der represented state or territorial agencies (/5). 

This survey captured various restaurant inspection 
agency characteristics across the United States, including 
estimates of complaints received and use of methods of 
grading, inspection results disclosure, and inspection 
violation schemes. It also captured counts of foodborne 
illness outbreaks, sporadic illness cases, and foodborne 
illness complaints. Survey recipients represented inspection 
agencies that disclosed inspection results online and those 
enrolled in the U.S. Food and Drug Administration (FDA) 
Voluntary National Retail Food Regulatory Program 
Standards program (Retail Program Standards). This 
program helps food regulatory programs meet the widely 
recognized Voluntary National Retail Food Regulatory 
Program Standards (2/). The FDA Food Code is a model 
set of science-based, comprehensive food safety guidelines 
that provides the technical and legal basis for local, state, 
tribal, and federal food codes that regulate retail food 
service in the United States (22). 

A limitation of the survey-reported data was the lack of 
important details on the etiologic agent (e.g., bacterial or 
viral pathogen) and setting of these outbreaks (/5). We 
sought to address these gaps by using data routinely 
reported by state public health agencies to the CDC through 
the Foodborne Disease Outbreak Surveillance System 
(FDOSS). FDOSS is a national, passive surveillance system 
that collects information on enteric and nonenteric food- 
borne outbreaks, including information on the number of 
cases, case outcomes, dates of illness onset, implicated 
foods, and locations of food preparation (1). The objective 
of our present study was to use FDOSS outbreak data to 


EVALUATION OF RESTAURANT GRADING SURVEY RESULTS 1001 


compare the number of outbreaks per 1,000 licensed 
restaurants by restaurant inspection grading and disclosure 
practices conducted by agencies responding to the initial 
survey (15). 


MATERIALS AND METHODS 


We used results from the previously conducted national 
survey of regulatory restaurant inspection agencies at state, 
county, city, district, and territorial levels as a baseline for this 
study (15). We limited analyses to local agencies representing city, 
county, or district jurisdictions (7 = 140), hereafter referred to as 
“agencies.” The decision to focus on local agencies is supported 
by the tendency of restaurant inspection programs to operate at the 
local government level (/5). The agencies were drawn from 34 
states representing all regions of the country (median = three 
agencies per state, range = | to 14). This current study used the 
following data from the original survey: jurisdiction of the survey 
respondents, number of licensed restaurants, number of com- 
plaints received from 2016 to 2018, method of inspection grading, 
and method of public disclosure of inspection results. 

As with inspection practices, inspection terminology can vary 
by agency. We defined public disclosure as the act of voluntarily 
and preemptively publicizing some or all inspection data to the 
public (e.g., posting at the restaurant or online). This study also 
defined grading method as the act of applying an ordinal ranking 
system to inspection results (e.g., numerical scores or letter 
grades). Disclosure at the POS is inclusive of any type of display 
of inspection results on the restaurant premises, regardless of font 
size or location. Complaints are reports to public health of possible 
foodborne illness from the public, including individuals or groups 
of individuals (7). 

We obtained foodborne outbreak data for our analysis from 
FDOSS; data also contained associated details about etiology and 
food preparation location. We applied the following inclusion 
criteria to the FDOSS data extracted on 18 November 2019: the 
primary mode of transmission was foodborne; the outbreak report 
was finalized; date of first illness was between | January 2016 and 
31 December 2018; the number of estimated primary illnesses was 
greater than one; the exposure location was within the jurisdiction 
of an agency that participated in our survey; and the location 
where food was prepared was a restaurant setting—including sit- 
down dining, buffet, fast food, or other or unknown restaurant 
type. 

We linked the FDOSS data to the survey data by jurisdiction, 
identified by the reporting agency. An outbreak was attributed to a 
regulatory agency if the agency’s jurisdiction was listed in FDOSS 
as the location in which the exposure occurred. Outbreaks in 
which exposure occurred in multiple counties were assigned to 
agencies based on the listed exposure locations. If a multicounty 
outbreak had exposure locations in jurisdictions for multiple 
agencies, each outbreak was counted once for each agency. 
Multistate outbreaks were excluded from analysis. Some counties 
contain city agencies that conduct inspections independently of the 
county agency. These incidences were identified by comparing the 
survey-reported population served by the county agency with the 
U.S. Census Bureau estimates of population for the jurisdiction. 
Using this method, city-level exposure data were used to identify 
and assign outbreak counts to the appropriate agency for four 
outbreaks. Outbreaks for which multiple pathogens were identified 
were counted only once in the outbreak total but were counted for 
each pathogen for pathogen-specific analyses. 

We grouped FDOSS restaurant outbreaks by etiology. 
Outbreaks in FDOSS with the suspected etiology of “other- 


7202 Ainr 6} UO JasN UOHUBAaIq PUB [OJUOD aseasiq JO} s19}UB_ Aq Jpd'Q001-2-G8-2606-P 6 | /6S78Z0€/0001/2/S84Pd-a}oI12/dj[/Woo'ssaidualje"uelpiewu//:dy}Y Woy pepeo|umMoq 


1002 KIM ET AL. 


TABLE 1. Etiological distribution of outbreaks in restaurant 
settings reported to the FDOSS for agencies participating in the 
restaurant grading project survey compared with all other 
jurisdictions, 2016 to 2018° 


J. Food Prot., Vol. 85, No. 7 


TABLE 2. Number and mean annual rate of outbreaks in 
restaurant settings reported to the FDOSS by disclosure methods 
and grading methods for agencies participating in the restaurant 
grading project survey, 2016 to 2018" 


Restaurant outbreaks for Restaurant outbreaks in 
survey group agencies, all other jurisdictions, 
n (%) (n = 381) n (%) (n = 1,257) 


Bacterial toxin 36 (9) 109 (9) 
Bacillus 11 23 
Clostridium 10 44 
Staphylococcus 6 37 
Unspecified 9 5 

Campylobacter 10 (3) 30 (2) 

Ciguatoxin 0 (0) 3 (0.2) 

Cryptosporidium 1 (0.3) 2 (0.2) 

Cyclospora 4 (1) 23 (2) 

Escherichia 6 (2) 18 (1) 

Hepatitis 1 (0.3) 9 (0.7) 

Norovirus 177 (46)? 489 (39) 

Salmonella 48 (13) 125 (10) 

Sapovirus 2 (0.5) 5 (0.4) 

Scombroid toxin 4 (1) 27 (2) 

Shigella 0 (0) 5 (0.4) 

Vibrio 39 (10)? 18 (1) 

Multiple etiologies 6 (2) 22 (2) 

Unknown etiology 47 (12)? 372 (30) 


“ FDOSS, Foodborne Disease Outbreak Surveillance System. 

° Proportion of outbreaks significantly different between survey 
group and all other jurisdictions. Norovirus (RR = 1.14; 95% CI 
= 1.01, 1.29) and Vibrio (RR = 2.94; 95% CI = 1.99, 4.35) were 
more frequently reported by agencies in the survey group, 
whereas unknown etiologies (RR = 0.48; 95% CI = 0.36, 0.66) 
were less frequently reported. 


bacterium” were reviewed; most were attributed to an unspecified 
bacterial toxin based on details provided by the reporting agency. 
These counts were then combined with Bacillus cereus, Clostrid- 
ium perfringens, and Staphylococcus aureus and collectively 
referred to as “bacterial toxin—mediated.” The proportions of 
outbreaks by etiology were compared between agencies that 
participated in the restaurant grading project survey (survey 
group) and all other agencies reporting to FDOSS. This 
comparison between the two groups enumerated the contributions 
of the survey group in the context of the overall national outbreak 
surveillance data for the study period. 

We calculated mean and median values for rates to identify 
trends in outcomes based on each category of grading method, 
disclosure method, and inspection violation scheme. Mean rates 
for the survey group and all other agencies were compared using t¢ 
tests, and P values were reported based on unequal variance 
assumptions. The level of significance was set at a = 0.05. 
Analysis was conducted using SAS 9.4 (SAS Institute, Cary, NC). 
Scatterplots and R* values were obtained using Microsoft Excel 
(Microsoft, Redmond, WA) to assess the relationship between the 
mean number of complaints reported and the mean number of 
outbreaks by etiology. 


RESULTS 


There were 2,608 single-state foodborne outbreaks 
reported to FDOSS during 2016 to 2018, with 1,638 
attributed to food prepared in a restaurant setting. Of these, 


Outbreaks per 


No. of 1,000 restaurants 
No. of | outbreaks in. }=——W—————_ 
agencies restaurants Mean(SD) Median 
Disclosure methods 
Point of service 8 24 0.29 (0.2) 0.3 
Online 36 226 0.70 (0.7) 0.4 
None 11 72 1.0 (1.0) 0.5 
Grading methods 
Letter grade 42 310 0.57 (0.7) 0.3 
Numerical score 19 148 0.69 (0.7) 0.4 
None 12 89 0.96 (0.9) 0.7 
Other 16 138 0.76 (0.8) 0.4 


“ FDOSS, Foodborne Disease Outbreak Surveillance System. 


outbreaks in the survey group jurisdictions accounted for 
23% (n= 381), and all other jurisdictions accounted for the 
remaining 77% (n = 1,257). 


Outbreak numbers and etiology by group. The 
proportion of outbreaks in restaurant settings was signifi- 
cantly higher among agencies in the survey group compared 
with all other agencies (relative risk [RR] = 1.10, 95% 
confidence interval [CI] = 1.03, 1.17; Table 1). The most 
common etiologies reported to FDOSS in restaurant settings 
from the survey group were norovirus (177 [46%] 
outbreaks), Salmonella (48 [13%] outbreaks), Vibrio spp. 
(39 [10%] outbreaks), and bacterial toxin—mediated (36 
[9%] outbreaks) (Table 1). The etiology was unknown for 
47 outbreaks (12%) (Table 1). The proportions of restaurant 
setting outbreaks attributed to norovirus (RR = 1.14; 95% 
CI= 1.01, 1.29) and Vibrio spp. (RR = 2.94; 95% CI= 1.99, 
4.35) were significantly higher among the survey group, 
whereas the proportion of unknown outbreaks was signif- 
icantly lower (RR = 0.48; 95% CI = 0.36, 0.64) among the 
survey group compared with all other agencies. 


Outbreak rates by inspection disclosure and grad- 
ing methods. There was a pattern of lower mean annual 
number of outbreaks per 1,000 licensed restaurants for 
agencies in the survey group that disclosed inspection 
results at the POS compared with agencies that either 
disclosed online or did not disclose (means: 0.29 POS 
versus 0.70 online, 1.0 did not disclose) (Table 2). A similar 
pattern was also seen for inspection grading methods; 
agencies with any form of grading method had a lower 
mean annual number of outbreaks per 1,000 licensed 
restaurants than agencies with no grading method (means: 
0.57 letter grade, 0.69 numerical score versus 0.96 no 
grading method). 


Comparison of POS and online disclosure methods. 
Inspection disclosure methods varied across agencies within 
states. For example, in 10 states that had six or more 


7202 Ainr 6} UO Jasn UOHUBAaIq PUB [OUD aseasiq JO} Ss19}UB_ Aq Jpd'Q00}-2-G8-2606-P 6 | /6S78Z0€/0001/2/S84Pd-a}oy12/dj[/Woo'ssaidualje"uelpiewu//:dy}Y Woy pepeo|umMoq 


J. Food Prot., Vol. 85, No. 7 


TABLE 3. Mean annual rate of outbreaks in restaurant settings 
reported to the FDOSS by POS disclosure versus online without 
POS disclosure for agencies participating in the restaurant 
grading project survey, 2016 to 2018* 


Outbreaks per 1,000 restaurants (n = 202) 


Mean (SD) Median P value? 
Disclosure method 0.002 
POS 0.3 (0.2) 0.3 
Online without POS 0.8 (0.7) 0.5 


“FDOSS, Foodborne Disease Outbreak Surveillance System; 
POS, point of service. 
° P value for comparison of means. 


agencies included in the survey, in only two states did all of 
the agencies in the state use the same practices for 
disclosing inspection results. Of the 28 agencies that 
disclosed at the POS according to the survey, 24 (86%) 
also disclosed online. However, there were fewer outbreaks 
reported by agencies that disclosed at the POS, compared 
with agencies that disclosed online without POS disclosure 
(0.3 POS versus 0.8 online, P = 0.002) (Table 3). 


Complaint rates by restaurant outbreak etiologies 
reported to FDOSS. There was a positive association 
(correlation coefficient, R*? = 0.29) between the mean 
number of complaints per 1,000 licensed restaurants per 
year reported to FDOSS and the mean number of restaurant 
outbreaks per year reported to FDOSS (R? = 0.29; Fig. 1). 
When reported restaurant outbreaks were stratified by 
etiology, there was a positive association between the mean 
number of complaints and the mean number of norovirus 
outbreaks in restaurants reported to FDOSS (R? = 0.10; Fig. 
2), and a positive association for bacterial toxin—mediated 
restaurant outbreaks (R* = 0.35; F ig. 3). Conversely, there 
was no meaningful trend for Salmonella (R* = 0.01; Fig. 4), 
suggesting that Salmonella outbreaks are not associated 
with foodborne illness complaints. 


DISCUSSION 


Relevance to practice. Our findings were consistent 
with previous survey (/5) results that showed that the 
disclosure of graded inspection results at the POS was 
associated with fewer outbreaks reported to FDOSS. These 
results provide further support for recommendations (/5) to 
post graded restaurant inspection results at the POS by 
demonstrating that agencies that used some grading system 
had lower mean numbers of FDOSS restaurant outbreaks 
per 1,000 establishments than did agencies that did not post 
graded inspection results. Agencies that used letter grades 
had the lowest mean and median numbers of FDOSS 
restaurant outbreaks per 1,000 licensed restaurants, al- 
though the study had limited power to distinguish among 
the grading methods. 

Restaurant inspections are a measure of how well a 
restaurant adheres to food safety guidelines that prevent 
foodborne illness. The finding that posting graded inspec- 


EVALUATION OF RESTAURANT GRADING SURVEY RESULTS 1003 


tion results at the POS was associated with fewer outbreaks 
occurring in restaurants based on FDOSS data is consistent 
with hypotheses that consumers use this information to 
guide their dining decisions (10, //, 23). Because access to 
this information is important to consumers, a favorable 
score may attract more consumers, whereas a less favorable 
score may provide food operators with additional incentive 
to improve their food safety performance. Disclosure of 
inspection results at the POS allows this measure of food 
safety performance to be readily available and interpretable 
to consumers at a location where many dining decisions are 
made. 


Distribution of outbreaks. The higher proportion of 
outbreaks reported by the survey group suggests that these 
agencies were more likely to report restaurant-associated 
outbreaks and were more likely to report outbreaks due to 
norovirus but were less likely to report outbreaks of 
unknown etiology than all other agencies. This suggests 
that agencies in the survey group were better at determining 
the outbreak setting and etiology of the outbreaks they 
investigated. The relative effectiveness of agencies in the 
survey group to detect and investigate outbreaks adds 
further support for the credibility of findings within this 
group regarding differences in outbreak reporting based on 
inspection grading and disclosure practices. 


Usefulness of consumer complaints. In addition to 
our findings regarding inspection reporting, the results of 
this study support the importance of agencies having a 
mechanism to receive foodborne illness complaints. Our 
finding of a positive correlation between the number of 
complaints received per 1,000 licensed restaurants and the 
number of restaurant outbreaks reported to FDOSS means 
that the ability to receive and investigate foodborne illness 
complaints may be an important predictor of the ability of 
the agency to detect foodborne outbreaks. In particular, 
the positive associations between complaints and restau- 
rant outbreaks of bacterial toxin—mediated and norovirus 
outbreaks reflects the reliance on complaint-based sur- 
veillance to detect these outbreaks with short incubation 
periods. It is primarily through complaint-based surveil- 
lance systems that these types of outbreaks, and others 
with short incubation periods, are detected by public 
health agencies, thereby underscoring the need for 
continued complaint-based surveillance systems (7). In 
contrast, Salmonella-associated outbreaks are detected 
primarily through pathogen-specific surveillance; this 
supports the finding of no effect between the occurrence 
of complaints and outbreaks of Salmonella, which has a 
longer incubation period than toxin-mediated pathogens 
(4, 17). 

Complaint-based surveillance is one of the two main 
methods of foodborne outbreak detection in the United 
States (7). Although this study does not assume that having 
the ability to receive complaints is indicative of the 
existence of a complaint system, it is notable that 81% of 
local health departments have a complaint-based surveil- 
lance system (/6) and approximately 75% of all foodborne 


7202 Ainr 6} UO JASN UOHUBAaIq PUB [OUD aseasiq JO} s19}UB_ Aq Jpd'Q001-2-G8-2606-P 6 | /6S78Z0€/0001/2/S84Pd-a}oy12/dj[/Woo'ssaidualje"uelpLew//:dy}Y Woy pepeo|umoq 


1004 KIM ET AL. 


35 
_—_ 
3) 
Qa 
n 3 
bol) 
.— 
£ 
g 
w 2.5 

@ 

o © 2 
ae ® 
-“ 
cw 
£¢ - 
x 8 is > oo Vero 
ve @ ave 9** 
2" o | wc 
3 1 | ®e ———_--_ a 8 
6 = ae 
= jj x ggeast 
SC ioe Letts 
© 0.5 2 % e 
= 2 @e* e e 

0 &e ” 

0 50 100 


J. Food Prot., Vol. 85, No. 7 


150 200 250 


Mean no. complaints per 1,000 restaurants 


FIGURE 1. Mean annual number of outbreaks in restaurant settings per 1,000 restaurants reported to the Foodborne Disease Outbreak 
Surveillance System (FDOSS) and the mean number of survey-reported complaints per 1,000 restaurants per year for agencies (@) 


participating in the restaurant grading project survey, 2016 to 2018. 


outbreaks are detected through complaint systems (6). The 
usefulness of complaints to detect outbreaks has been 
demonstrated by multiple studies (/2, /6-/8, 25). A 
survey of local health departments identified a positive 
correlation between outbreak and complaint rates per 
population served; agencies that received more complaints 
detected more outbreaks (/6). An analysis of the Florida 


2.5 


1:5 


Mean no. norovirus outbreaks in restuarant 
settings per 1,000 restaurants 


0 50 100 


Department of Health’s complaint and outbreak reporting 
system found that 56% of foodborne outbreaks were 
identified through complaints (/8). Likewise, complaints 
led to detection of 80% of foodborne outbreaks in Rhode 
Island (25) and 79% of confirmed foodborne outbreaks in 
Minnesota (/7). Not only can complaints be used to detect 
outbreaks, but they can also help identify specific 


R? = 0.1033 


vooure* 


150 200 250 


Mean no. complaints per 1,000 restaurants 


FIGURE 2. Mean annual number of norovirus outbreaks in restaurant settings per 1,000 restaurants reported to the Foodborne Disease 
Outbreak Surveillance System (FDOSS) and the mean number of survey-reported complaints per 1,000 restaurants per year for agencies 
(®) participating in the restaurant grading project survey, 2016 to 2018. 


7202 Ainr 6} UO JasN UOHUBASIq PUB [OUD aseasiq JO} S19}UB_ Aq Jpd'Q001-2-G8-2606-P 6 | /6S78Z0€/0001/2/S84Pd-a}oy12/dj[/Woo'ssaidualje"ueIpewu//:dy}Y Woy pepeojumMoq 


J. Food Prot., Vol. 85, No. 7 


0.9 
© 
2 

408 

2 = e 
28 
ap 
BE og 
se” 
oo 
vor 

0.5 
Es 
€a 
8 % 04 AAA 
coe ee er 
£3 = @ oe 
§ 8 03 wa 
0 5 eo oo 
2] 5 02 eee 
Om | eer* 
c+ : 
£201 

) 

) 50 100 


EVALUATION OF RESTAURANT GRADING SURVEY RESULTS 1005 


ry deg 


150 200 250 


Mean no. complaints per 1,000 restaurants 


FIGURE 3. Mean annual number of bacterial toxin—mediated outbreaks in restaurant settings per 1,000 restaurants reported to the 
Foodborne Disease Outbreak Surveillance System (FDOSS) and the mean number of survey-reported complaints per 1,000 restaurants 
per year for agencies (@) participating in the restaurant grading project survey, 2016 to 2018. 


indicators of risk. For example, a study of consumer 
complaints in Washington, DC, found that complaints were 
significantly correlated with cited inspection violations of 
improper holding temperatures and contaminated equip- 
ment (/2). These studies highlight the usefulness of 
consumer complaints and underscore the need for 


0.9 
w 
pore) 
£ 
5 08 
wu 
uw 
eS 
Ss 0.7 
I 
. 
8 2 06 
£f 

= 
2805 
5S } = 

Oo 
3 =) 0.4 ® 
sa : 
— — 
2 203 
S ® 
& e ° ° e 
A 02 |- 
ee a ee 
‘aa ee a ae 
: © 
Ss 01 ° eo @ 
> ° °° 

0 
; - 100 


complaint-based surveillance in foodborne outbreak de- 
tection for pathogens with short incubation periods. 


Strengths and limitations. Strengths of this study 


include the use of national data (FDOSS) through a well- 
established outbreak surveillance system to validate out- 


150 200 250 


Mean no. complaints per 1,000 restaurants 


FIGURE 4. Mean annual number of Salmonella outbreaks in restaurant settings per 1,000 restaurants reported to the Foodborne Disease 
Outbreak Surveillance System (FDOSS) and the mean number of survey-reported complaints per 1,000 restaurants per year for agencies 
(®@) participating in the restaurant grading project survey, 2016 to 2018. 


7202 Ainr 6} UO JasN UOHUBAaIq PUB [OUD aseasiq JO) S19]UB_ Aq Jpd'Q00}-2-G8-2606- P76 | /6S78Z0€/0001/2/S84Pd-a}oI12/dj[/Woo'ssaidualje"uelpiew//:dyY Woy pepeo|umMoq 


1006 KIM ET AL. 


break counts reported via survey. The surveyed agencies 
accounted for nearly one-quarter of restaurant setting 
outbreaks reported to FDOSS. This study did not adjust 
for potential confounders such as jurisdiction size, geo- 
graphic region, state-level food program inspection and 
reporting requirements, funding, and staffing of the 
inspection agency. These factors may have affected an 
agency’s ability to investigate consumer complaints, detect 
outbreaks, and subsequently report them to FDOSS. 
However, there did not appear to be an association between 
jurisdiction size and reported outbreak rate (R? < 0.01). In 
most states there was considerable variation among 
agencies with respect to restaurant grading and disclosure 
practices. As noted above, the higher proportion of 
outbreaks attributable to norovirus and lower proportion 
of outbreaks with unknown etiology among the surveyed 
agencies may reflect that they had a better capacity to 
investigate foodborne illness outbreaks than did agencies 
that did not respond to the survey. 

There are inherent limitations to the use of FDOSS 
data. First, because the FDOSS database is dynamic, 
agencies are permitted to submit, update, or delete reports 
at any time. Data used in the analysis for this study were 
pulled at one point in time; therefore, previous and future 
analyses using FDOSS data extracted in a similar fashion 
may produce slightly different results. Second, outbreak 
counts are reflective of those that were able to be detected. 
Not all outbreaks are identified by public health agencies, 
and as noted previously, the majority of foodborne illnesses 
are not a part of recognized outbreaks. It is unknown how 
well the etiologies and locations implicated in outbreaks 
reflect those of sporadic foodborne illnesses, i.e., illnesses 
not associated with outbreaks. 

Limitations related to using the survey methods 
described include the use of a convenience sample of 
agencies that were enrolled in the Retail Program Standards 
program, which limited the representativeness of these 
results to enrollees. Agencies that enroll in this voluntary 
program may differ from those that choose not to enroll; 
however, because most (98%) of the agencies participating 
in the study were participants in the Retail Program 
Standards program, participation in the Retail Program 
Standards program is unlikely to bias the findings with 
respect to the main effect measures. Due to the inquiry of 
data from multiple time points (survey results during 2019 
to 2020 and outbreak data during 2016 to 2018), survey 
responses may not be truly reflective of practices during the 
time the outbreaks occurred. 

A consumer’s propensity to file a foodborne illness 
complaint involving a restaurant is influenced by a variety 
of factors, including poverty status. Unpublished work 
studying the association of foodborne illness and inspection 
report data in Hennepin County, MN, found that census 
blocks with high poverty levels were associated with fewer 
foodborne illness complaints (OR = 0.31; 95% CI: 0.13 to 
0.73) (24). Nevertheless, underlying poverty status in the 
survey group was not deemed an important confounder in 
our analysis. Because the ability to detect outbreaks in 
restaurants heavily relies on complaint-based surveillance, 


J. Food Prot., Vol. 85, No. 7 


any biasing effect that poverty status may have on consumer 
propensity to file a complaint would also be reflected in the 
number of outbreaks. There are also different kinds of 
complaints that can be received about a restaurant: those 
that relate specifically to foodborne illness and those that 
relate to specific good retail practice violations. Although 
our study did not differentiate between the two types, it is 
plausible that the occurrence of violations may be an 
indicator of food safety practices that could lead to 
foodborne illness in the future. 

Although this was a cross-sectional study that cannot 
control for the effects of policy changes within inspection 
programs, our associations are consistent with studies in 
Los Angeles County (20) and New York City (9) that 
demonstrated reductions in the occurrence of foodborne 
illnesses after implementation of posting of inspection 
grades at the POS. This study assessed the impact of the 
presence of disclosure at the POS, rather than the specific 
manners (e.g., location, font size) by which it occurred. If 
additional evidence were needed to encourage local food 
regulatory agencies to adopt a practice of grading and 
posting inspection results at the POS, then a randomized 
community-control trial could be considered as a next step. 


Policy implications. Surveys of public health agencies 
that are validated by national surveillance data can be 
powerful tools to identify model practices that contribute to 
prevention of foodborne outbreaks and illnesses. Particu- 
larly, our cross-sectional study findings are consistent with 
previous observations that linked the practice of posting 
graded inspection results at the POS with reduced 
occurrence of foodborne illnesses and outbreaks associated 
with restaurants. Other food regulatory practices, such as 
maintaining a robust foodborne illness complaint system, 
may improve foodborne illness surveillance, outbreak 
detection, and response. Improving foodborne illness and 
outbreak surveillance is a prerequisite for improving and 
measuring the effectiveness of our food safety systems. 
Support for foodborne illness surveillance programs and 
food regulatory activities at local health agencies is 
foundational for food safety systems coordinated at state 
and federal levels. 


ACKNOWLEDGMENTS 


This study was funded through cooperative agreement 
6NU380T000300 between the Centers for Disease Control and Prevention 
(CDC) and the National Environmental Health Association (NEHA). The 
findings and conclusions are solely the responsibility of the authors and do 
not necessarily represent the official views of CDC and NEHA. Additional 
support was provided by the Global Food Ventures MnDRIVE Fellowship 
at the University of Minnesota. 


REFERENCES 


1. Centers for Disease Control and Prevention. 2018. Foodborne 
Disease Outbreak Surveillance System. U.S. Department of Health 
and Human Services, Atlanta. Available at: https://www.cdc.gov/ 
fdoss/index.html. Accessed 30 August 2021. 

2. Centers for Disease Control and Prevention. 2019. Surveillance for 
foodborne disease outbreaks, United States, 2017: annual report. U.S. 
Department of Health and Human Services, Centers for Disease 
Control and Prevention, Atlanta. Available at: https://www.cdc.gov/ 


7202 Ainr 6} UO Jasn UOHUBAaIq PUB [OUOD aseasiq JO} S19}UB_ Aq Jpd'Q00}-2-G8-2606-P 76 | /6S78Z0€/0001/2/S84Pd-a}oI12/dj[/Woo'ssaidualje"uelpieww//:dy}y Woy pepeo|umMoq 


J. Food Prot., Vol. 85, No. 7 


10. 


11. 


12. 


13. 


14. 


fdoss/pdf/2017_FoodBorneOutbreaks_508.pdf. Accessed 30 Decem- 
ber 2020. 

Centers for Disease Control and Prevention. 2021. FoodNet Fast: 
pathogen surveillance tool. U.S. Department of Health and Human 
Services, Atlanta. Available at: http://wwn.cdc.gov/foodnetfast. 
Accessed 11 August 2021. 

Chai, S. J., W. Gu, K. A. O’Connor, L. C. Richardson, and R. V. 
Tauxe. 2019. Incubation periods of enteric illnesses in foodborne 
outbreaks, United States, 1998-2013. Epidemiol. Infect. 147:e285— 
e292. 

Choi, J., and R. Scharff. 2017. Effect of a publicly accessible 
disclosure system on food safety inspection scores in retail and food 
service establishments. J. Food Prot. 80:1188-1192. 

Council to Improve Foodborne Outbreak Response. 2013. Develop- 
ment of target ranges for selected performance measures in the 
CIFOR guidelines. Council of State and Territorial Epidemiologists, 
Atlanta. 

Council to Improve Foodborne Outbreak Response. 2019. Foodborne 
illness surveillance and outbreak detection, p. 67-85. Jn Guidelines 
for foodborne disease outbreak response, 3rd ed. Council of State and 
Territorial Epidemiologists, Atlanta. 

Dewey-Mattia, D., K. Manikonda, A. J. Hall, M. E. Wise, and S. J. 
Crowe. 2018. Surveillance for foodborne disease outbreaks—United 
States, 2009-2015. Morb. Mortal. Wkly. Rep. Surveill. 67:1—-11. 
Firestone, M. J., and C. W. Hedberg. 2018. Restaurant inspection 
letter grades and Salmonella infections, New York, New York, USA. 
Emerg. Infect. Dis. 24:2164—-2168. 

Firestone, M. J., and C. W. Hedberg. 2020. Consumer interest and 
preferred formats for disclosure of restaurant inspection results, 
Minnesota 2019. J. Food Prot. 83:4715-4721. 

Fung, A., M. Graham, and D. Weil. 2007. Full disclosure: the perils 
and promise of transparency. Cambridge University Press, Cam- 
bridge, UK. 

Jemaneh, T. A., M. Minelli, A. Farinde, and E. Paluch. 2018. 
Relationship between priority violations, foodborne illness, and 
patron complaints in Washington, DC, restaurants (2013-2015). J. 
Environ. Health 80:14-19. 

Jones, T. F., and F. J. Angulo. 2006. Eating in restaurants: a risk 
factor for foodborne disease? Clin. Infect. Dis. 43:1324—1328. 
Jones, T. F., and J. Yackley. 2018. Foodborne disease outbreaks in the 
United States: a historical overview. Foodborne Pathog. Dis. 15:11— 
15. 


EVALUATION OF RESTAURANT GRADING SURVEY RESULTS 


15. 


16. 


17. 


18. 


19. 


20. 


21, 


22. 


23. 


24. 


25. 


1007 


Kim, T. N., M. J. Firestone, N. DeJarnett, L. Wildey, J. C. Bliss, D. T. 
Dyjack, J. Edwards, H. Stueven, and C. W. Hedberg. 2021. 
Disclosing inspection results at point-of-service: affect of character- 
istics of food establishment inspection programs on foodborne illness 
outcomes. J. Environ. Health 83:8-13. 

Li, J.. G. H. Shah, and C. Hedberg. 2011. Complaint-based 
surveillance for foodborne illness in the United States: a survey of 
local health departments. J. Food Prot. 74:432-437. 

Li, J., K. Smith, D. Kaehler, K. Everstine, J. Rounds, and C. 
Hedberg. 2010. Evaluation of a statewide foodborne illness 
complaint surveillance system in Minnesota, 2000 through 2006. J. 
Food Prot. 73:2059-2064. 

Li, X., A. C. Sapp, N. Singh, L. Matthias, C. Bailey, J. A. DeMent, 
and A. H. Havelaar. 2020. Detecting foodborne disease outbreaks in 
Florida through consumer complaints. J. Food Prot. 83:1877-1888. 
Scallan, E., R. M. Hoekstra, F. J. Angulo, R. V. Tauxe, M. A. 
Widdowson, S. Roy, J. L. Jones, and P. M. Griffin. 2011. Foodborne 
illness acquired in the United States—major pathogens. Emerg. 
Infect. Dis. 17:7. 

Simon, P. A., P. Leslie, G. Run, G. Z. Jin, R. Reporter, A. Aguirre, 
and J. E. Fielding. 2005. Impact of restaurant hygiene grade cards on 
foodborne-disease hospitalizations in Los Angeles County. J. 
Environ. Health 67:32-36. 

U.S. Food and Drug Administration. 2019. Voluntary national retail 
food regulatory program standards—November 2019. Available at: 
https://www.fda.gov/food/voluntary-national-retail-food-regulatory- 
program-standards/voluntary-national-retail-food-regulatory- 
program-standards-november-2019. Accessed 26 July 2021. 

U.S. Food and Drug Administration. 2019. FDA Food Code. 
Available at: https://www.fda.gov/food/retail-food-protection/fda- 
food-code. Accessed 26 July 2021. 

Wong, M. R., W. McKelvey, K. Ito, C. Schiff, J. B. Jacobson, and D. 
Kass. 2015. Impact of a letter-grade program on restaurant sanitary 
conditions and diner behavior in New York City. Am. J. Public 
Health 105:e8 1-e87. 

Young, H. (University of Minnesota). 2019. Personal communica- 
tion. 

Yousaf, A., B. Viveiros, and G. Caron. 2019. Rhode Island 
Department of Health foodborne illness complaint system: a 
descriptive and performance analysis. J. Food Prot. 82:1568-1574. 


7202 Ainr 6} UO JasN UOHUBAaIq PUB [OUOD aseasiq JO) S19}UB_ Aq Jpd'Q00-2-G8-2606-P 6 | /6S78Z0€/0001/2/S84Pd-a}oy12/dj[/Woo'ssaidualje"uelpHewu//:dy}Y Woy pepeo|umMoq