The use of non-invasive temperature testing methods like temporal artery thermometers (TATs) is growing exponentially in the face of the ongoing COVID-19 pandemic. We performed a retrospective analysis of over 1.8 million emergency department electronic health records to identify assess the performance of TAT measurement using patients with near-contemporaneous temperature measurements taken via rectal or oral approaches. Using over 17,000 matched measurements, we show poor fever sensitivity using TAT. We show that sensitivity is significantly improved by lowering the fever threshold and describe limits of agreement between methods of measurement. Our findings suggest that private, public, and healthcare delivery organizations may need to reconsider how we perform high-volume screening during this time of crisis and has implications for return-to-work protocols.