The outbreak of the COVID-19 pandemic is a substantial threat to the health of all populations worldwide, and old age is a robust risk factor for poor prognosis of COVID-19 infection. To reduce the fatality rate of COVID-19 infection, further understanding of elderly patients with COVID-19 is necessary. We aimed to investigate the prognostic factors in elderly patients with COVID-19. This was a multicenter and retrospective study. Overall, 340 elderly patients with COVID-19 were enrolled in 3 hospitals in Daegu, South Korea. Death and severe pneumonia requiring oxygen treatment were defined as poor clinical outcomes. Of the patients studied, 15% died and 35.2% were classified as having severe pneumonia. In binary logistic regression analysis, activities of daily living (ADL) impairment, fever during hospitalization, initial infiltration on chest radiograph, and initial increased C-reactive protein (CRP) were significantly associated with severe pneumonia (OR = 5.33, p < 0.001; OR = 3.2, p = 0.002; OR = 2.32, p = 0.044; and OR = 1.33, p < 0.001, respectively). ADL impairment, comorbidity, fever during hospitalization, and initial increased CRP were significantly associated with death (OR = 7.13, p < 0.001; OR = 3.28, p = 0.005; OR = 3.15, p = 0.032, and OR = 1.18, p < 0.001, respectively). ADL impairment, fever, and initial CRP were poor prognostic factors in elderly patients with COVID-19. Understanding these poor prognostic factors is necessary to control the disease in elderly patients.