Coronavirus disease 2019 (COVID-19) displays high clinical variability but the parameters that determine disease severity are still unclear. Pre-existing T cell memory has been hypothesized as a protective mechanism but conclusive evidence is lacking. Here we demonstrate that all unexposed individuals harbor SARS-CoV-2-specific memory T cells with marginal cross-reactivity to common cold corona and other unrelated viruses. They display low functional avidity and broad protein target specificities and their frequencies correlate with the overall size of the CD4+ memory compartment reflecting the immunological age of an individual. COVID-19 patients have strongly increased SARS-CoV-2-specific inflammatory T cell responses that are correlated with severity. Strikingly however, patients with severe COVID-19 displayed lower TCR functional avidity and less clonal expansion. Our data suggest that a low avidity pre-existing T cell memory negatively impacts on the T cell response quality against neoantigens such as SARS-CoV-2, which may predispose to develop inappropriate immune reactions especially in the elderly. We propose the immunological age as an independent risk factor to develop severe COVID-19.