Background: Since December 2019, the outbreak of coronavirus disease (COVID-19) has been occurred by novel coronavirus (SARS-CoV-2). The rapid and sensitive immunoassays are urgently demanded for detecting specific antibodies as assistant diagnosis for primary screening of asymptomatic individuals, close contacts,suspected or recovered patients of COIVD-19 during the pandemic period. Methods: The recombinant receptor binding domain of SARS-CoV-2 spike protein (S-RBD) was used as the antigen to detect specific IgM and the mixture of recombinant nucleocapsid phosphoprotein (NP) and S-RBD were used to detect specific IgG by the newly designed quantum-dot lateral flow immunoassay strip (QD-LFIA), respectively. Results: A rapid and sensitive QD-LFIA based portable fluorescence smart-phone system was developed for detecting specific IgM/IgG to SARS-CoV-2 from 100 serum samples of COVID-19 patients and 450 plasma samples from healthy blood donors. Among 100 COVID-19 patients diagnosed with NAT previously, 3 were severe, 35 mild and 62 recovered cases. By using QD-LFIA, 78 (78%) and 99 (99%) samples from 100 COVID-19 patients serum were detected positive for anti-SARS-CoV-2 IgM or IgG, respectively, but only one sample (0.22%) was cross-reactive with S-RBD from 450 healthy blood donor plasmas that were collected from different areas of China. Conclusion: An ultrasensitive and specific QD-LFIA based portable fluorescence smart-phone system was developed fo r detection of specific IgM and IgG to SARS-CoV-2 infection, which could be used for investigating the prevalence or assistant diagnosis of COVID-19 in humans.