SARS-CoV-2, the virus responsible for COVID-19, is causing a devastating worldwide pandemic, and there is a pressing need to understand the development, specificity, and neutralizing potency of humoral immune responses during acute infection. We report a cross-sectional study of antibody responses to the receptor-binding domain (RBD) of the spike protein and virus neutralization activity in a cohort of 44 hospitalized COVID-19 patients. RBD-specific IgG responses are detectable in all patients 6 days after PCR confirmation. Isotype switching to IgG occurs rapidly, primarily to IgG1 and IgG3. Using a clinical SARS-CoV-2 isolate, neutralizing antibody titers are detectable in all patients by 6 days after PCR confirmation and correlate with RBD-specific binding IgG titers. The RBD-specific binding data were further validated in a clinical setting with 231 PCR-confirmed COVID-19 patient samples. These findings have implications for understanding protective immunity against SARS-CoV-2, therapeutic use of immune plasma, and development of much-needed vaccines.