Human Leukocyte Antigen (HLA) class I genes mediate the cytotoxic T-cell response against HIV by presenting viral epitopes at the surface of infected cells. Due to polymorphism in their binding groove, individual HLA alleles bind to different arrays of epitopes, resulting in differential responses to HIV infection; e.g. HLA-B*57:01 confers protection while HLA-B*35:02 confers susceptibility to HIV progression. Research over the last decades has robustly established a strong association between HLA variants, particularly in HLA-B, and spontaneous HIV control as well as progression to AIDS. However, the functional basis for this association and more specifically the extent to which HLA-bound HIV epitopes explain this association are largely elusive. Using an unprecedented clinical dataset of 6,311 chronically HIV infected patients we computationally predicted the binding affinities of each patient’s HLA class I alleles for all possible HIV epitopes. We show that set point viral load (spVL), an established correlate of HIV disease progression, is negatively associated with the breadth of the epitopes bound by a patient’s HLA alleles. In linear regression models, specific HLA-bound epitopes explain more variation in spVL than the established HLA genotype associations. The model was further improved when HLA-bound epitopes were predicted from autologous HIV sequences, available for a subset of the patients, rather than from the HIV reference sequence. In conclusion, our findings provide a novel functional explanation for the well-established association between HLA class I variation and HIV control. The specific epitopes that we found to have strong impact on spVL could provide new targets for antiretroviral drugs.