Diminishing returns refers to the phenomenon that the same advantageous mutation is less beneficial when occurring in fitter genotypes than when occurring in less fit genotypes. Although diminishing returns has been frequently observed in experimental evolution, its prevalence in natural polymorphisms is unknown. Further, the cause of diminishing returns remains elusive. Here we address these questions using the genome sequences of 1005 haploid segregants produced from a cross between two budding yeast strains and the mitotic growth rates, a proxy of fitness (F), of these segregants in 47 different environments. Under each environment, we estimated the fitness effect size (s) of each single nucleotide polymorphism (SNP) from 100 segregants. Depending on the environment considered, 60-95% of SNPs exhibit diminishing returns (i.e., s decreases as the mean F rises from the 100 least fit segregants to the 100 fittest segregants). Surprisingly, given a median F, we found s to decrease as the environment quality (Q), measured by the mean fitness of all segregants in the environment, rises, revealing a previously unrecognized property of diminishing returns. The fraction of SNPs showing diminishing returns in an environment increases with Q, with or without the control for the median F of the segregants compared across environments. Our results are explainable by neither idiosyncratic nor global epistasis, but may arise under a modular epistasis model where the fitness of a genotype can be improved in a limited number of ways. Together, our findings reveal that diminishing returns is a general phenomenon that depends on both genotype quality and environment quality and that it arises from modular epistasis. Our findings explain why the poorer the environment the faster the adaptation.