Natural populations are finite in size and thus subjected to genetic drift. Genetic drift changes the genetic variance structure and decreases genetic variation within populations. However, selection may counteract or reinforce the effects of genetic drift, and environmental conditions determine not only the strength and direction of selection but also the loci targeted by selection. We performed a large-scale experimental evolution study using 42 replicate populations of Drosophila melanogaster subjected to three different rates of genetic drift, by rearing them at population sizes 10 (N10), 50 (N50) or 500 (N500), and two different ecological relevant thermal regimes, one stable across generations and one increasing in minimum, maximum and average temperature. RAD sequencing of pooled samples at five time points across 20 generations revealed a higher rate of loss of genetic diversity in small compared to large populations. Given the hypothesis that effective population size (Ne) is a fraction of the census size (N), due to deviations from Fisher-Wright’s idealized population, and constant across different N, we estimated Ne/N. N10 populations had Ne/N ratio close to 1, with this ratio decreasing with increasing population size (N50 and N500), showing significantly slower loss of genetic diversity in small populations than expected. This pattern is potentially due to directional selection in the large populations, strong associative overdominance during the sustained bottleneck of the small populations or a decrease in the variance in reproductive output in the small populations. The loss of genetic diversity was generally not affected by the stressful ramping thermal regime. In conclusion effects of demography on the loss of genetic diversity deviate from the typical expectations of small populations, and interact with environmental stress in shaping the genetic variance structure across chromosomes.