One of the biggest challenges in the era of genomics is linking genes to phenotypes and specifically to the body parts they affect. Many tools have tried to address this through indirect approaches using features such as expression levels, biochemical pathways and molecular function. Here, we present Gene ORGANizer (geneorganizer.huji.ac.il), a tool we have developed to directly link genes to organs. Gene ORGANizer allows researchers to analyze the anatomical effects of genes, and to understand the shared impact of groups of genes. We used more than 100,000 gene-phenotype associations to build the Gene ORGANizer database, which contains links of more than 7,000 genes to ~150 organs, body systems and anatomical regions. Using the tool, we showed that the most substantial regulatory changes in recent human evolution happened in the vocal tract. This trend is unique to modern humans and arose after the split from the Neanderthal and the Denisovan. We also confirmed previous hypotheses that chromosome X is significantly enriched with genes affecting the brain and the reproductive system. Surprisingly, we found an even more pronounced, yet previously unreported, pattern: chromosome X is significantly enriched with genes that affect facial features. These results suggest that chromosome X experiences a unique selection regime, where genes that affect some body parts are preferentially represented, while others are selected against. We expect Gene ORGANizer to be useful in a broad range of evolutionary, medical, and molecular studies, with applications in any analysis aimed at characterizing phenotypic consequences of genetic changes.