Early models of sequence evolution made several important regularity assumptions to describe the process of nucleotide or amino-acid evolution in a simple and manageable manner. Much of the methodological work in statistical phylogenetics over the past decades has been devoted to relaxing these assumptions. As a result, nowadays the most popular models of sequence evolution assume that not all transitions between nucleotides or amino-acids are equally likely, that different sites evolve independently, but that they evolve at different rates and even sometimes according to different transition matrices. These models however still make several patently unrealistic assumptions. In particular, they assume that sequence evolution has operated according to a unique transition matrix along the entire phylogeny, an assumption that is not realistic notably for all data sets in which nucleotide or amino-acid composition varies among sequences. In this talk I will review models of sequence evolution that relax this assumption, that I will globally call branch-heterogeneous models. I will explain why these models are particularly difficult to use in either the maximum likelihood or the Bayesian frameworks, and I will present our current efforts to improve their usability.