The mechanisms underlying the coordinated beating of cilia and flagella remain incompletely understood despite the fundamental importance of these organelles. The axoneme (the cytoskeletal structure of cilia and flagella) consists of microtubule doublets connected by passive and active elements. The motor protein dynein is known to drive active bending, but dynein activity must be regulated to generate oscillatory, propulsive waveforms. Mathematical models of flagellar motion generate quantitative predictions that can be analysed to test hypotheses concerning dynein regulation. One approach has been to seek periodic solutions to the linearized equations of motion. However, models may simultaneously exhibit both periodic and unstable modes. Here, we investigate the emergence and coexistence of unstable and periodic modes in three mathematical models of flagellar motion, each based on a different dynein regulation hypothesis: (i) sliding control; (ii) curvature control and (iii) control by interdoublet separation (the 'geometric clutch' (GC)). The unstable modes predicted by each model are used to critically evaluate the underlying hypothesis. In particular, models of flagella with 'sliding-controlled' dynein activity admit unstable modes with non-propulsive, retrograde (tip-to-base) propagation, sometimes at the same parameter values that lead to periodic, propulsive modes. In the presence of these retrograde unstable modes, stable or periodic modes have little influence. In contrast, unstable modes of the GC model exhibit switching at the base and propulsive base-to-tip propagation.