Empirical mode decomposition (EMD) is a self-adaptive method and suitable to analysing the non-stationary and nonlinear signals. The envelope technique and mode-mixing problem are the most important topics of the EMD. A nonlinear and nonstationary signal is modeled as a multicomponent amplitude modulation-frequency modulation (AM-FM) signal and each intrinsic mode function (IMF) of the EMD will be modeled as a single AM-FM signal. Through studying the envelope technique of single AM-FM signal and the mode-mixing problem caused by separating multicomponent AM-FM signals with the EMD algorithm, a new necessary condition of envelope and the numerical calculation method of the new conditional envelope algorithm are presented. Based on the new conditional envelope algorithm, a new estimation algorithm for phase and instantaneous frequency of single component AM-FM signal is proposed. A solution is presented to the mode-mixing problem that occurs when multicomponent AM-FM signals are separated. The efficacy of the proposed method is verified by several simulation signals and a measured data of rubbing fault of rotor system.