In order to improve the communication rates of brain-computer interface(BCI's), scientists are developing appropriate signal processing methods to extract the user's messages and commands from electroencephalograph (EEG). A fast fixed-point algorithm for independent component analysis(FastICA), possesses the advantages of simply structure and fast computation. However, in some cases, many signals are not completely independent, the stability of the algorithm won't be as ideal as people have expected. In fact, the reason that system does not converge steadily is the fixed step size in FastICA algorithm, that is, The negentropy J(wn+1TZ) of random vectors no longer monotonic increasing in the iterative process of separated vectors. We define a cost function δJ=J(w<inf>n+1</inf><sup>T</sup>Z)-J(w<inf>n</inf><sup>T</sup>Z) and a time-variant step size μ(t), and put forward a algorithm of adjusting step size by the variety of the cost function in iterative process. Results from a series of simulation and experiments show that, the stability and convergence of algorithm is improved.