Understanding factors that drive urban growth is essential to cellular automata (CA) based urban modeling. Multicollinearity among correlated factors may cause negative effects when building CA transition rules, leading to a decrease in simulation accuracy. We use a nonparametric generalized additive model (GAM) to evaluate these relationships through flexible smooth functions to capture the dynamics of urban growth. A GAM-based CA (termed GAM-CA) model was then developed to simulate the rapid urban growth in Shanghai, China from 2000 to 2015. GAM highlights the significance of each candidate factor driving urban growth during the past 15 years. Compared to logistic regression, the GAM-CA transition rules fitted the observed data better and yielded improved overall accuracy and hence more realistic urban growth patterns. The new CA model has great potential for capturing key driving factors to simulate dynamic urban growth, and can predict future scenarios under various spatial constraints and conditions.