Maternal smoking during pregnancy has repeatedly been associated with decreased sperm counts in sons. Nevertheless, our team recently detected a lower total sperm count in the sons of smoking fathers as compared to sons of non-smoking fathers. Since paternal and maternal tobacco smoking often coincide, it is difficult to discriminate whether effects are mediated paternally or maternally when using questionnaire- or register-based studies. Therefore, getting an objective measure of the maternal nicotine exposure level during pregnancy might help disentangling the impact of paternally and maternally derived exposure. Our aim was to study how paternal smoking at the time of the pregnancy was associated with semen quality in the sons after adjusting for the maternal levels of nicotine exposure during pregnancy. We recruited 104 men (17-20 years old) from the general Swedish population. The participants answered a questionnaire about paternal smoking. Associations between smoking and semen volume, total sperm count, sperm concentration, morphology and motility were adjusted for levels of the nicotine metabolite cotinine in stored maternal serum samples obtained from rubella screening between the 6th and 35th week of pregnancy. We additionally adjusted for the estimated socioeconomic status. After adjusting for the maternal cotinine, the men of smoking fathers had 41% lower sperm concentration and 51% lower total sperm count than the men of non-smoking fathers (p = 0.02 and 0.003, respectively). This was robust to the additional adjustment. Our results suggest a negative association between paternal smoking and sperm counts in the sons, independent of the level maternal nicotine exposure during the pregnancy.