Treatment of extensive bone defects requires autologous bone grafting or implantation of bone substitute materials. An attractive alternative has been to engineer fully viable, biological bone grafts in vitro by culturing osteogenic cells within three-dimensional scaffolds, under conditions supporting bone formation. Such grafts could be used for implantation, but also as physiologically relevant models in basic and translational studies of bone development, disease and drug discovery. A source of human cells that can be derived in large numbers from a small initial harvest and predictably differentiated into bone forming cells is critically important for engineering human bone grafts. We discuss the characteristics and limitations of various types of human embryonic and adult stem cells, and their utility for bone tissue engineering.