Frederick Stanley Olson was born on April 4, 1930, in the town of New Ulm, Minnesota, a town of about 9,000 population at the time, of German heritage, named after the city of Ulm in Germany. Other than the emigrants from Ulm, most of the early pioneer settlers in the mid-19th century were of German-Bohemian descent, with farming backgrounds.
Fred's early pioneer families were the Gags and the Forsters on his mother's side, who settled in New Ulm around 1860. His father's father, with his three brothers, all professional men of Norwegian descent, came from Milwaukee, Wisconsin to Minnesota in the late 1890's, with three of them settling in other parts of southern Minnesota while his grandfather, Albert A. Olson, settled in New Ulm and became Vice-President of the town's large Eagle Roller Mill. His father's mother was of the Schaeffer families, who settled a farm northeast of New Ulm and were survivors of the last Sioux Uprising in 1862, known locally as the "Indian Massacre." His father's mother, Elizabeth Schaeffer, was born during these Indian wars and, because of the kindness of her father, who had fed the starving Sioux, they spared his farm while others around them were burned to the ground.