The most diverse military ever assembled by the United States Army passed through the streets of Fort Scott, Kansas in 1865. White soldiers of the 3rd Wisconsin Cavalry, black soldiers of the 1st Kansas Colored Infantry, and red soldiers of the 1st Indian Home Guard were separated by race but joined as Union soldiers. The racial composition of Fort Scott had come a long way since its original establishment in 1842. Back then, white soldiers were to protect the Permanent Indian Frontier, designated by the United States government to set aside land for the use of relocated Indians from the east. From 1842 to 1853, Fort Scott fulfilled the multiple missions of keeping the Indians from entering Missouri, keeping Missourians from entering Indian Territory, and controlling illegal liquor trade with the Indians.

In 1854, with the establishment of the Kansas-Nebraska Act, settlers could legally come into Kansas Territory. Amid the strife commonly referred to as "Bleeding Kansas," the town of Fort Scott was established. Soldiers returned periodically to Fort Scott because of the violence between pro- and antislavery forces. In 1859, an uneasy, albeit temporary, peace returned.

This brief calm at Fort Scott ended with the firing on Fort Sumter. Now the town became a supply center for Union armies fighting in NW Arkansas, SW Missouri, and the Indian Territory. The August 31, 1861 Fort Scott Democrat said, "Our city presents the appearance of one grand military camp. Every vacant tenement has been taken possession of by the Government for quarters for Uncle Sam's men. Villages of tents surround the town, and the measured tread of the sentinel and the tramp of cavalry may be heard at all times of the day and night." The fort soon also became a refugee center as Pro-Union civilians - black, white, and red - streamed into the safe haven of Fort Scott. In July 1862, Kansas became the first Union State to recruit and train black soldiers. Free Blacks and escaped slaves from Missouri were mustered in as members of the 1st Kansas Colored Infantry at Fort Scott. Union Officers recognized the unit's excellent combat record. On July 26, 1863, Major General James G. Blunt wrote, "The First Kansas (Colored) particularly distinguished itself, they fought like veterans and preserved their line unbroken throughout the engagement. Their coolness and bravery I have never seen surpassed." In November 1863, Brigadier General John McNeil was effusive: "The negro regiment is a triumph of drill and discipline, and reflects great honor on Col. Williams, in command. Few volunteer regiments that I have seen make a better appearance. I regard them as first-rate infantry." And Colonel James M. Williams himself noted, several months later, "The officers and men all evinced the most heroic spirit, and those that fell died the death of a true soldier."