Maryland is best known for the prison at Point Lookout, a peninsula formed by the confluence of the Chesapeake Bay and the Potomac River. It attracted attention from the outset of English colonization in America. In 1830 the federal government erected a lighthouse on the tip of the Point. In 1862, following General George B. McClellan's unsuccessful campaign to capture Richmond, the federal government erected Hammond Hospital at the tip of the Point. The ward buildings radiated in spoke fashion from a central bay. Wounded and sick soldiers began pouring in for treatment.
The following year, after the Battle of Gettysburg, Union authorities started sending Confederate prisoners to Point Lookout for incarceration. As the prisoner population swelled to 20,000 and more, a wooden walled prisoner pen was constructed on the bay shore. The rebel captives were held inside and were given only tents for shelter. Exposure, disease, and starvation took their toll. Of the 50,000 men held at the Point between 1863 and 1865, nearly 4,000 died.
In 1864, the Maryland Confederate General Bradley T. Johnson attempted a daring raid on the prison. His plan was to liberate the prisoners, arm them, and march on Washington as part of General Jubal Early's offensive. Intelligence of his plan reached Union authorities in time for them to make preparations, and Johnson abandoned his plan when the Confederate authorities found out that the plans has been published.
Among the Federal Army units to rotate from the front to serve as guards at Point Lookout were African-American soldiers of the United States Colored Troops. Ironically, in some cases, these soldiers had occasion to guard their former masters, which led to instances of brutality, or of kindness, depending on the nature of their relationship previous to the war. Sgt. Christian A. Fleetwood of the 4th U.S.C.T., a Baltimore native who had never been a slave, and Medal of Honor winner for his bravery at the Battle of Chapin's Farm in front of Richmond, and Sgt. Charles Douglass, of the 5th Massachusetts Colored Cavalry, and son of Frederick Douglass the noted black abolitionist, were among the soldiers rotating through Point Lookout. Elements of the Veteran Reserve Corps also served as guards, and in the hospital as orderlies and stewards. The First Regiment, and Fourth Regiment U.S. Volunteers, were organized from "galvanized" Confederate prisoners at Point Lookout, and shipped out from there for service in the west fighting Indians.