The History of American Slavery

“Nothing to Stay Here For”

How enslaved people helped to put slavery to death in New Orleans.

A group of “Contrabands,” the War for the Union, 1862.
A group of “Contrabands,” the War for the Union, 1862.

Courtesy of James F. Gibson/Library of Congress

This article supplements Episode 9 of the History of American Slavery, our inaugural Slate Academy. Please join Slate’s Jamelle Bouie and Rebecca Onion for a different kind of summer school. To learn more and to enroll, visit

Excerpted from Beyond Freedom’s Reach: A Kidnapping in the Twilight of Slavery by Adam Rothman. Published by Harvard University Press.

Touring the United States for the London Times, war correspondent William Howard Russell reached New Orleans late in May 1861, where he discovered a city all ablaze. Confederate flags flew from the public buildings and private homes. Uniformed soldiers paraded through the streets in smart columns of dash and pomp. Gentlemen at the St. Charles Hotel pored over the latest papers for news of the dawning war. The police were rounding up suspected abolitionists, and every night mysterious fires flared up around the city—set, it was rumored, by the slaves.

After hobnobbing with politicians, planters, and merchants in New Orleans, Russell observed that the Confederate elite “believe themselves, in fact, to be masters of the destiny of the world.”1 They would soon discover that they were not even masters of their own homes.

Secession was followed by war, and the war reached the New Orleans levee in the spring of 1862 in the form of Union gunships. The federals came to restore Louisiana to the Union, not to liberate the slaves, but slavery began to fray anyway from what Abraham Lincoln called the “friction and abrasion” of war. New Orleans and its environs became an arena of chaotic struggle between slaves, masters, and Union authorities.2

Slaves were neither blind nor deaf to the tumult in the city. They watched the parades and presentations of flags. They read the newspapers or gleaned the information secondhand. They saw the young men march off to wage a distant war to keep them from tasting freedom. The perceptive Russell was not convinced by his hosts’ insistence that the slaves were happy. He thought many appeared “morose, ill-clad, and discontented” and noted that the patrols had been strengthened.3

Households crackled with conversations about politics stoked by the latest rumors to appear in the newspapers. All their masters’ loose talk about the black Republicans and abolitionism created an expectation among slaves that the coming of the federal troops would break their chains. Small wonder, then, that slaves working on the levees along the Mississippi saluted the Union gunboats steaming upriver to New Orleans in late April 1862.4 “Thousands of negroes welcome us with various demonstrations of pleasure,” noted Rufus Kinsley of the 8th Vermont in his diary as he came up the Mississippi.5 John De Forest of the 12th Connecticut wrote that the black people along the river “gathered to stare at us, and when there were no whites near, they gave enthusiastic evidence of good will, dancing at us, waving hats or branches and shouting welcome.” He was close enough to shore to hear an old woman’s joyful outburst: “Bress de Lawd! I knows dat ar flag. I knew it would come. Praise de Lawd!” De Forest viewed the scene as funny and accentuated its comedy by transcribing the woman’s words in a demeaning way. After several weeks among New Orleans’ resentful white Confederates, he would come to appreciate the black inhabitants’ unionism: “As to the Negroes, they are all on our side.”6 The real question was whether the Union was on the negroes’ side.

Strong political headwinds—the demands of the Union slave states, opposition from Northern Democrats, constraints imposed by the Constitution—slowed anti-slavery Republicans and frustrated Northern abolitionists. Progress toward emancipation was ad hoc and incremental, and it was pushed forward by slaves and free people of color who pressed for change. The arrival of Union troops marked the beginning of the end of slavery in New Orleans, but the end of slavery in New Orleans was not a foregone conclusion.7

Union authorities clashed with each other over the problem of slavery. Gen. Benjamin Butler was no abolitionist. When he got to New Orleans, he moved cautiously on the slave question. He wanted to cultivate the buried seeds of unionism among local white people and to preserve the economy, which depended on slave labor. But the arrival of the federals loosened masters’ hold of this portion of their property. Many slaves began to flee to the Union army. Butler found employment for some but turned others away. He could not take them all, he protested. “What would be the state of things if I allowed all the slaves from the plantations to quit their employment and come within the lines is not to be conceived by the imagination.”8

Brig. Gen. John Wolcott Phelps, a patrician abolitionist from Vermont, thought differently. Upon arriving in New Orleans, Phelps established Camp Parapet just upriver from the city and made it known that he would welcome and protect slaves who managed to escape to his encampment. They came in droves, men and women, old and young, alone and in groups, on foot and by water. Some had been kicked out by their masters. Many arrived on their own. Others were brought in by Phelps’ raiders. Several hundred escapees gathered under Phelps’ protection by summertime. A Union officer whom Butler sent to Camp Parapet reported that the soldiers’ influence was creating chaos on nearby plantations. The slaves refused to work “as they say they have only to go to the fort to be free, and are therefore very insolent to their masters.”9

Phelps not only harbored runaway slaves. Following passage of the Second Confiscation and Militia acts in July 1862, he also began to arm and drill the black men who came into the camp. Phelps asserted that “the best way of preventing the African from being instrumental in a general state of anarchy is to enlist him in the cause of the Republic.”10 Butler denied that Phelps had any authority to do so and feared that arming the slaves would lead to a revolt. “We shall have a negro insurrection here I fancy,” Butler confided to his wife in late July.

Rather than send black men and women back into slavery, Phelps resigned in protest and went home in early August. He had been ahead of the curve. Union policy soon caught up with the logic of events on the ground, and Butler himself welcomed slaves and free people of color into the Union army later in the year.

The recruitment of black soldiers crippled slavery in Louisiana. More than 24,000 black men entered the Union service from Louisiana, more than from any other state. Some came from the ranks of free men of color, but most had been slaves. Military service was their ticket to freedom and their chance, finally, to wage war against slavery.

Black soldiers fought on multiple fronts. Not only did they fight against the Confederacy, but they fought for fairness within the Union army by demanding equal pay with white soldiers, the opportunity to see combat, and the chance to serve as officers. While vindicating the “manhood of the colored race,” black military service had a powerful impact on the homefront. Recruitment of able-bodied men left planters short of labor.

The slaves who stayed behind pressed for better conditions. While black soldiers gained honor on the battlefield, black women waged their own battles in contraband camps and on the homefront. Soldiers’ families faced harassment and reprisals, but they also stood up for themselves. Women ran away. They talked back. They refused to be beaten. They reclaimed their children from owners. A “war within” slave-owning households in New Orleans tumbled out into public view in the streets and the courts of the city.11

Ann Wilkinson Penrose was one of those Confederate women in New Orleans whose privileged world fell apart when the Yankees arrived. She had a fine pedigree. Her grandfather was James Wilkinson, a famous general. Her husband, Clement Biddle Penrose, was a wealthy Philadelphian who moved to Louisiana and then died of yellow fever. One son died just before the war, and the other—grandly named Joseph Biddle Wilkinson Penrose—marched off to fight in the East, leaving his mother to look after her elderly father, three other “retired & quiet” women, and their slaves. In her spare time she kept a diary composed of unsent letters to her soldier son. She may have hoped he would read it someday, but its more immediate purpose was to let her vent her anxiety and frustration at living under Union misrule.12

Penrose was 60 years old and hobbled around on crutches. She suffered from sore feet and anxious headaches. She felt almost trapped in New Orleans by her family’s slaves. Taking care of the property was easier to write about than to do. Penrose’s mastery was undermined by the presence of the Union and the resistance of the slaves. Just weeks after the federals arrived, Penrose began to hear of slaves running off in a “stampede” to Fort Jackson and other places. Some runaways had been put to work fixing a breach in the levee, and she hoped their new masters “will keep them hard at it.”13 On May 27, a slave named James Guy ran off “very coolly and deliberately.” He had been sent to the market and never came back, although he had the politeness—perhaps it was a jab—to send home the groceries and his streetcar ticket.14

Things went downhill from there. At the end of May, Penrose’s father was arrested by Yankee soldiers for plotting to have Butler assassinated. The allegation came from one of the family’s slaves, a man named Ben Travis, who had been one of the first to run away. Travis arrived with the soldiers to arrest Wilkinson. He entered the house and “behaved in the most insolent manner to us.” “If accusations of vindictive servants are to be taken, no one can feel safe,” Penrose complained, “so indulged, I may almost say, petted, as those servants have been, and such confidence as we had in them; now, I have in none, if they could turn against us.”15

Penrose’s fear and rage at the crumbling of slavery mounted throughout the summer and fall of 1862. She complained of the “misrule & anarchy prevailing among a certain class … and they do anything they please with impunity; no matter how insolent they are, or what provocation they give.” If punished, the slaves appealed to the provost marshal, who would arrest their masters and impose a fine or throw them in prison.16 Nerves tautened when “Proclamation Phelps” began to arm the black men flocking to Camp Parapet.

By wintertime, the whole society seemed upside down: “Our negroes run rioting about; in any jarring between those who stay at home & their owners, their part is invariably taken; masters have accusations laid against them by their slaves, & true or false are imprisoned or fined at their instigation. Regiments of negroes parade our streets, in short they are our masters.”17

Penrose’s grip on her own household loosened. “Our servants do pretty much as they please,” she fumed in September.18 Time and again Penrose complained of their insolence:  

Feb 2 [1863]: Old Lyddy was here this morning, and very insolent indeed, telling Rebecca that she was free, and signifying she would do as she please, having previously used words to that effect, and even more insolent still to Kate.19

April 13: This morn. at breakfast, the bread was as usual intolerably bad, and as the flour is good it is the result of Becky, who is our cook at present; when the cakes came in they were also as bad as could be, heavy as lead, and like dough; I rose and went into the kitchen to speak to Becky; she was leaning down, with her back towards me as I entered, and I could not resist giving her a good hard slap on the shoulder, which bye the bye hurt my hand, I have no doubt, more than it did her, at the same time I asked her how she dared to send in such bread & cakes; she started up, looked furiously at me, and exclaimed, “don’t you do that again, let it be the last time, or I’ll just march out of this yard.”20

May 6: Margaret especially is a little ungrateful humpbacked wretch, who for wee-[ks] at a time has been laid up in her bed, with the Dr attending on her, & now repays us with her insolence.21

May 21: Clara had been most insolent and insubordinate to both yr aunts Vir. & Julia. Yr aunt V. took hold of her by her sleeve, I think, and attempted to slap her, whereupon she broke from her & ran screaming down the street, to a house about a square from this, which is the headquarters of some of the officers, and where there are always a number of negroe men in their service & uniform. There she laid a complaint, that she had been whipped with a cowskin, beat with a stick, and pitched down stairs. I was very uneasy knowing the consequences that have followed such reports, true or false, for they always take their side.22

Although Penrose fumed over their laziness, incompetence, and impudence, Lyddy, Becky, Margaret, and Clara refused to be scolded and slapped. They were not going to put up with their mistresses’ verbal and physical abuse anymore. With no able-bodied white men at home, and black soldiers stationed nearby, the Penrose household collapsed into a private civil war.

Penrose’s diary unwittingly depicted slaves’ family ties, community networks, and political connections. It shows that the war for freedom was waged by women cooking in kitchens as well as soldiers massing on battlefields. It shows, moreover, that slavery crumbled in New Orleans not from a single dramatic blow or stroke of a pen, but from the accumulated weight of resistance by slaves such as Lyddy, Margaret, Becky, and Clara once the Union army arrived.

In October 1863, with “nothing to stay here for,” Ann Wilkinson Penrose fled to the Confederate bastion of Mobile, Alabama.23 Her son Joseph probably never read her painful chronicle, since he was killed in battle just three days before Lee surrendered at Appomattox. If he had survived to make it back home, he would have scarcely recognized New Orleans.

1. William Howard Russell, My Diary North and South, ed. by Eugene Berwanger (Philadelphia: Temple University Press, 1988), 157–172 (quotation on 172).

2. “Friction and abrasion” comes from Abraham Lincoln’s plea to representatives from the border slave states to accept compensated emancipation on July 12, 1862, Collected Works of Abraham Lincoln, ed. Roy P. Basler, vol. 5 (New Brunswick, New Jersey: Rutgers University Press, 1953), 318. See James Oakes, Freedom National: The Destruction of Slavery in the United States, 1861–1865 (New York: W.W. Norton, 2013), 288–293. For the collapse of slavery in New Orleans, see Ira Berlin, Barbara J. Fields, Thavolia Glymph, Joseph P. Reidy, and Leslie S. Rowland, Freedom: A Documentary History of Emancipation, 1861–1867, ser. 1, vol. 1, The Destruction of Slavery (New York: Cambridge University Press, 1985), ch. 4.

3. Russell, My Diary, 233.

4. Robert Reinders and Frank D. Harding, “A Wisconsin Soldier Re-ports from New Orleans,” Louisiana History, 3 (Autumn 1962): 362; James Parton, General Butler in New Orleans (New York: Mason Brothers, 1864), 267.

5. David C. Rankin, Diary of a Christian soldier: Rufus Kinsley and the Civil War (New York: Cambridge University Press, 2004), 92.

6. John De Forest, A Volunteer’s Adventures: A Union Captain’s Record of the Civil War (New Haven, Connecticut: Yale University Press, 1946), 17 (“Bress”), 31 (“our side”). 

7. Ira Berlin, Barbara J. Fields, Steven F. Miller, Joseph P. Reidy, and Leslie S. Rowland, Slaves No More: Three Essays on Emancipation and the Civil War (New York: Cambridge University Press, 1992); Eric Foner, The Fiery Trial: Abraham Lincoln and American Slavery (New York: W.W. Norton, 2010); Oakes, Freedom National.

8. Silvana Siddali, From Property to Person: Slavery and the Confiscation Acts, 1861–1862 (Baton Rouge, Louisiana: Louisiana State University Press, 2005), 52–54. For Butler’s political career before the war, see Pierson, Mutiny, 156–160.

9. War of the Rebellion: A Compilation of the Official Records of the Union and Confederate Armies (Washington, D.C.: U.S. Government Printing Office, 1886), ser. 1, 15:446 (hereafter cited as OR). For the tussle between Butler and Phelps, see Berlin et al., Destruction of Slavery, 192–195. Phelps’ controversial proclamation against slavery from December 1861 can be found in OR, ser. 1, 15:486–490. On Phelps’ career, see William F. Messner, “General John Wolcott Phelps and Conservative Reform in Nineteenth Century America,” Vermont History, 53 (Winter 1985): 17–35.

10. OR, ser. 1, 15:535.

11. For statistics on black military service in the Civil War, see Berlin et al., Slaves No More, 203. On Afro-Louisianans’ military service during the Civil War, see Hollandsworth Jr., The Louisiana Native Guards, 104 (“manhood”); Stephen J. Ochs, A Black Patriot and a White Priest: André Cailloux and Claude Paschal Maistre in Civil War New Orleans (Baton Rouge, Louisiana: Louisiana State University Press, 2000). For an overview of the history of black military service during the Civil War with a selection of marvelous sources, see Freedom: A Documentary History of Emancipation, 1861–1867, ser. 2, The Black Military Experience, ed. Ira Berlin, Joseph Reidy, and Leslie Rowland (New York: Cambridge University Press, 1982). On the “war within,” see Thavolia Glymph, Out of the House of Bondage: The Transformation of the Plantation Household (New York: Cambridge University Press, 2008), esp. ch. 4.

12. Ann Wilkinson Penrose Diary and Family Letters, 1861–1865, Mss. 1169, Louisiana and Lower Mississippi Valley Collections, Louisiana State University Libraries (hereafter cited as PD), 2 June 1862 (“Retired & quiet”). On the Penrose’s family, see Josiah Granvil Leach, History of the Penrose Family of Philadelphia (Philadelphia: D. Biddle, 1903), 89, 113. Joseph Penrose (25) and his mother Ann (51) appear in the 1860 federal census for Plaquemines Parish. Joseph is noted as having $25,000 in real estate and $10,000 in personal estate. They are adjacent neighbors to Ann Penrose’s brother, Robert Wilkinson, a wealthy sugar planter who served as an officer in Louisiana’s 15th Regiment and was killed at Second Manassas. The 1860 slave schedule lists 11 slaves under J.B.W. Penrose. The 1860 census for Jefferson Parish, Louisiana lists Joseph B. Wilkinson as a “gentleman” of 74 years and a personal estate valued at $40,000. On the experience of Confederate women in New Orleans, see George Rable, “ ‘Missing in Action’: Women of the Confederacy,” in Divided Houses: Gender and the Civil War, ed. Catherine Clinton and Nina Silber (New York: Oxford University Press, 1992), ch. 8; Drew Gilpin Faust, Mothers of Invention: Women of the Slaveholding South in the American Civil War (Chapel Hill: University of North Carolina Press, 1996), 207–214; Jacqueline G. Campbell, “ ‘The Unmeaning Twaddle about Order 28’: Benjamin F. Butler and Confederate Women in Occupied New Orleans, 1862,” Journal of the Civil War Era, 2 (March 2012): 11–30.

13. PD, 23 May 1862.

14. Ibid., 17 May 1862.

15. Ibid., 3 June 1862.

16. Ibid., 15 August 1862.

17. Ibid., 2 December 1862.

18. Ibid., 20 May 1862 (“Proclamation Phelps”), 7 August 1862 (“4000”), 25 September 1862 (“Our servants”).

19. Ibid., 2 February 1863.

20. Ibid., 13 April 1863.

21. Ibid., 6 May 1863.

22. Ibid., 21 May 1863.

23. PD, 5 October 1863.