Over 3.1% of the people in the south owned slaves. But since 32.1% of the people in the south were slaves themselves, that's not a reasonable number. 4.7% of the free people in the south owned slaves.
How you count is important, though- a plantation owner was most likely the one person counted as owning the slaves, but his wife, children and any hangers-on were just as much a part of the slaveowning population. Equally important was the free population that depended on the slaveowners- they may not have owned slaves themselves, but they had built their lives on the money that came from slavery.
Using the census data on that Wikipedia page I linked to, slave states (I include DC) had a total population of 12,315,406 and a slave population of 3,953,693. 32.1% of the people in the slave states were slaves. There were 393,967 slaveowners (
see Table 4, sourced from
Historical Statistics of the United States, 1970).
None of those people were victims of "northern aggression." Slaveholding interests were overrepresented in Congress from the beginning (remember the 3/5 compromise?)- Virginia's 490,865 slaves in 1860 gave Virginia representation in the House corresponding to 294,519 people- nearly the entire population of Vermont, and more than that of Rhode Island, Minnesota, Florida, Delaware or Oregon. Look at that again- Virginia alone had more seats in the House of Representatives than 5 states simply by counting its slave population. Which was not represented by these Representatives.
They used these seats earned by counting slaves to impose things like the Fugitive Slave Act on free states, denying them the right to decide who was free in their own states. When eventually the political winds shifted against slavery, slaveowners suddenly became fans of states' rights and seceded. If you haven't read Georgia's declaration, go back and look at it. It's disgusting, really. IIRC, Texas' was longer and worse, but this morning I read Natalya's post from Atlanta and stuck with Georgia.