There where a small number of people that profited greatly from slavery. However, economic growth is more than just what happens to rich people. As much hoopla as surrounded slavery in the US they always maintained a small portion of the total population and per capita generated less goods and services than the average american.