Considering that the total population of the United States in 1850 (including territories) was
23,191,876 I'm guessing that no, there were not tens of millions of slaveholders.
Heck, even in the south the percentage of actual slaveholders among the white population was small (small farms generally couldn't afford them). But the percentage of people who either owned slaves or approved of the people who owned them (and would have owned them if affordable) was high.