It's not just 1 that causes the damage. It is when a child swallows several at one time or over a course of time. They can suddenly snap together in the digestive tract causing internal injuries and in most cases death.
But i agree with you. If you are a parent you protect your child. Placing plastic plug things over outlets, cabinet hooks, ect. You don't leave dangerous magnets at toddler level.
Might as well leave a loaded hand gun on the table.