From Ladyknight link we have a winner for the brain hurt thread, "The most common reason cited by blacks for not knowing how to swim, however, was a fear of drowning."
This explanation may be specific to the towns my students are from.
We talked about this in the teacher lounge - after a new teacher asked a veteran teacher, who is black, about a similar comment from a child. (I'm NOT going to swimming lessons (4th grade PE) because blacks don't swim)
She explained back in Jim Crow era blacks didn't have access to pools/swim lessons. They only really had access to the somewhat murky Brazos river. Going to the coast was also a problem because of segregation. So many blacks didn't learn to swim then if they ended up in deep water (like falling out of a boat) they were more likely to drown. Which lead to superstitious beliefs that they 'couldn't' swim.
It wasn't to the mid 70's that the pools in the area were truly socially intergrated (though legally they had been for years). A good number of our students have grandparents and great grandparents who can't swim and may pass a fear of the water to the children because of that inability. It is dying out. In this case the veteran teacher go the family to allow the child to take swim lessons.
I've heard of a similar mentality among the fishermen, where mom grew up. Water was for working not playing, so many of them drowned after being knocked or washed off deck. There the fear was expressed as The sea was angry at people for playing in here.