The entire coastline where the waves break on the land is like a giant skimmer, so that is where all the contaminants are superconcentrated. However, if you have access to equipment that would allow you to gather sand farther out in the ocean, I don't see a problem.
However, there is always the possibility that you could introduce parasites or pathogens into your tank. Let me just say this... if I lived by the ocean when I started up my tank, I probably would have used sand from the beach.
ps- the sand you buy in the store is guaranteed to be high in calcium, magnesium, and other beneficial trace elements. The sand from the beach may or may not have these.