Yes, the pool water will damage your sunglasses because of the acid material and other damaging materials. When you swim, you'd better wear the swimming goggles. Or else, your eyes will be infected by the bacterium in the pool. You should also wash your eyes after the swimming which will promise your clear vision. You'd better not forget these points.
Ok, I can see that you like your lovely sunglasses very much and are wondering whether the pool water would decay your sunglasses right? Swimming pool water could not decay your sunglasses, nor could it make it less shining. However, if your sunglasses are of great value and are specially made. You must take care of them, lest the water could decay them and make them less useful. Of course it cannot wear them in the water, try to get a pair of swimming goggles over eBay at a low price.
Yes of course you can go to swim with the sunglasses. If you take good care of your sunglasses to prevent it from scratch or damage, your sunglasses will be stronger than your skin. If your skin would be safety contacted with the pool water, why cannot the sunglasses? If the water is clear and without impurities, you can go to the pool with your sunglasses. Please keep your glass clean and softly sorbs the water on the surface of the sunglasses. Make sure not to erase the glasses, because the anti-ultraviolet coating film would help your eyes from the damage of ultraviolet. You must get rid of the water after the using of sunglasses and keep it in a dry place to save for the using of next time.