Swimming with contacts can result in eye infections, irritation, and potential sight-threatening conditions such as a corneal ulcer. So I don’t think it’s a good plan.
On your own risk. There are bugs and pathogens that chlorine doesn’t kill, which could potentially cause damage to the cornea, infection, or ulcers. You can wear goggles to protect your eyes.
You don’t have to wear glasses in a swimming pool cause you don’t need to see very clear under such circumstances. But if you don’t like seeing things vague and reply on your contacts, just remember to take them off very quickly after the swim.