In our opinion, the answer depends on if the lenses are new or not.
"New" contact lenses are stored in a sterile solution while "opened" contact lenses are supposed to be stored in a disinfecting solution. Disinfecting solutions are recommended to be stored and used at room temperature. Studies have shown decreased effectiveness of these solutions at higher temps and even though they didn't discuss low temps, just the idea that their effectiveness can be altered by ambient temperature makes us apprehensive.
Provided the internal temperature of the refrigerator is not cold enough to cause even a slight phase-change of liquid, the lens structure itself will be fine.