It is normal to have worse vision without glasses because your eyes get used to see clearly when you are wearing your glasses. When you remove the glasses, your eyes need to adjust again to find the focus without the glasses' help. So your vision seems to be worse than wearing glasses.
Wearing glasses won't make your vision worse. It is because you often overuse your eyes which make your eyes too tired. The excessive eye fatigue is the main reason that cause the vision become worse. So you should relax your eyes as often as possible even if you wear glasses.