Usually it will not cause a problem, while it can iritate your eye temporarily, you need to wash it out soon. But try not use your fingers becasue you can get an infection in your eye if there is bacteria unless you have contacts. And if you put your contacts in, you need to see your optometrist immediately or you can get a serious infection.
You needn't worry, that won't harm your eyes. But a little bit discomfort. Actually, it can make your eyes more moist. I understand the blurred vision make people feel bad. So followed the steps I listed, you'll be fine.
Clean your eyes with pure water or physiological saline at first, then blink your eyes to make the oil gathered at the corner of eyes, rewash your eyes again, use artificial tears to relieve your eyes in the end.
Well, in my opinion, you do not need to worry about it and it can be fine. And in the first, step, you can just have some pure water and wash your eyes gently. In that way, it will just wash the oil out of your eyes. And then, you can also have some warm cloth to weep your eyes. And that can just clean your eyes more carefully. And then, just be relaxed, and it can be fine. But if you still feel uncomfortable, just go and see the eye doctor.
I use Vitamin E oil to remove makeup from my eyes and to make my eyelashes stronger before I go to bed. And whenever I do, I get Vitamin E oil in my eyes. The downsides are that it'll make your vision blurry and might make you tired (albeit that isn't a problem for when I need to go to sleep quickly--it helps with sleep I find but that doesn't mean you should go putting it in your eyes), and if you don't use artificial tears or saline to rinse out the oil, it might stay for over 12 hours. So that being said, I don't think a little Vitamin E oil in the eyes is something to worry about. Hope this helps. :^)