Respect that guy.
He is a real virtuous creature. Staring at women is not bad at all, but if he keeps staring at you, even after you noticed him ogling is really bad. This makes you uncomfortable and this is something wrong from the guy's side.
So, if a man turns his face immediately when a woman notices him staring at her is somewhat a good quality. This is really something cute from his side. Nothing to feel bad about it unless he keeps staring at you like he is gonna eat you up when you are alone.