I'm glad you came here, it is a puzzling and anxious time for you. Your situation will have been recognized by many here, trying to build a relationship with someone who does not talk is hard and can of course go two ways.
From what you say doing the things you are doing now is not enough, OK you see you are being followed, but there is little else.
In any friendship - or something deeper -the two people have to talk, for enjoyment, let each other know their thoughts, and umpteen other things. Without that there is a great temptation to 'imagine' what the other person is like, and that may be completely wrong.
I know it will be a difficult thing to do, and the thoughts of a possible rejection very daunting, however do you think you might write a letter or email, setting out what you would like and asking how he feels about it - and does he want the same.
Plain speaking may well help you both