Does your cat do this and do they ever grow out of it?

I think your cat just loves you and probably finds comfort in that particular piece of clothing that you have. If I were you I would be happy that you have a cat that adores you and give him lots of love back....and get him neutered because that will make him even more happy and healthy for years to come. Enjoy your kitty he sounds wonderful.