Git lens hover takes 3-5 seconds to appear on every line

GitLens blame hover information takes 3-5 seconds to appear, making it useless for quick code archaeology. This happens even on small repos with fast SSDs. The delay seems to be caused by unnecessary API calls to fetch contributor avatars on every hover.