My guess is bedding. Barrel oscillations move the barrel,or the recoil stress moves the receiver slightly,and the barrel/action settles into a different place. Only a few .0001's of an inch will move the hits. Of course,it could be ammo,since you don't reload. Another firearm mystery to be solved. You are talking bolt action rifle here?
I've bedded every bolt gun I own with Acraglas Gel. That includes the triggerguard and floorplate,but the magazine box is a teeny bit loose,so as not to stress the receiver.