GreyMustang has the root of it.Think of recoil, the longer it takes the bullet to exit the bore, the more time the bore has to rise. So generaly speaking a faster load will shot lower, on a rifle with sights set for slower loads. Somebody is bound to get real technical, cuz there are exceptions, or situations that can be set up, but this is likely what your experiencing.
Its not really dropping more. The effect your seeing is the differences of the barrel angels when the shot exits the bore.
Even if the recoil of the faster light bullet load seems less, the same or greater, remeber the light bullet at faster speed exited the barrel much sooner than the slower heavy bullet load, and recoil effect (continued muzzle rise) continued long after the light bullet exited the barrel relative to the heavy bullet.
Faster bullets exit the muzzle faster and earlier in the recoil cycle. They shoot straight.
Slower bullets exit the muzzle much later in the recoil cycle leading to higher shots.
It seems counter intuitive until you think it through.