No it is not normal.
And no obvious fix for your problem comes to mind, however, I'll go down the list. Sounds as if you might be doing something different between loading singles, and feeding from the magazine. I'm shooting in the dark here, but maybe something will help...
Make sure the screws in the receiver and the fore end cap are snug, as well as the scope mounts and the rings.
How are you supporting the rifle while firing? Firing from a bench? Standing holding the rifle?
If firing from a bench, make sure the fore end is supported on something soft, a back pack, a roll, or at least your hand.
Make sure the fore end support is at exactly the same place for every shot.
Give the rifle plenty of time for the barrel to cool between shots. Check the barrel for temperature with your hand.
Make sure you are sitting in a comfortable position, sitting up straight, and not all scrunched up, and with the rifle at a comfortable height for you.
Use a good recoil pad, or padding between the butt stock and your shoulder.
Try another brand of ammo. Jacketed bullets are usually more accurate than cast.
Give proper attention to squeezing the trigger.
Remove your sling. Make sure your rear sling stud isn't recoiling into your rear sand bag.
Try mounting a different scope.
Have a friend shoot a group or two.
One of the last things I'd do is remove the fore stock and check clearances between the wood and the barrel and magazine.
I hope someone else can suggest something more specific.