Talk about late to the party but BINGO!Late to the show here, but out of interest, after the first two shots taken are the next fliers generally in the same direction/poi? There are a lot of variables that could be in play here, but a barrel machined in a manner that has an outside diameter not concentric to the center of the bore can show this particular symptom. The same goes for fluted/ribbed barrels if the flutes cut are not concentric and equal in depth to center of the bore. Basically, as metal gets hot it expands, and if more material on one side of the bore than other it takes the path of least resistance so to say and moving the point of impact. I used to manufacture barrels and guns, and have seen it before in testing, first few rounds hit the bulls-eye dead on, then a jump to the right in the pattern or whatever direction it wants to go. It usually takes some heat for this to show its head, and varies by the type of barrel, dimensions involved, firing protocols, etc. Heck, I remember the military mentioning one time that barrels sitting out in the full middle eastern heat/sun of summer in their vehicles, could sometimes have a different point of impact from the earlier bore-sighting/testing procedures they had done when cooler.
Anyway, if your problem is related to that possible barrel mfg issue, there is no real fix... good luck to you.
Drinva, try this to test your rifles barrel seat to the receiver: Fire 5 shots to thoroughly heat your barrel, then shoot a group. If you get tighter groups without the flyer, then what AZ stated is correct! Shoot your warm up shot quick and when shooting your group, try to shoot your shots within 60 seconds or less of each shot.
I've spoken at length with Adam at RPP regarding this and re-seating the barrel to the receiver is one of the first steps he does to accurize a Marlin. Adam can explain what all he does to make a more perfect barrel to receiver seat. Then what some of the other posts suggest come into play.
My 2007 308MX is soon heading to Adam for a similar problem. During summer heat, the cold shot prints nearly two inches high, 2nd shot 1 inch high, 3rd and 4th shot gets the barrel hot enough to expand enough for the barrel to receiver seat to stop moving and it's ready to shoot a decent group. At present it's unacceptable.
Oddly enough, during winter temps, it shoots tighter groups from the 1st shot due to less heat moving the barrel against it's seat. Obviously, less heat equals less barrel movement.