First I stripped it, then brushed a heavy coat of red mahogany stain I've had sitting around for years.
Okay, I changed my mind. You provided new information that changed it. Stain is only good for about a year after it's been opened and up to two if unopened, and it's kept in a cool place that does not reach freezing temps. I was wondering about the uneven staining and I think that the stain has been sitting around for years accounts for that.
I can't explain all the science behind it, but basically, the ingredients degrade over time and the solvents that hold the stain in solution break down. Birch is a very tight fine grained wood and it should take stain fairly evenly since there are not a lot of variation in the grain. The mottled appearance of your stain job, I think, is because the stain was old.
Reason #2, mahogany would not be my first choice of stain. If you like mahogany, than that's fine, but it's not a traditional gunstock wood so there's that.
With this new info, I'd strip it again and use a new can of stain. You have to be patient with stain, especially with dense woods that take a while to drink it up. I think if you redo it, in walnut, or whatever you like, you will get a result you'd be more satisfied with.