On the flip side, I gain significant muscle mass without specifically programing for hypertrophy. The down side of this is that if I dont work out, it all just goes to fat. I dont seem to gain or lose significant amounts of weight, but the amount of weight that I lift definitely goes up and my body composition really changes.
I question how you are judging that your muscle "goes to fat". Dense grainy muscle takes years to build. It isn't like you can lift for 6 weeks, look huge, stop for two months and look like Danny Devito. If you are talking about taking years off and looking fat. It isn't that your muscle "turned", it is more likely you kept eating like you were lifting heavy 4-6 times a week, and were actually sitting on the couch.
Hardgainer is a word used by people that don't eat enough. Caloric needs are different for everyone, and there are any number of reasons someone isn't eating enough, but the fact still stands. Hardgainer = not eating enough. Period.
Once you get past a certain point your daily diet will effect your performance, not just weight gain or loss.