This isn't the answer you'll be looking for, but, to further understand this, you're approaching it from the wrong angle.
Compensation patterns are rarely or probably never down to one particular muscle, or even muscle group. You need to look at the bigger picture - think 'movements, not muscles' - what movements are you doing incorrectly. Figure this out FIRST, then get specific.
If you think your feet / ankles are the problem, then the movment your talking about in an OH Squat is dorsiflexion (toes to shin ROM), so - test it. If it is, then there's lot's of things you can to to fix it - foam roll and stretch the calfs and plantar fascia, mobilise the ankle joint (work on dorsiflexion movement specifically), and, strengthen the dorsiflexors (front of shins).
If you think it's your upper back, then again, what MOVEMENT? In an OH squat, specifically, your talking about restrictions in Thoracic EXTENSION (bending the upper spine backwards, really). But regardless what muscles you think are holding it back, you can expose more weak movements - scap retraction, depression, and upward rotation. Train those movements, and the muscles that need to work, will work.
The bottom line is, with these example, the faulty movements mentioned is ankle ROM, and thoracic ROM. Train those movements, and things will improve. If you try to narrow it down to one or two specific muscles, you'll just get frustrated and lose motivation. Normally the solution to these things is actually very simple. The process of finding the solution can be tough, but the actual solution in itself is generally a couple of movements and stretches at the most....
If you're really interested in this and want to hit the nail in the head then I would advise you get "Athletic Body in Balance" by Gray Cook. It includes a "Self Movement Screen", which includes the OH squat, as well as progressions to being able to do it. The approach is suprising but it's very effective.