But if your goal is to DL 400# one day, do you really get there faster by spending more time with lower weights?
In my experience, the answer is yes. I've seen more people stall out trying to race to a heavy weight than I have seen people stall out by creeping up on it and getting as much as they can out of the lower weights. If you could get strong just by lifting heavy every time you hit the gym plus throwing in some accessories, gyms everywhere would be full of super-heavy bench pressers. They aren't, because once you exhaust your linear gains, you need varied loads and periodization - and 5/3/1 is just another example of varied loads and periodization.
Besides, you do get to pull heavy on 5/3/1. That 1+ day, even when you start, is 95% of 90% of your 1RM. Within a few cycles (very few at the low end), you're pulling over 90% of your one rep max for multiple reps. That's pretty heavy, it's just mixed in with lighter stuff that'll get you strong in ways that a lower volume doesn't get you.
Anecdotally, I have a guy who wants to DL 600. He's gotten 550. He's on 5/3/1 and pulling lots of reps in the 435-495 range right now. 495 is, what, 80% of his goal 1RM and 90% of his previously tested 1RM. So he is pulling heavy, one week out of every 4, and close to heavy on two other weeks. He'll be back to pulling 90% of his actual goal 1RM on 1+ day pretty soon. He might not hit 600 when we test, but he'll demolish 550, I'm sure, because he's pulling close enough to that to matter over and over and over again. Just with lots of rest in between.