Alright I'm 26 years old, 5' 11'' male. [...] and have been pushing to get down to 180 - eventually 170 lbs. .
I mean why a certain weight...
I can probably answer that. 180 at 5'11" is a BMI of 25. His doctor probably did the calculation for him.
In my opinion, BMI of 25 is probably optimal for most people. You shouldn't try to get any lower. Better is to try to add muscle and wind up at a muscular BMI of 25 or more. 196 is a little heavy but not so much that I would focus on weight loss at the expense of muscle gain. A lot depends on your bone structure.
Are you seeing a physio therapist for your knees? You should be strengthening the muscles around your knees to support them. Your aversion to heavy lifting may work against you. What do you do in the gym?
I noticed you're eating bread at both breakfast and lunch. As Bob mentioned, you probably want to reduce your bread intake. (I'm sure some people here call me the Grain Nazi but I'm trying to be restrained.)