What are your goals to working out. If you are specifically looking for bigger muscles (bodybuilding as opposed to pure strength), then you are hurting yourself by not eating before a workout.
Studies have shown that consuming carbohydrates before a workout can modify your body’s hormonal responses to the training. In particular, glucose and insulin levels are above baseline which leads to an anabolic (grow) environment for your muscles. Further, the elevated glucose and insulin levels blunted the cortisol response (by 92%!) thus stopping cortisol’s catabolic (break down) activity on muscle. This resulted in a 19% to 22% greater gain in muscle tissue.
So, if you’re looking to build muscle fast, then you should have a pre-workout “meal”. A quick shake is fine. If nothing else, then have a glass of 1% milk which will give you both cards and protein. For even more carbs, put the milk in a blender with half a banana. Or, if possible, get a protein & carb powder and add it to your milk. That’s all you should need to see greater results.
Oh, and here is a link to one of the studies done on this.