OK, I base my opinion here on two things: a couple of years of wearing kevlar in the US Army and alot more years designing and using armor for a medieval re-creation group (the SCA).
1. Armor actually makes you easier to hit and only mitigates damage taken.
2. The human body is generally not designed to easily accept another several centimeters of hardshell on it. This can me made more tolerable by having armor custom fitted, but that takes additional time/expense (in d20 terms, masterwork). While you can in fact do several interesting stunts (cartwheels, etc.) while wearing fitted armor, it is nevertheless very fatiguing and claustraphobic to wear, and most people have an almost uncontrolable urge to get the helmet and torso section off if they are not actually expecting immanent combat. This urge can be lessened with training, but for most folks, the weight and encumberance of armors is a factor in it's use.
3. Armor only mitigates damage. Armors can only rarely actually stop an assault on the body with no consequenses to the person wearing it. This is on the order of a tactical vest or helmet stopping a punch or swung baton. Anything more than this does increasingly more damage to the wearer, though not life threatening damage in most cases. The fact is that several pounds per square inch of pressure applied to any armor is going to bruise, cause muscle sprains, cause stress injuries on joints or organs, and otherwise be quite annoying to the wearer. But this damage is most often not immediately life threatening, though in some cases it is (impact related heart arythmia [sp], blood poisoning due to bruising, etc). Simply put, an opponent in armor requires some extra work to kill, but not that much more, really. Once you defeat the material technology, there is only a soft bloody bag behind it. And that bag bruises pretty easily. Trust me on this. I've had some pretty spectacular multi-colored beauties on various parts of my body all in 'fun'...
4. There are several portions of the body that are difficult to armor and are almost immediately fatal if penetrated, and severely debilitating otherwise. These are: the jucture of the neck shoulder (carotid artery), the armpit (there is only lung tissue between an impact and the heart and aorta at the armpit), and the groin area (femoral[sp] arteries). These areas are almost impossible to armor effectively, though deflective armoring techniques can dramatically reduce the dangers involved.
So, taking all that into account, the system that most realistically portrays armor use is, IMHO, the RuneQuest 3rd ed. system. Hit locations (and hit points) are divided up into Head, Chest, Abdomen, Left and Right Arm and Left and Right Leg. The heavier the armor, the more points of damage can be resisted. So, for example a bandit wearing Hardened Leather with Padding might have 3 pts of Armor and 4 Hit points in his Right Arm. So, in order to injure that limb, an opponent must first hit the limb and then do more than 3 pts damage. Once an individual reaches zero hit points in that hit location, it is useless until healed. HOWEVER, the body also has a given number of Fatigue Points and each layer of armor deducts from those FP. Once you pass zero Fatigue, you begin to loose accuracy in your blows (a negative modifier to your Attack rolls). Once you reach negative-the-starting-value (you have 33 FP, and you reach -33), you are exhausted and may no longer fight.
But all this adds alot bookeeping and more steps in a combat round...
So, as usual, we are back to the gritty realism versus dynamic play arguement.
But there you go.