It's a moving part with springs, clips, locks etc - plenty of stuff that can brake that can't on a fixed stock.
Eh - based on real-world experience, on any modern folding stock - if a piece like that is broken enough that it might affect accuracy, it's because the stock as a whole has become non-functional. But as far as reducing accuracy on a regular basis? Not a factor.
As point of fact - a high percentage of modern "sniper" or "designated marksman" type rifles actually utilize some form of folding or collapsable stock to reduce overall length for transport - and they certainly aren't seeing any reduction in accuracy or other issues because of it.