I would start from the fact that software production is an industrial, long-term endeavour: just think reuse (aka configuration management), which is only one component of a whole picture, and already that should remind not only the level of complexity (complexities!) involved, but, even more importantly, that an efficient and highly successful production unit (not just a successful project) is not an occasional target and cannot be! (Indeed, that is why it was called "development".)
Currently, what I see is a general trend of naming "agility" what is actually a quest for the ultimate (worst) RAD and the ultimate (over-)simplification: and that is the clash of industrial vs. commercial [but, more and more so, also the overall economic and financial] cultures and strategies. What has happened starting with the so called democratisation of programming is, so to speak, that our customers have taken our place! In practice, building software has become feasible to everybody, with the result that most technical offices are literally up-side down, and the whole marketplace has been seriously [if not irremediably] diluted.
But, as noted initially, software production is a most complex engineering and requires highly sophisticated and specialised competences, to the point that plain common sense (the "magic recipes", invariably just out of the social-mediatic hat) just will not cope. Indeed, software engineering is the *most* complex engineering that there is, to the point that the majority of its fundamental notions are genuinely counter-intuitive: as a result, plain common sense not only is utterly inadequate, it is actually doomed to failure.
Bottom line, I believe most of today's agility is a cover up for what is in fact pure plain RAD, and the software crisis is a myth and a spell that can be (easily, to say) dis-solved: just there is an essential distinction to make, between proper and improper software production, with the vast majority of the software endeavours nowadays belonging to the latter category. Then, and just to begin with, the very shape of the industry-specific performance surveys would change... In fact, I would claim that proper software developers (and, scaling up, proper software production units) do, already today, consistently provide near to optimal results, all environmental circumstances considered!