Implementation Driven Development

"I just write code that works, theory comes later."

[WORK IN PROGRESS]

1. Implementation Driven Development, IDD for short, is a software development methodology. ^{[1] [2]}

1.1. The discipline is formal (as opposed to agile, i.e. informal).

1.2. The process is evolutionary (as opposed to e.g. iterative: the immediate difference being that evolutionary is ideally continuous, though it has milestones, too).

1.3. The aim is verified highest-quality software at the lowest cost!

2. Development starts with specification and proceeds top-down (logically, in practice with feedback loops) to implementation and eventually deployment.

2.1 All artefacts ("deliverables") must be runnable, starting with specifications.

2.2. All artefacts must satisfy the formal requirements of soundness, correctness and closure.

(TECH: Strive for *no bugs*, rather keep closure by wrapping *every hole* into an "exception" - that is a "todo" in documentation.)

...to be continued...

[1] A precursor is in the rationale accompanying my TicTacToe project (2007), where for the first time I have mentioned the idea and the name publicly.

[2] A concrete example is the progression of my experimental constructive logic solver (2024), see the revision history in particular.

[CREATED 2023-05-08 18:41 UTC+2]

Collective disaster

"Your code might not be as valuable as you think."
Philomatics, The Value of Source Code, YouTube
Subtitle: Your code is worthless

The video is as cute as it is totally misguided. Not all code, just the poorly thought out and written code that is ubiquitous nowadays, in all guidance as in all practice, is actually a cost, not just worthless: a generally unperceived cost since that is 99% of the industry by now, indeed the very rationale that founds and sustains it is that that is the best we can do.

The fundamental misconception that the problem domain and the solution domain map one-to-one is but one ingredient of that yet-another global corporate-driven fraud and collective disaster.

Against Test-First

Against Test-First in synthesis:

CONS:

• negatively affects product quality:

• obliterates technical analysis/specification
  (which is the fundamental aberration, in theory and in practice)
  => conflating problem with solution domains;

• obliterates the separation coder/tester
  (akin to validating data models against the training sets)
  => conflating QA with the methods of production;

• negatively affects product/process development:

• obliterates external/internal distinctions (up to the worst RAD)
  (yet what we develop is *what allows building* what we sell, not directly it: whence, together with reasons of intrinsic and utmost complexity, software production driven by the commercial/PM/QA functions, once upon a time rather auxiliary, is just doomed)
  => no development proper: no consolidation of assets, nor of (sane) processes, even less of competences and the chance itself of acquiring those competences;

• enforces the gradual local change (up to TDD)
  (yet the normality hypothesis for program code is false)
  => no restructuring, no global perspective;

• glorifies and institutionalises the trial and error
  => no improvement, no global sanity;

PROS:

• none known so far!

See also:

• the notion of complex system up to that of phase transition in particular;

• the need for "radical (global/strategic/discontinuous) change" vs "gradual (local/tactical/smooth) change": the history and need for BPR over TQM specifically.

[CREATED 2023-04-11 21:49 UTC+2]

Commercial vs production functions and projects

The commercial function, among other things, sells products and services to customers. The production function develops and keeps alive an internal system of competences and technologies which, among other things, enables delivering the products and services sold by the commercial function.

Commercial projects (aka external projects) should be driven in conjunction by a client account for the commercial side and a production analyst for the technical side (meaning roles and functions more than specific people). On the technical side, among other things, the analyst should deliver a requirements analysis which is the basis to place then follow any order to production.

On the other hand, production is strictly technical and wants full autonomy. To begin with, because what we develop is way more than just what we sell, hence the need for an autonomous and strictly technical management and resourcing of production projects (aka internal projects). And, at least as importantly, because, as with all production of complex systems, engineering facts about software and systems are for the most part so utterly counter-intuitive that good old common sense, even the good old common management sense, here is actually worse than just ineffective.

<rant>
Indeed, production driven by the commercial functions is certainly doomed, both on the production side and on the commercial side. And production driven by product or project management is a common incarnation of such an upside-down (dis)organisation. Not per chance 90%+ of software projects keeps badly underperforming. Nor this kind of (mis)management has anything to do with agility proper, which is rather and expressly about a lowest level of formality, certainly not about an upside-down flow of (in)competences and (ir)responsibilities.
</rant>

Code monkeys

Monday morning, skimming UK software job ads:  I have ended up writing an article on "the sorry state of an entire industry": Code monkeys (LinkedIn/Pulse).

The source of all certainty

"If anything can go wrong, it will" is a certainty!

Corollary: there is no uncertainty in engineering.

Hint: containment constraints (aka closure).

Definition of Agility

As far as technical definitions go, agility is the antipole of formality along a methodological dimension. Specifically, agility is "low formality", where by formality we mean high level of control, extensive documentation, and heavy bureaucracy.

Note that the agile-formal dimension is orthogonal to the process models, such that we can have agile waterfall, agile iterative, etc. just as we can have formal waterfall, formal iterative, etc. (Corollary: there is no such thing as agile vs. waterfall!)

Note that the agile-formal dimension is also orthogonal to the cowboy-disciplined dimension, such that we can be agile or formal, still we are required to be disciplined. (Indeed, this was the greatest achievement with agility, that before it we only had "cowboy" vs formal!)

Competitiveness in production

K   :  Competitiveness     (aka Performance)
Ea  :  Efficacy
Ei  :  Efficiency
Qp  :  Quality of Product
Qs  :  Quality of Service
T   :  Time
C   :  Cost

Ea  =def  Qp * Qs
Ei  =def  1 / (C * T)

K  =def  Ea * Ei  =  (Qp * Qs) / (C * T)

The definition of competitiveness is common to all kinds of engineering, but with software (in fact, with all knowledge-related production), contrary to what happens with conventional engineering (the "hard" kinds of production), efficacy and efficiency are directly proportional. In other words, and contra the usual assumptions, with software, higher quality goes with lower costs. In symbols:

Ea ∝ 1/Ei for "hardware"
Ea ∝ Ei for "software"

On the "software crisis"

I would start from the fact that software production is an industrial, long-term endeavour: just think reuse (aka configuration management), which is only one component of a whole picture, and already that should remind not only the level of complexity (complexities!) involved, but, even more importantly, that an efficient and highly successful production unit (not just a successful project) is not an occasional target and cannot be!  (Indeed, that is why it was called "development".)

Currently, what I see is a general trend of naming "agility" what is actually a quest for the ultimate (worst) RAD and the ultimate (over-)simplification: and that is the clash of industrial vs. commercial [but, more and more so, also the overall economic and financial] cultures and strategies.  What has happened starting with the so called democratisation of programming is, so to speak, that our customers have taken our place!  In practice, building software has become feasible to everybody, with the result that most technical offices are literally up-side down, and the whole marketplace has been seriously [if not irremediably] diluted.

But, as noted initially, software production is a most complex engineering and requires highly sophisticated and specialised competences, to the point that plain common sense (the "magic recipes", invariably just out of the social-mediatic hat) just will not cope.  Indeed, software engineering is the *most* complex engineering that there is, to the point that the majority of its fundamental notions are genuinely counter-intuitive: as a result, plain common sense not only is utterly inadequate, it is actually doomed to failure.

Bottom line, I believe most of today's agility is a cover-up for what is in fact pure plain RAD, and the software crisis is a myth and a spell that can be (easily, to say) dis-solved: just there is an essential distinction to make, between proper and improper software production, with the vast majority of the software endeavours nowadays belonging to the latter category. Then, and just to begin with, the very shape of the industry-specific performance surveys would change... In fact, I would claim that proper software developers (and, scaling up, proper software production units) do, already today, consistently provide near to optimal results, all environmental circumstances considered!

Agile vs. waterfall?!

[This post in the Software Craftsmanship Google Group (*) has cost me being banned from the group: I am reposting it here to denounce how compromised the situation is and to keep the discussion open.]

Agile vs. waterfall is another one we should clearly stand against: it's just the marketing of a false dichotomy, but it is as wrong and a damage as it can be because it promotes massive misconceptions not only about the software process but about the very state of the art.

Rather than setting up primary schools on the art of tying shoelaces, we'd better focus on getting the basics straight.

(*) Agile vs. waterfall ?! (Software Craftsmanship Google Group)

To keep it simple you need to be smart

To keep it simple you need to be *very* smart.

Corollary: simple and stupid is just stupid.

Hint: software engineering is the most complex engineering that there is.

Essential software cycle

One step back, two steps forward:

1. Deconstruct

2. Reconstruct

3. Extend

The essential software cycle.

The old school

Once upon a time we had:

1. Do it properly.

2. Keep it simple.

3. Follow the rules.

The old school of engineering.

Software craftsmanship

"Crafting the details" to me evokes hours spent filling in the little gaps and cleaning up, while rethinking the whole thing over and over again. Hours when few lines of code get written, yet the overall quality and understanding improve dramatically. I was thinking how much I value and do it...

Software craftsmanship is when we
spend at least 20% of the project time
crafting the details.

Posted to the Software Craftsmanship user group:
http://groups.google.com/d/topic/software_craftsmanship/5AEub97Mr2k/discussion

I am a software professional

[Draft manifesto, partly in response to the Software Craftsmanship manifesto.]

I am a software professional:

1) I build software that solves my customers' needs, explicit and implicit.

That is the only software I will qualify as "working".

2) The software I build not only works, it keeps working.

Always around the optimal balance between minimality and continuous improvement.

3) I build software with honesty, responsibility, vision, and passion.

My customer is king and I will tell him when he is wrong: no excuse, rather quit.

4) I believe in professional integrity and real competence.

This is no place for unscrupulous marketers and incompetent wannabies.

I am a software professional, and I am damn proud of it!

Why the Software Industry is sucking more and more

With inflation and pollution come ignorance and even more confusion...

To begin with, the A in the ABC of software production:

It is not about doing it right, it is about doing the right thing!

Software is about the conceptual side and only marginally about the technicalities. About using our mind, sensibility and intelligence to provide an effective and reliable product and service to people.

This is certainly clear to our customers: Information Technology is a wonderful tool, yet an enabler, never a driver [except that even this is now failing, courtesy progress up to Industry 4.0 (2023-11-09)]. This is seldom clear to present-day software managers and even developers, who should rather realise that the matter in question is predominantly conceptual even at the lowest levels.

Yet another argument: against test-driven

Mr Jason Gorman, in his post "Test-driven Enterprise Architecture" (September 16, 2008) [*], writes:

I've discovered that the most effective way to align business goals with the software and systems we deliver is through heirachies of tests, not through pretty pictures and dotted lines.

I have to disagree, essentially. It is certainly true that most of those dotted lines get drawn without criterion, which is indeed what makes the difference between an *analysis* and a bunch of dotted lines!

On the other hand, tests cannot drive solution development: the problem domain and the solution domain just do not map one-to-one to each other, rather the latter is a sub-product of the former. In practice, as information engineering teaches, the solution domain emerges out of the opportunities for automation identified within process and data models at the business level. Here is a distinction, just like the distinction between syntax and semantics, that must not be neglected, otherwise not only the understanding but the very quality we are after is gone, with all that in fact counts.

[*] http://parlezuml.com/blog/?postid=693

If I jump into a Ferrari and... run away?

OK, let's suppose somehow I get around the boxes of an F1 circuit at the time of a race: difficult surely, but if you take into account a bit of luck, pretty doable.

Let's then suppose somehow I manage to find an F1 car in the proper conditions, that is as much ready to start as needed: so difficult that, again, with a bit of luck, it is definitely possible.

Let's finally suppose I manage to start and go: not so difficult, especially if I have *trained* for hundreds of hours in a proper simulator...

...brooOOEBY*&CROXZYQNRX8-7pVWOaei72)D9####F[---

Wow... let's assume I wake up from a crash at the very first turn after the start. Hmm, what has gone wrong?

Flash-forward, watching TV in London: in a show called Top Gear [*], a journalist is to have a try at driving an F1. He has some previous *experience*, so they first let him try a lower formula, so that he can prepare to the F1 experience. He is not that fast but quite comfortable on the F3, but at his first attempt on the F1, that is, after he manages to start the car, he is not even able to go as fast as to keep tires in temperature. At his second attempt, and not before some magic from the mechanics, he does it: funny driving, anyway he is able to complete his two laps with no more damage than a couple of testa-coda in the smaller turns.

Bottom line: simulation doesn't sum up to competence, and I'd rather go into journalism than try some more training.

Indeed, do not steal a Ferrari, just *** it.

[*] Richard drives a F1 car round Silverstone - Top Gear (YouTube)

On readability -- Pure induction in action

Radical engineering principle: on readability

Memento:

Engineering principles are true by good convention.

Principle:

Optimal "readability" is readable _optimality_.

Comment:

Never ever optimize. <*>

Do make code readable. <**>

Clean up the dark side. <***>

Readability before all. <****>

The only way to optimal code. <>

Extended:

<*> Includes refactoring, the semantic side of optimization.

<**> Includes restructuring, the syntactic side of readability.

<***> Includes comments, the dark side of readability.

<****> Includes the dark side of optimality.

<> Readability dis-including itself.

Hint:

Optimization of a sub-optimal program (algorithm, system, product, procedure, argument) is a pointless hack that leads to solutions at local minima. Improvement of such local-minimal solutions is impractical, given that the underlying landscape is so redundancy-reduced as to result completely chaotic.

In practice, if a sub-optimal program gets optimized (once upon a time there was the blessed post-optimization-upon-need, nowadays we have optimization as an integral programming practice called refactoring and its cousin performance), any required improvement needs stepping back a number of iterations until another approach, more basic but broader enough for the extended problem, gets in sight. The more premature the optimizations, the more paralyzing the sub-optimized local-minimal results. And the iteration starts over...

To break the deadly cycle, just stop optimizing in any form, start instead striving for readability. To get a basic idea of readability, we can think of the optimal target as a program code that can be _read as easily_ by a professional in the field as something in between a technical article and a page of mathematics. That is, code must speak for itself!

[A side note to the casual reader: please don't get too strict on terminology, I'm using all terms in their _broader_ sense, so that, for instance, should you be a software engineer, where I say "coding", I actually mean any act of production, including all that there is from analysis to delivery. The same holds for other disciplines. The only restriction is maybe in that I try to stay within the bounds of "engineerings" and "applied sciences", although, sooner than later, we'll catch up with "social sciences"...]

That said, apart from avoiding the inconveniences of optimization (taken in its broadest and weakest sense), readability brings its own distinct advantage, indeed an exclusive property: readability and only readability leads to optimal programs. We still miss a formal proof, but this might be apparent if we think what an _optimal_ program is: on a side, readable entails manageable so much as the latter requires the first; on the other side, every _ideal_ relates to needs that, in any ultimate sense, are "human" and only "human". Indeed, "who" needs _what_?

[For example, say one is programming and keeps readability in mind.
That means:
1) writing code with readability in mind (rather than "writability");
2) designing a system under permanent (re)structuring, down from its very (im)permanent foundations;
3) keeping the whole environment always clean and tidy, as if important guests are to arrive at any moment soon (and that includes documentation; yes, please, if in doubt, just DROP IT ALL!!).
That leaves plenty of room for movement, and the code stays always around the optimal balance between minimality and continuous improvement.]

We can't predict the future, we make it: we are pure induction in action.

I need money!

Only £4K/month! Plus, the more you are the more you share!!

Please leave a comment here with your email address, a reasonable explanation why you would support, and your maximum amount: comments are moderated. I will then publish any details you may ask, including yours.

Thank you!

On correctness -- Correct is good!

Radical engineering principle: on correctness

Memento:

Engineering principles are true by good convention.

Principle:

Correct is good! Period.

Comment:

As far as it passes systems testing, a system is "correct", and the problem becomes "good" testing.

Extended:

We do solve the halting problem in everyday life, though asymptotically: up to the next failure.

Hint:

Consider the full set of systems for a given problem: IS* = S + oo + 0/0, where S is the set of bounded correct systems, oo is the set of full-correct systems, and 0/0 is the set of failing systems. IS* so defined can be treated in terms of interval arithmetic.

References:

G.W. Walster, "Introduction to Interval Arithmetic", 1997 (slightly adapted):

The interval paradigm has a number of valuable characteristics, including:

1. Fallible data can be represented using intervals.

2. Any arithmetic computation can be performed with guaranteed bounds on the set of all possible results.

3. Bounds can be provided on errors from all sources:

• observation or measurement errors;
• modeling errors;
• machine rounding errors; and
• interactions among all of the above and the inherent instability of the problem.

4. Problems can and have been solved that are otherwise thought by many mathematicians, scientists and engineers to be in principle unsolvable. Exemplars include:

• nonlinear systems of equations; and
• nonlinear programming (or nonlinear global optimization).

For additional information on interval arithmetic and how it can and has been used to compute guaranteed bounds on answers to otherwise unsolvable problems, see:

• http://cs.utep.edu/interval-comp/main.htm [*].

[*] As an alternative link: http://web.archive.org/.../walster-papers.html

Credits:

Thanks to the invaluable support I got from the sci.math group (no endorsement implied): Computing infinity modulus and zero modulus.

On refactoring -- You should clean up!

Radical engineering principle: on refactoring

Memento:

Engineering principles are true by good convention.

Principle:

You shouldn't refactor, you should clean up!

Comment:

You can refactor when finished, i.e. never.

Extended:

Clean up before and after yourself.

Hint:

You can be many, but you can't be one.

Italian Agile Day 2007

agileday.it (2007)

Great event in Bologna, with some 300 participants and sessions going for the whole day across three different tracks, from classic sessions for beginners, to open space sessions for advanced discussion.

Very interesting relations and ideas were exposed, with most of the speakers sharing real life experiences. Most of the discussion revolved around how to introduce agility in a team, how to track agile projects, how to apply agility to legacy code, how to agile contracts, and even something about agility and ISO 9001.

I was very pleased to see how agility is gaining momentum even in Italy, despite our industry has been almost sinked in the last few decades, and the IT field is then suffering for the lack of serious engagement.

Below a personal list of concepts I have noted down for my future meditation and adoption (this list simply reflects my current state of thinking and professional needs; DEV=development, PM=project management, LGY=legacy):

1. DEV - Make stories/tests from bugs

2. PM  - Reduce the work in progress

3. LGY - Refactoring cycle: 1) Select; 2) Break dependencies; 3) Write tests; 4) Refactor

4. LGY - Test used/modified code only

5. Use a wiki for tracking

6. Use a Progressive contract as an umbrella/framework contract over multiple Fixed Price contracts for new development phases; use Time & Materials for maintenance and small changes (max 2 days).

The one thing I would criticize does not relate to the event itself, it relates to the content of most relations. Even Tim Mackinnon, the keynote speaker and well known agility pioneer, said it: you need OO technologies to be agile. IMHO, that is plain wrong, and we should simply be talking about component based development, that is applying the most basic engineering design principle: modularity. This is not a cavil, the nowadays widespread emphasis on OO technologies is making a lot of damages, mainly among the younger professionals, all in the name of this idiotic idea that you can ultimately solve problems with tools-based/frameworks-based development. Welcome to the shop...

TicTacToe 1.2.B released for preview

TicTacToe 1.2.B has been released for preview.

« Tic-tac-toe is a classic case in the field of computer science, though our primary goal here is to investigate software development concepts and practices, and infer engineering-level indications to formal methods and patterns of construction. »

Contents:

1. Iteration zero and beyond
2. Top-level design and source code
3. Next open issues

Read the full document here: TicTacToe 1.2.B / Docs

---

TicTacToe is an open project.
You are encouraged to
send questions and feedback, and
ask more about getting involved
to ThisTeam's box:
thisteam@julio.diegidio.name

---