Validity & Veracity

I finally summoned up motivation to read The Annotated Turing by Charles Petzold. I bought the book in Jan2012. Ever since then it was a pending TODO. Now that I have time on my hands, I decided to attend to it while the grey cells were still active.

 From pg. 45. "In Hilbert's conception, the construction of a formal mathematical system begins with definitions, axioms, and rules for constructing theorems from axioms."

My Take:

• The regular expressions that are used by in the Lexical Analysis phase of the compiler are Definitions .
• The productions of the grammar rules, used in the Syntax Analysis phase, are Axioms. The output is an Abstract Syntax Tree (AST)
• Finally the Semantic Analysis phase is the one where the rules of the language are used to extract the operations that are implied by the AST. The output is an Intermediate Representation (IR). That is where the front end of a compiler ends. Subsequent steps are transformations to make the code execute on some CPU
• Thus the IR output of the compiler is a theorem constructed from axioms.
• The rules and axioms are coded in to the compiler.
• A compiler therefore qualifies as a formal mathematical system.
  
  

From pg. 46. " ... but the rules also imply the syntax of a well formed formula ... that is possible within the system. You can assemble a well-formed formula without first deriving it from your system, and then you can attempt to show that it's a consequence of the axioms by applying the axioms and rules in a proof."

My Take:

• When we write a program in a high-level language we in fact "assemble a well-formed formula without first deriving it from your system".
• Then when we successfully compile it we in fact show that "it's a consequence of the axioms by applying the axioms and rules in a proof"

From pg. 47. "Provability is a syntactic concept; it is based on the axioms of the system and the rules used to derive theorems. Truth, however, is semantic concept that depends upon the actual meaning we give to the symbols in the system."

My Take:

• Successful compilation proves Validity.
• Successful compilation does not prove Veracity (Truth).
• There is no getting away from well designed tests. Such tests are close approximations of a "proof" of Veracity.

The Hare & The Tortoise

The fable of the Hare & the tortoise has been used to draw lessons in competing and co-operating.

The fable also applies to software organizations.

The Hare Company.

• No prototypes are being worked on. (Hare Sleeping)
• No experiments being done to Cross The Chasm. (Hare Sleeping.)
• Wishful thinking about own capability. (Hare Dreaming.)
• RFP is received. (Hare wakes up.)
• Estimate based on myth.
• Mad scramble to bung in a business proposal. (Hare racing.)
• Loses Contract. (Tortoise wins. Hare team gets more dysfunctional.)

The Tortoise Company.

• Prototypes concepts that are likely to win in the marketplace. (Steady progress.)
• Experiment. Be ready. (Steady Progress.)
• Fast feedback loops. (Course correction. Prepare to undertake "Op Overlord")
• Has true picture of own Capability. Uses CMM as a "dashboard" instrument.
• Estimate based on track record.
• RFP is answered based on known facts. Risk minimized.
• Wins without breaking a sweat.
• Proceeds with Agile implementation of project.

Surprisingly, It is the Tortoise that is Agile!

Dilbert: Agile

Today's Times Of India, shows why an Agile process is needed. Will Pointy-haired managers get it?

Humility & Courage

Humility is not lack of confidence. Bravado is not courage.

I believe that Humility & Courage are essential traits for doing quality software work. The reality of software development teaches that. Refusal to learn this lesson is a root cause for poor software quality and dysfunctional teams.

Humility.
Humility means being aware that one could be wrong; that one's understanding has gaps. Humility leads us to always question our assumptions; to always put them to the test.

Defensive coding is humility. Assert statements check one's mental model of the state of execution of the code. It is accepting that one could be wrong.

Recognizing the primacy of unit tests that can be easily and frequently run, and provide appropriate coverage metrics, is humility. Adding tests that increase coverage metrics is humility.

Humility is a prerequisite for learning.

Courage.
Visually challenged persons are not be able to walk briskly. But they learn to progress by testing their way with the help of a cane. That is courage.

As software professionals we should realize that we are "visually challenged". Courage is "tapping" our way forward. The "tapping of the cane" are experiments that we must conduct to improve our understanding. These experiments have to be small steps.

Resolutely "tapping" our way to our objective is Courage.

Agile.
An Agile process is about Humility & Courage.

Prose & Code

The ABC of writing good prose: Accuracy, Brevity, Clarity.

The ABC of writing good code:

• Accuracy: A function should do one thing and one thing only.
• Brevity: Coding standard must specify the acceptable length of functions. If the function is getting long; Refactor.
• Clarity: 
• Track complexity metrics. Refactor
• Use names that convey intention. 
• Avoid globals.
• Write Executable Specifications.

Architecture of Internal Combustion Engines

As I mentioned in an earlier post architecture is about "What" gets done "Where".

Form follows functionality is a principle of the architecture of buildings that also applies to software architecture. I recommend reading the complete article.

Why am I talking about the internal combustion engine?

• Most automobiles, and motorcycles, have a 4-stroke engine. A 4-stroke engine has four processing stages - Intake, Compression, Ignition/Working/Expansion, and Exhaust. All done in the same processing space - The cylinder.
• What about Jet Engines? Jet Engines too are internal combustion engines. They too have the same four processing stages -  Intake, Compression, Ignition/Working/Expansion, and Exhaust. Except, in a jet engine these are not done in the same place. The processing is pipe-lined.

A jet engine is an internal combustion engine with a pipe-lined architecture!

Takeaway. Architecture is not restricted to buildings. Architecture is about the way work is organized.

Note: I assert my IP right to all of the above.

Management Objectives & Coder Objectives

Technical Debt.
Management has business goals - both short term and long term. The two need to be balanced. That is the art of management. Excessive focus on one, or the other, will lead to failure. The short term, by its very nature grabs attention; in most cases to the detriment of the long term. In software development this leads to the taking on of Technical Debt.

Consequences.
As Technical Debt piles up the code becomes increasingly fragile. Fixing one defect introduces other defects. Feature addition becomes a nightmare. Customers become increasingly dissatisfied. The brand erodes.

Causes.
Technical Debt 101 explains the factors that cause technical debt. I strongly recommend reading the article.

Concentrating on today's fires is exciting. Trouble is, the excitement is paid for by taking on technical debt. In this, management have very "willing allies" - the Coders. What are a coder's objectives? The most important are:

1. Improve job market prospects. Hence acquire skills, knowledge.
2. Ensure that they are not easily replaceable by their current employer. The higher the technical debt the greater the job security.

Practices required to prevent buildup of technical debt do not figure in the objectives of most coders. In the rare case that it does, management, by their actions, ensure that it is soon given up.

Software Exorcism is a book that deals with Technical Debt. The book, as it's title says, is about getting rid of the "evil spirits" that can take up residence in software. An evil spirit, in Hindustani, is called Bhoot. Once evil spirits take up residence the software becomes what, in Hindustani, is called a Bhoot Bangla - a Haunted House. Now there is a vested interest in not exorcising the evil spirits.

Exorcism: First Step.
Technical Debt must be managed. Managing anything - whether diabetes, or sports teams, or whatever - requires metrics. Managing financial debt requires financial metrics. Managing technical debt requires technical debt metrics. Managing technical debt will ensure a development process that,

1. Prevents unhealthy build up of Technical Debt. 
2. Takes corrective action to reduce it to defined acceptable levels.

The Conjunction "And"

I try and follow a rule when writing: If the sentence contains the conjunction "and" I see if I can split the sentence into two, or more sentences. An "and" indicates that I am trying to convey more than one aspect/purpose of an idea in the sentence.

I am an advocate of intentional naming. Intentional names can result in long names. If a function name contains an "and", it is an indication that the function is intended to do more than one thing.  It breaks the rule that a function should "do only one thing and do it well" - also known as the Hedgehog Principle.

Being In The Zone

Coding can be exciting. Especially when you are in the flow - in the zone. That's why we like to code. The total absorption in the task, I believe, is the same as being in a state of meditation.

I should know. Why else would I be coding in my seventies?

However, as with anything that we enjoy doing we must learn to do it in a disciplined manner.

In the zone one needs the discipline to :

1. Write the Executable Specification before implementing the code that it will test.
2. Use the idioms of team's coding standard. 
3. Spend time on thinking up names that convey the intention/purpose/responsibility of the files, objects, structures, functions, and variables.
4. Commit frequently to the version control system (I advocate using Git). Do a diff of the source files with the earlier commit. Document the reason for the changes in the commit log.
5. Refactor ruthlessly.

The discipline has to be practiced when in the zone. 

Easier said than done. I should know. Six-Pack Abs need discipline 😊

Why Did Some Dog Not Bark?

In an earlier post I had described as to how unit tests are "dogs that should not bark". Meaning that Unit tests run "All Green" as code is added, modified, refactored. When a unit test fails, prompt action is taken to get back to an "All Green" state.

One situation when "silent dogs" should be a serious concern is when the signature of a function is changed. That means unit test for the function is missing. That means TDD was not followed. That means a Defect in the Process.

Another situation is when a defect is discovered while inspecting the code. A little while back I had experience of that.

It was an expression that computed the radius of a circle given the length of a chord and the distance of the midpoint of the arc from the chord. High school stuff. The expression was correctly worked out in the explanatory documentation. But in the code I forgot to divide the expression by 2. I plugged the function into the module and merrily carried on. The defect was found by a member of the team that I was coaching. I asked for comments from the team. The best answer: "The dog did not bark". I had the satisfaction of knowing my coaching had been effective😊

I had not followed TDD. My Process Had been Defective. 😢

Lesson: No matter how simple the algebraic expression, use TDD to implement the function.

Validation & Verification

Validity & Verity
I am asked. "Did you have lunch today?"

I answer, "Monday."

That answer is invalid. The question has just two valid answers - "Yes" or "No".

If I give one of the valid answers, it does not mean it is the truth (verity). My answer has to be verified.

Validation checks for Validity. Verification checks for Veracity.

Valid Programs & Correct Programs
A C program that can be compiled without errors is a valid C program. Is it a correct program? By executing the program, it can be verified that the program meets its specification. It can be verified that the program is correct. Correct programs are a subset of Valid programs.

Validation Criteria & Quality
What if the compiler options are set to treat all warnings as errors? Now if there are warnings the program will not compile. The program is no longer a valid program. There is nothing to verify.

What if, to successfully check-in code to the version control system, the code must meet Complexity Metrics?

What if the code cannot be checked-in because coding standards are not met?

There is more to qualifying as valid code than just clearing compiler errors.

If you want to improve code quality, enforce stricter validation criteria.

Beautiful Architecture

People eat, sleep, entertain themselves, meet biological needs within physical structures. These structures could be student hostels, barracks, small flats, large flats, bungalows. Each of these have very different architectures. It does not mean that students cannot live in flats, or houses; or that families cannot live in hostels. Its neither convenient, nor efficient. These different architectures cater to groups that have different "processing" needs.

Aircraft hangars and automobile service stations are both meant for maintenance. The "processing" needs are different. Their architecture differs radically; and it is not a matter of scale.

Airports and bus terminals both cater to the transportation of people. The "processing" needs are different. Their architectures are different.

You can operate business establishments and godowns in a palace. Bhagirath Palace is an example. The original architecture was designed for a different purpose. By all accounts it was beautiful. Definitely not beautiful now.

In addition to "processing" needs, there is the matter of values. Design is about trade-offs.Values are used to decide trade-offs. Values decide what is beautiful and what is not. In a residence where should the toilet be? Some three decades ago, I have seen toilets that were separate from the main residence. It reflects a value system. In Chickpet, which is a dense business plus residential area, I have seen a murrah buffalo in the front yard of a multi-storey building. It reflects a value system.

Architects who design buildings must understand both the "processing" needs and the value needs of their customers. Buildings that make the "processing" pleasurable and efficient and meet the values of the customer, have an architecture that is beautiful.

Software architects must understand the "processing" needs of the problem domain. They must also understand the values of good software engineering, the values of stake holders, and the tension between these. To the extent that architecture satisfies the "resulting balance point" of these tensions, it can be said to be beautiful.

An Architectural Story
New Delhi used to have a number of hutments. These were very near the very impressive buildings, on Raisina Hill, that were designed by Lutyens. The hutments were speedily constructed during World War II. They were meant to be temporary and to be torn down after the war. They were shabby, but met the "resulting balance point" at that point in time. Four decades later, most of them were still in use as government offices and family residences. They were hard to live in and hard to work in. Definitely not beautiful.

Some code that I have seen remind me of  "hutment" architecture.

Schooling of Software Architects

In my previous post I briefly described what I understand by software architecture. Software architecture is created by software architects.

When coders writes their first function, they start their schooling as architects. And the schooling continues all through their professional lives as coders. Here's why:

• As more code gets added, complexity increases, code will need to be refactored. If testing is getting complicated code will need to be refactored. Refactoring is architectural work. 
• Which functions are to be hidden and which ones to be made accessible (declared in a .h file) is an architectural issue. 
• Re-organizing functions so that the module (a .c file) has functions that cohere. That is architectural work. 
• Taking corrective action if the Law of Demeter is violated, is architectural work.

Caring about issues of complexity, coherence, coupling, privacy, interfaces, is the behavioral trait of an architect. Coders need to - and can - develop this trait from their earliest days. Coders who lack these traits, though they have decades of experience, are not architects. Putting "Architect" on a business card does not make an architect.

Software architects are not gurus preaching in an ashram. They operate in the heat and the dust of the bazaar. The more remote architects are from code, the more atrophied will be their architectural skills. Unlike building architects, software architects have to start as "brick layers" and "mistrys" (masons) and they must always continue to be competent "brick layers".

Creating architecture calls for judgement. Judgement develops with experience. Learning from one's own experience is a must. But learning from the experience of others - team members and the wider professional community - furthers it. Only a learning organization can be a school for architects. And real architects want to never graduate from school.

Kipling & Software Architecture

My previous post attempted to show that Kipling's wise men had good advice for project managers.

Of the six wise men, five have good advice for the design of software architecture.

1. What. The functionality expected of the application.
2. Why. The problem that delivery of the functionality will solve; the advantage/improvement that it will deliver.
3. How. The broad, coarse-grained steps that need to be executed to deliver the functionality. This is the high-level algorithm; the high-level flow-charts (both control and data).
4. Where. 
1. The blocks of  the flow-chart is where the action takes place.
2. Step-wise refinement of the flow-chart leads to more precisely defined blocks with progressively narrow responsibility. This is recursive application of steps #1 thru #3.
3. The connection between the blocks are directed edges. It denotes that the block at the tail of the edge accesses the block at its head. The interface for this access must be defined (the .h files). The data that is transferred in either direction must be defined. 
4. This is the step that leads to modules/sub-modules (the .c files).
5. Refinement should be guided by the principles of:
1. Minimization of inter-module coupling. Changes in a module that do not affect the interface, should not effect other modules. 
2. Maximization of intra-module cohesiveness. 
1. Data required by functions in the module should either be passed in as parameters, or be present as data items in the module, or obtained by calls to collaborating modules.
2. Functions in the module should all be related to the purpose of  the module.
5. When. The sequence-diagrams.

Comment

Deciding the "Where" is what architecture is about. Deciding that, has to be based on broad principles. Like all design it involves finding a compromise subject to conflicting forces. That requires a clear understanding of values.

Kipling's Advice for Project Managers

I keep six honest serving men
(They taught me all I knew);
Their names are What, and Why and When;
and How and Where and Who.

– Rudyard Kipling (1865-1936).

Any plan must

1. Set out the objective - the "What". 
2. Explain the reason - the "Why".  
3. Specify the steps that are to be executed - the "How".
4. Specify time sequence for execution - the "When".
5. Specify the place, or the objects, where, or on which, the steps are to be completed - the "Where".
6. Specify the persons responsible for the execution of the steps- the "Who".

Come to think of it, source code documentation too, with the possible exception of #6, should have the same information.

Specifications & Unit Tests

In my preceding post I said that writing requirements & specifications must precede the writing of code and they must be written in the source file. Stands to reason. If code has to be written, it must be for a purpose. The purpose must be known before writing the code that is expected to meet the purpose. Therefore, it can - and should - be recorded before coding.

But specifications, instead of being entered as documentation in source files, can be put down in far more useful form. They can be recorded as Executable Specifications. What are executable specifications? They are Unit Tests. Watch this exposition by Misko Hevery (from ~17:28 till ~ 24:00) for how to write Executable Specifications. BTW, the complete video is worth watching.

Coders Write Specifications

Coders write code. Should they not write down specifications like a darzi? Let me describe an example.

I go to a developer and say, "I want a routine that gives me the roots of a quadratic." (user requirement).

In less than an hour I get an executable "quadratic.exe". The developer asks me to run it. I get prompt a prompt: "Please enter a: ". The developer informs me what is "a" and also what the following two inputs "b" & "c" are.

       I enter 0 for "a" and 1 for "b" and "c". I get a divide by zero error. I am expecting the result to be       -1 (x + 1 = 0)
       I enter 1 for a, 0 for b, and 1 for c. I expect to get imaginary 1 (j1). I get a NAN.
       I enter 1 for a, 0 for b, and -2 for c. I expect 1.414. I get 1.414213
       ....

I did not get what I wanted. Should I be disappointed? No. What I got was a prototype.

Prototypes are very useful. They help to refine user requirements and specifications. I do that now. Also, I add that I require to have  a function that will compute the roots and return it in a structure. I then define the structure for the developer.

The developer should now put the requirement and specifications in the function header. And it should be done before making any changes to the source code. As a matter of fact there should have been a header for the prototype code.

Function headers, and file headers are not comments. Comments are post action. Comments are made by Commentators.

Function headers, and file headers are requirements and specifications. They precede the action. Acts need to have a purpose; a desired result. They need to be performed when required preconditions are met.

Cricketing example: Dhoni (or Virat) sees a ball coming at him. It meets the preconditions for a six. He knows the Specifications for the action he has to take. He takes the action. The result follows. The commentators watch the result and pass Comments.

Commentators pass Comments. Coders write Specifications.

Unexploited Value of the CMM

Note:

• I use CMM to refer to both the CMM and its successor the CMMI. 
• What  I have to say about the CMM is also applicable to other process related classification tools e.g. SPICE.

As Wikipedia describes it, the CMM started out as a tool to assess vendors for government (US) software contracts. This, I consider, has led to a distorted view of the CMM tool.

The Distortion

• Software companies, view the CMM rating as merely a marketing aid. 
• As a result, the objective becomes getting certified - not process change. Any process change is just for the benefit of certification inspectors. It is window dressing. The actual process remains unchanged. There is a saying in Hindustani which accurately describes this behavior, "An elephant has two sets of teeth; One for display, another for eating". It is what some (?) of our garment exporters used to do; sample lots used one process, shipment lots used a "practical" process. Garment exporters could sell their rejects in the local market. Software exporters do not have that option.

My Thesis.

• The CMM is a tool for top management, just as much as a balance sheet is.
• Balance sheets too are subject to window dressing. However, top management - unless they are drinking their own kool-aid - would be fully aware of the true situation.

What is The CMM?
I will take the terms in reverse order.

• Model (the 2nd M). The statistician Box said, "All models are false; some are useful."
• False, in the sense that no model captures the entirety of the subject. 
• Useful in the sense that relevant characteristics are captured.
• Relevance is determined by the problem/application/domain. Or, to be tautological, usefulness is determined by use.
• Box was a statistician. Statisticians deal in numbers. No metrics; No models.
• Maturity. 
• I take this as the analogue of what it means for a human to exhibit mature behavior. What is mature human behavior? It is the opposite of immature behavior. It is childish behavior (note: childish and childlike are not the same). The main characteristics of mature behavior are:
• Consistency.
• Predictability.
• Calibrated response.
• Awareness of consequences.
• Low variance of process metrics indicates higher maturity. That is what Six-Sigma processes are about.
• Capability. 
• The Model mentioned above models Capability. As mentioned, modelling requires metrics.
• The facets for which the metrics are required:
• Domain knowledge
• Code
• Size
• Quality metrics
• Project Management. 
• On-time delivery
• Responsiveness to change
• Time to fix defects
• Administrative & Financial.
• Personnel
• Infrastructure
• Project resources
• Metrics mean making measurement. That means demonstrated performance - not imagined performance.

Both process capability, and process variability (maturity), are important. 

Relevance to Management.

• Project Management is about making and keeping commitments.
• Commitments require estimating future performance.
• Estimates have necessarily to be based on the past - the demonstrated capability.
• The reliability of estimates is indicated by the variance of previous differences between estimates and actual performance - the demonstrated maturity. 

Action.

• Software organizations must, on a regular basis, check the health of their process. The CMM is an instrument for performing the check.
• A regular CMM check for the process is as necessary as a regular medical check-up for people.
• Performing a regular CMM check should be the top task of the QA team.
• The report should have the attention of the CEO and CTO