Documenting Source Code - How It Was Driven Home to Me

This happened almost two decades ago. It was my first startup. We had built a radar Moving Target Detector using four DSP processors, plus another DSP as a controller. I had taken it from proof of concept, to field trial, to ruggedized prototype. Over that time the team size had grown from one (me) to three.

The ruggedized prototype was now to be put through acceptance checks. We gave a final check and then packed it for transporting to the customer's site. When we assembled it, and put it into operation, the ouput would not settle down as it was meant to. That got me into a flap. I was sure that during transportation some damage had been done to the hardware. I had visions of doing a messy hardware debug on the customer's premises. Luckily, I decided to sleep on the problem.

The next day I asked one of my engineers if anything had been done to the code that was in the EEPROMs. He said that there was a long sequence of code which did nothing but write zeroes into the RAM. He had taken that out. The data memory was no longer initialized.

So was my engineer at fault? No, I was. This was code that I had written almost at the start of the project. The purpose of the code was clear to me. But I was no longer "in contact" with the code. And I had failed to document it.

I have since then, been rather fanatical about documenting the intent of any block of code. Documenting the WHAT [is intended to be acheived], and the WHY [it is important that it be acheived]. The HOW is not that important; the code should say that - unless of course the code is convoluted. But then one should not be writing convoluted code in the first place:-)

This incident led me to formulate my extension to Murphy's Law:

If anything can go wrong, it will - and it will happen in the presence of the customer!

Toyota Recall: Hubris was the root cause

Quote from Under the Hood of Toyota's Recall: 'A Tremendous Expansion of Complexity'. Watch the video

Prof. Takahiro Fujimoto says:
(Quote)
I would probably say middle managers, particularly at headquarters, started to deviate from the Toyota Way by being arrogant, being overconfident, and also they started not to listen to the problems that customers raised. Toyota is a problem-finding, problem-solving company. This culture is still there in the factories and in product development centers. But in some parts of the headquarters, someone started to say, "Hey, this is our problem. I am responsible for finding my problems and solving my problems. It's not [for] you [outside Toyota] to find our problems."

Sometimes I'm critical of Toyota. But they get angry. They always say, "We want to find problems. So please, give us any clues on the problems you see." But if I actually say, "This is a problem for you," they say, "This is none of your business. We have to find the problem. Not you." This attitude was growing for some time, I think, in some parts of headquarters. That was very dangerous. It is a good time to correct this kind of attitude and go back to the basics of the Toyota system.
(Unquote)

Phenomena, Hypothesis, & Defect

A few days ago my washing machine developed a problem. The phenomena was: The inflow of water would not stop when the power was off. Resulting in flooding.

A service engineer came and inspected the machine. He observed the phenomena. But then he went a step further and created a hypothesis: The water did not stop as the input valve was stuck at open because of sludge(I live in Noida). Then he formulated the corrective action: Chemical wash of the machine to clean out sludge. And that was all that went in his inspection report - no mention of the phenomena.

The workshop people came and collected the machine. The next day they returned a bright-as-new washing machine. But the reported phenomena was still observed! Obviously a chemical wash was not the correct corrective action.

Lesson: Always report the observed phenomena that is to be investigated. Hypotheses should be marked for what they are - reasonable guesses. Most importantly - do not specify a corrective measure without first establishing the defect.

Project Roadmap or Project Plan?

Quite frequently I see the two terms used interchangeably. I think there is a difference.

A road map is not a plan. To use a software term: A road map is just a data structure.

To get from city A to city B by road, one consults a map to decide on means of transportation; the route to take; contingency routes; what are the other cities on the way; road conditions; refuelling stops; weather conditions; ... That does not make it a travel plan. It is when we plug in estimates of dates and estimates of travel time and estimates of stop times, and decide on preparatory measures that it becomes a travel plan.

Different travellers can have the same road map for getting from city A to city B. Yet they may have different travel plans.

A Project Roadmap just has a set of paths to desired project outcomes. It is estimates of time, and allocation of resources, that create a Project Plan.

Toyota: Runaway Cars & Genchi Genbutsu

Genchi Genbutsu ("go and see for yourself") is one of the 14 principles of the Toyota Production System (TPS). Toyota applied Genchi Genbutsu to determine american driving conditions before designing cars for the american market


Was any Genchi Genbutsu done when reports started coming in about runaway cars? Or was it that TPS being a production system, the concept of Genchi Genbutsu did not extend to after-sales problems?

Toyota: What happened to the andon cord?

The Toyota Production System (TPS) empowers any worker on the production line to stop production for any suspected problem on the production line, by "pulling the andon cord" .

So why the problems of the runaway Toyotas? There is a perception that Toyota has been tardy in investigating problems - http://nyti.ms/c29NS2. I suspect the andon cord has not been "extended" to customer support.

As its name says, TPS is a production system. It includes upstream processes of Toyota's vendors and the famed JIT. But does it extend to customer support? Will customer incidents result in the andon cord being pulled? News reports, like the one referred to above,  seem to indicate, not.

Quality: Just Talking the Talk

I was instrumental in introducing some rudimentary processes for the business unit that I have been leading  since 2002. After it had been in place for sometime, my CEO visited a prospect in Japan, along with one of our senior engineers. The engineer had adopted the practices and was a role model. The prospect was a large semiconductor house. The presentation by the senior engineer included actual artefacts produced by the process. That gave some authenticity to the presentation. The president of the company was so impressed that he decided to visit us. The visit resulted in the company outsourcing work to us.

Needless to say my CEO became a convert to the process that I had been advocating. He used to tell everyone that the reason why we got a new customer was because of the process that was demonstrated. That indeed we were walking the talk.

Now that had an interesting side-effect. In subsequent presentation we started talking our process with great pride. But we forgot that good practices have to be re-inforced everyday and deviations have to be corrected. More importantly processes have to be evolved after due deliberation. Not by random variations caused by "practical" problems. We were no longer walking the talk. We had started talking the talk. We forgot: 6-pack abs need hard work to keep them that way

Has Toyota too fallen prey to talking the talk?

More on that in a subsequent post.