Software Product Development–the Unsolved Mystery of High Tech Part I
November 9, 2009 by admin
Filed under Management
Phil Morettini asked:
One of the most puzzling things in high technology, especially for executives on the business side of things, is the software development process. It’s the high tech equivalent to the “Black Hole” phenomenon made famous in Astronomy. Endless resources can be poured into a software development project, yet there never seems to be an end in sight.
And why is this so? It seems that in such a typically high tech, yet now familiar activity, we would have long ago figured it out. You would think that the process of software development would, by now, amount to simply turning a crank—yet it seems it hasn’t advanced much since the dawn of the PC age.
I don’t mean to be overly dramatic here. But I have been in the high tech and software industries since 1983, and I have never been involved with—or even personally known of a software project—that came in on time and under budget. Never. Not even ONCE. That’s pretty incredible. Now, I realize that there are almost certainly examples of on-schedule projects out there, but they are in the overwhelming minority of all software that is developed.
THEY ALWAYS SLIP
It’s just accepted in the software business that projects will slip, particularly when the end result is an actual commercial product. The businesses I’ve been involved in have tried everything. We’ve tried an approach of “No upfront planning”-starting coding as soon as possible. We’ve tried “extensive and laborious upfront planning”-with a detailed spec, and a prototype, completed prior to initiating production coding. I’ve seen many projects that tried using intermediate steps, falling between the two extreme approaches above. We’ve tried to start projects by purchasing as many “pre-written” modules as possible, used various languages and platforms, hired dedicated debugging personnel, tried code-generators, assembled both small teams & large teams, you name it—we’ve tried it. Project schedules have been written with the utmost conservatism, at the insistence of senior management. No matter. Across a number of different companies, EVERY project has slipped out beyond the wildest nightmares or everyone involved.
ONE LINE OF CODE, TWO WEEK DELAY
Once I asked our lead programmer to change ONE LINE OF CODE in a well-established product. He estimated it would take just a few seconds to make the change, and a few hours to test it. The change would be final by the end of the day, at the latest. Two weeks later I was still waiting for a solid product.
Now, don’t misunderstand. I’m not writing this to bash software developers. While not every developer I’ve worked with over the years has been a world-beater, I’ve had the fortune to work with quite a number whom I consider to be outstanding. But no matter how much thought, time and effort went into it, our projects always slipped. A lot. We usually ended up with a commercially successful product, but how much better we could have done, had we figured out a way to bring the product to market on time? The only saving grace was the competition had the same problem.
MORE ART THAN SCIENCE
The reason, I believe, is that writing software remains much more of an art than a science. This statement is a bit surprising, until you look a little deeper. There is certainly much methodology available to guide a team to use sound, time-tested practices in developing software. However, a software program is really just a document written in a foreign language. That’s why C++ and Java are called Programming Languages. In writing a novel you are essentially creating a unique work that has never been done quite the same way before. Also true for a software program. If you knew exactly how the writing of a novel or software program would go before you began, there would be no need to write it—it would have already been done. While there are plenty of rules (representing the science) to writing good software, at the end of the day it’s a unique, written creation (the art).
COMPLEXITY OVERWHELMS EXPERIENCE
Another key reason why conquering the software development process has appeared to be impossible, is the vastly increased complexity associated with software projects today. Let’s face it, the average piece of software today does a lot more, and is quite a larger in terms of the number of lines of code, than at the dawn of the PC era. The creation of graphical user interfaces really started the explosion in the size of software code. So much more code is needed, to bring the user-friendly products of today to life. And what enabled this, of course, was the dawn of the modern operating systems, especially the overcoming of the 640K limit that the original DOS operating system required PC programs to run in. Windows and other modern operating systems almost eliminated the need to write software efficiently, at least from a code size perspective. It’s interesting to speculate—if we were still writing in the 640K box, would software development have evolved to a more predictable science today? Maybe, but the world would be a less productive as a result.
WHAT TO DO FROM A BUSINESS PERSPECTIVE?
As you can tell from this discussion, I don’t have a great set of answers on how to bring software to market on time. It’s one of the great frustrations of my career. I still strongly believe that getting the best people you can get will make the problem better, even if it can’t be solved completely. I also believe in keeping development teams small, with the minimum of structure necessary to run the project. It’s also wise, in my opinion, to structure your product releases to be more frequent, while adding fewer new features per release. This should at least minimize the pain of each release slipping, since the slip time of each release should be less. And knowing what you’re going to be coding, developing a spec document and sticking to it (no feature creep!) is also sound practice, although I’ve found it to be no panacea. Beyond that, I’m at a loss. Maybe one of you has a strong opinion on how to bring projects out on time? If so, post a comment—this is a discussion worth having.
Monitoring the progress of a software project can be like peering into the darkness of a seemingly bottomless pit.
And why is this so? It seems that in such a typically high tech, yet now familiar activity, we would have long ago figured it out. You would think that the process of software development would, by now, amount to simply turning a crank—yet it seems it hasn’t advanced much since the dawn of the PC age.
I don’t mean to be overly dramatic here. But I have been in the high tech and software industries since 1983, and I have never been involved with—or even personally known of a software project—that came in on time and under budget. Never. Not even ONCE. That’s pretty incredible. Now, I realize that there are almost certainly examples of on-schedule projects out there, but they are in the overwhelming minority of all software that is developed.
One of the most puzzling things in high technology, especially for executives on the business side of things, is the software development process. It’s the high tech equivalent to the “Black Hole” phenomenon made famous in Astronomy. Endless resources can be poured into a software development project, yet there never seems to be an end in sight.
And why is this so? It seems that in such a typically high tech, yet now familiar activity, we would have long ago figured it out. You would think that the process of software development would, by now, amount to simply turning a crank—yet it seems it hasn’t advanced much since the dawn of the PC age.
I don’t mean to be overly dramatic here. But I have been in the high tech and software industries since 1983, and I have never been involved with—or even personally known of a software project—that came in on time and under budget. Never. Not even ONCE. That’s pretty incredible. Now, I realize that there are almost certainly examples of on-schedule projects out there, but they are in the overwhelming minority of all software that is developed.
THEY ALWAYS SLIP
It’s just accepted in the software business that projects will slip, particularly when the end result is an actual commercial product. The businesses I’ve been involved in have tried everything. We’ve tried an approach of “No upfront planning”-starting coding as soon as possible. We’ve tried “extensive and laborious upfront planning”-with a detailed spec, and a prototype, completed prior to initiating production coding. I’ve seen many projects that tried using intermediate steps, falling between the two extreme approaches above. We’ve tried to start projects by purchasing as many “pre-written” modules as possible, used various languages and platforms, hired dedicated debugging personnel, tried code-generators, assembled both small teams & large teams, you name it—we’ve tried it. Project schedules have been written with the utmost conservatism, at the insistence of senior management. No matter. Across a number of different companies, EVERY project has slipped out beyond the wildest nightmares or everyone involved.
ONE LINE OF CODE, TWO WEEK DELAY
Once I asked our lead programmer to change ONE LINE OF CODE in a well-established product. He estimated it would take just a few seconds to make the change, and a few hours to test it. The change would be final by the end of the day, at the latest. Two weeks later I was still waiting for a solid product.
Now, don’t misunderstand. I’m not writing this to bash software developers. While not every developer I’ve worked with over the years has been a world-beater, I’ve had the fortune to work with quite a number whom I consider to be outstanding. But no matter how much thought, time and effort went into it, our projects always slipped. A lot. We usually ended up with a commercially successful product, but how much better we could have done, had we figured out a way to bring the product to market on time? The only saving grace was the competition had the same problem.
MORE ART THAN SCIENCE
The reason, I believe, is that writing software remains much more of an art than a science. This statement is a bit surprising, until you look a little deeper. There is certainly much methodology available to guide a team to use sound, time-tested practices in developing software. However, a software program is really just a document written in a foreign language. That’s why C++ and Java are called Programming Languages. In writing a novel you are essentially creating a unique work that has never been done quite the same way before. Also true for a software program. If you knew exactly how the writing of a novel or software program would go before you began, there would be no need to write it—it would have already been done. While there are plenty of rules (representing the science) to writing good software, at the end of the day it’s a unique, written creation (the art).
COMPLEXITY OVERWHELMS EXPERIENCE
Another key reason why conquering the software development process has appeared to be impossible, is the vastly increased complexity associated with software projects today. Let’s face it, the average piece of software today does a lot more, and is quite a larger in terms of the number of lines of code, than at the dawn of the PC era. The creation of graphical user interfaces really started the explosion in the size of software code. So much more code is needed, to bring the user-friendly products of today to life. And what enabled this, of course, was the dawn of the modern operating systems, especially the overcoming of the 640K limit that the original DOS operating system required PC programs to run in. Windows and other modern operating systems almost eliminated the need to write software efficiently, at least from a code size perspective. It’s interesting to speculate—if we were still writing in the 640K box, would software development have evolved to a more predictable science today? Maybe, but the world would be a less productive as a result.
WHAT TO DO FROM A BUSINESS PERSPECTIVE?
As you can tell from this discussion, I don’t have a great set of answers on how to bring software to market on time. It’s one of the great frustrations of my career. I still strongly believe that getting the best people you can get will make the problem better, even if it can’t be solved completely. I also believe in keeping development teams small, with the minimum of structure necessary to run the project. It’s also wise, in my opinion, to structure your product releases to be more frequent, while adding fewer new features per release. This should at least minimize the pain of each release slipping, since the slip time of each release should be less. And knowing what you’re going to be coding, developing a spec document and sticking to it (no feature creep!) is also sound practice, although I’ve found it to be no panacea. Beyond that, I’m at a loss. Maybe one of you has a strong opinion on how to bring projects out on time? If so, post a comment—this is a discussion worth having.
Monitoring the progress of a software project can be like peering into the darkness of a seemingly bottomless pit.
And why is this so? It seems that in such a typically high tech, yet now familiar activity, we would have long ago figured it out. You would think that the process of software development would, by now, amount to simply turning a crank—yet it seems it hasn’t advanced much since the dawn of the PC age.
I don’t mean to be overly dramatic here. But I have been in the high tech and software industries since 1983, and I have never been involved with—or even personally known of a software project—that came in on time and under budget. Never. Not even ONCE. That’s pretty incredible. Now, I realize that there are almost certainly examples of on-schedule projects out there, but they are in the overwhelming minority of all software that is developed.
The Appliance of Science
keegan asked:
And now for the perhaps shocking news: there are times when Viagra, Cialis and Levitra are of no help at all for treating erectile dysfunction. You have all been persuaded to believe that Viagra and the other PDE-5 inhibitors will always be able to help, even if only a little. Unfortunately, that’s not true. There are times when the degree of damage is too great for a “natural” solution. And let’s be clear. In relative terms, the use of Viagra and the others allows the body to respond to sexual stimulation as it was designed to do. Viagra simply removes a physical barrier to a natural ********.
This means we are into the surgical options. The first to mention is vascular reconstructive surgery. In a small number of cases, the surgeon may be able to reconstruct the arteries so that blood will flow more freely into the corpus cavernosum. Alternatively, surgery may help those suffering from venous leakages. When the blood inflates the corpus cavernosum, it compresses the veins that allow the blood to drain out of the *****. If the compression is inadequate, the blood will not be retained inside the corpus and the ******** will deflate. Unfortunately, the surgical theories have mostly failed in practice. Only about 5% of those having reconstructive surgery have found improved performance. This surgery is expensive. It is a long and difficult operation, and any incision can cause nerve damage. So it is not often used today.
Thus, when all else has failed or is not acceptable for some reason, we come to the somewhat radical solution of implanting a penile prosthesis.
How does this work?
There are three tubes in the penile shaft. Running along the bottom of the penis, the urethra drains the urine from the bladder out of the body through the *****. Along the top of the penis, the two more substantial tubes run side-by-side. They are designed to work in much the same way as the tyre on a car, i.e. with an inner and outer tube. So when the inner tubes fill with blood, they push against the outer tubes and, as the volume of blood increases, lift the ***** into an erect position.
Since the 1930s, medical researchers have been experimenting with implants to mimic this process of inflation and create the effect of an ********. We therefore have more than seventy years of experience among more than one million men to draw on. Two different approaches have come to dominate. One range of implants remains semi-rigid. These devices are easy to implant and very rarely fail mechanically. The problem, however, is that they produce a permanent ******** which can be difficult to conceal. Although they may be flexed up and down, most have limited mobility. The Dura II manufactured by American Medical Systems (AMS) has a more sophisticated system relying on interlocking disks strung on a central cable under tension which offers greater control over positioning, but may have an increased risk of mechanical failure.
The others are hydraulic, e.g. Dynaflex and CXM, and inflate, relying on a pump contained inside the ******* to transfer fluid (almost always water) from a reservoir inside the abdomen. These devices may have:
1.one, rod-like inflatable chamber which is hinged;
2.a two chamber system, e.g. the Ambicor; or
3.a three chamber system. All these systems replace the original corpa cavernosa. Today’s three-piece inflatable devices produced by AMS and Mentor Corporation offer flaccidity and ********* that approach the natural state. Whichever system you opt for, you should expect some pain and discomfort for five or six weeks following surgery. And, once installed, the implants cannot replace the “real thing”. Thus, if you have already lost your libido, it will not return. Similarly, if the nerve damage had caused a loss of sensation, or you were unable to achieve ****** or ejaculation, prosthetics will not restore those functions.So what are the advantages of this technology? Well, when the oral medications such as Viagra, Cialis and Levitra have failed, the convenience factors of the alternatives become important. Implants are safe and work no matter what the original cause of the ED. They are particularly useful to treat cases of damage following priapism and Peyronie’s Disease, fibrosis after injections, and to restore function after surgery for prostate cancer. So long as the partners are counselled on the strengths and weaknesses of the systems available, they provide satisfaction in more than 90% of all cases. Once installed, there are no further costs and they require no further maintenance (only about 3% of all installed devices ever need attention). From a psychological point of view, there is no external equipment to distract — as in the case of injections and vacuum devices. Equally, there is no need to remember to bring the Viagra, Cialis or Levitra along. If you are in the mood, you can simply act without having to wait for the medication to take effect.
Nevertheless, there are disadvantages to consider. This is a once-and-for-all procedure. Once the implant has been installed, it is no longer possible for a natural ******** to occur. It is always possible that, as science develops, different non-invasive procedures may become available. One may solve the ED problem you have. If your view is that a more natural form of treatment is better, then you may decide to wait. On the issue of informed consent, there are always risks in surgery involving an anaesthetic. In particular and despite a coating of antibiotics to modern implants, you will learn there is a small risk of infection following the surgery. This is relatively rare, but if it does occur and irrigation fails, the only response is to remove the prosthesis. However, once installed, there are three more general complaints.
1.Because the ******** is not being caused “naturally”, the systems only produce hardness in the body of the *****. Neither system affects the head of the ***** which remains soft. This can feel unsatisfactory and make *********** uncomfortable.
2.The body of the ***** does not warm as in the natural ******** which can feel strange to your partner.
3.The surgery can sometimes produce a shorter ***** which you and your partner may find unsatisfactory.
Finally, let’s dispose of one more general worry that some of you may have. There have been a number of problems with the safety of some silicone implants, e.g. in ****** enhancements. None of the inflatable prostheses use any form of silicone as the fluid.
And now for the perhaps shocking news: there are times when Viagra, Cialis and Levitra are of no help at all for treating erectile dysfunction. You have all been persuaded to believe that Viagra and the other PDE-5 inhibitors will always be able to help, even if only a little. Unfortunately, that’s not true. There are times when the degree of damage is too great for a “natural” solution. And let’s be clear. In relative terms, the use of Viagra and the others allows the body to respond to sexual stimulation as it was designed to do. Viagra simply removes a physical barrier to a natural ********.
This means we are into the surgical options. The first to mention is vascular reconstructive surgery. In a small number of cases, the surgeon may be able to reconstruct the arteries so that blood will flow more freely into the corpus cavernosum. Alternatively, surgery may help those suffering from venous leakages. When the blood inflates the corpus cavernosum, it compresses the veins that allow the blood to drain out of the *****. If the compression is inadequate, the blood will not be retained inside the corpus and the ******** will deflate. Unfortunately, the surgical theories have mostly failed in practice. Only about 5% of those having reconstructive surgery have found improved performance. This surgery is expensive. It is a long and difficult operation, and any incision can cause nerve damage. So it is not often used today.
Thus, when all else has failed or is not acceptable for some reason, we come to the somewhat radical solution of implanting a penile prosthesis.
How does this work?
There are three tubes in the penile shaft. Running along the bottom of the penis, the urethra drains the urine from the bladder out of the body through the *****. Along the top of the penis, the two more substantial tubes run side-by-side. They are designed to work in much the same way as the tyre on a car, i.e. with an inner and outer tube. So when the inner tubes fill with blood, they push against the outer tubes and, as the volume of blood increases, lift the ***** into an erect position.
Since the 1930s, medical researchers have been experimenting with implants to mimic this process of inflation and create the effect of an ********. We therefore have more than seventy years of experience among more than one million men to draw on. Two different approaches have come to dominate. One range of implants remains semi-rigid. These devices are easy to implant and very rarely fail mechanically. The problem, however, is that they produce a permanent ******** which can be difficult to conceal. Although they may be flexed up and down, most have limited mobility. The Dura II manufactured by American Medical Systems (AMS) has a more sophisticated system relying on interlocking disks strung on a central cable under tension which offers greater control over positioning, but may have an increased risk of mechanical failure.
The others are hydraulic, e.g. Dynaflex and CXM, and inflate, relying on a pump contained inside the ******* to transfer fluid (almost always water) from a reservoir inside the abdomen. These devices may have:
1.one, rod-like inflatable chamber which is hinged;
2.a two chamber system, e.g. the Ambicor; or
3.a three chamber system. All these systems replace the original corpa cavernosa. Today’s three-piece inflatable devices produced by AMS and Mentor Corporation offer flaccidity and ********* that approach the natural state. Whichever system you opt for, you should expect some pain and discomfort for five or six weeks following surgery. And, once installed, the implants cannot replace the “real thing”. Thus, if you have already lost your libido, it will not return. Similarly, if the nerve damage had caused a loss of sensation, or you were unable to achieve ****** or ejaculation, prosthetics will not restore those functions.So what are the advantages of this technology? Well, when the oral medications such as Viagra, Cialis and Levitra have failed, the convenience factors of the alternatives become important. Implants are safe and work no matter what the original cause of the ED. They are particularly useful to treat cases of damage following priapism and Peyronie’s Disease, fibrosis after injections, and to restore function after surgery for prostate cancer. So long as the partners are counselled on the strengths and weaknesses of the systems available, they provide satisfaction in more than 90% of all cases. Once installed, there are no further costs and they require no further maintenance (only about 3% of all installed devices ever need attention). From a psychological point of view, there is no external equipment to distract — as in the case of injections and vacuum devices. Equally, there is no need to remember to bring the Viagra, Cialis or Levitra along. If you are in the mood, you can simply act without having to wait for the medication to take effect.
Nevertheless, there are disadvantages to consider. This is a once-and-for-all procedure. Once the implant has been installed, it is no longer possible for a natural ******** to occur. It is always possible that, as science develops, different non-invasive procedures may become available. One may solve the ED problem you have. If your view is that a more natural form of treatment is better, then you may decide to wait. On the issue of informed consent, there are always risks in surgery involving an anaesthetic. In particular and despite a coating of antibiotics to modern implants, you will learn there is a small risk of infection following the surgery. This is relatively rare, but if it does occur and irrigation fails, the only response is to remove the prosthesis. However, once installed, there are three more general complaints.
1.Because the ******** is not being caused “naturally”, the systems only produce hardness in the body of the *****. Neither system affects the head of the ***** which remains soft. This can feel unsatisfactory and make *********** uncomfortable.
2.The body of the ***** does not warm as in the natural ******** which can feel strange to your partner.
3.The surgery can sometimes produce a shorter ***** which you and your partner may find unsatisfactory.
Finally, let’s dispose of one more general worry that some of you may have. There have been a number of problems with the safety of some silicone implants, e.g. in ****** enhancements. None of the inflatable prostheses use any form of silicone as the fluid.




