Software Product Development–the Unsolved Mystery of High Tech Part I
November 9, 2009 by admin
Filed under Management
Phil Morettini asked:
One of the most puzzling things in high technology, especially for executives on the business side of things, is the software development process. It’s the high tech equivalent to the “Black Hole” phenomenon made famous in Astronomy. Endless resources can be poured into a software development project, yet there never seems to be an end in sight.
And why is this so? It seems that in such a typically high tech, yet now familiar activity, we would have long ago figured it out. You would think that the process of software development would, by now, amount to simply turning a crank—yet it seems it hasn’t advanced much since the dawn of the PC age.
I don’t mean to be overly dramatic here. But I have been in the high tech and software industries since 1983, and I have never been involved with—or even personally known of a software project—that came in on time and under budget. Never. Not even ONCE. That’s pretty incredible. Now, I realize that there are almost certainly examples of on-schedule projects out there, but they are in the overwhelming minority of all software that is developed.
THEY ALWAYS SLIP
It’s just accepted in the software business that projects will slip, particularly when the end result is an actual commercial product. The businesses I’ve been involved in have tried everything. We’ve tried an approach of “No upfront planning”-starting coding as soon as possible. We’ve tried “extensive and laborious upfront planning”-with a detailed spec, and a prototype, completed prior to initiating production coding. I’ve seen many projects that tried using intermediate steps, falling between the two extreme approaches above. We’ve tried to start projects by purchasing as many “pre-written” modules as possible, used various languages and platforms, hired dedicated debugging personnel, tried code-generators, assembled both small teams & large teams, you name it—we’ve tried it. Project schedules have been written with the utmost conservatism, at the insistence of senior management. No matter. Across a number of different companies, EVERY project has slipped out beyond the wildest nightmares or everyone involved.
ONE LINE OF CODE, TWO WEEK DELAY
Once I asked our lead programmer to change ONE LINE OF CODE in a well-established product. He estimated it would take just a few seconds to make the change, and a few hours to test it. The change would be final by the end of the day, at the latest. Two weeks later I was still waiting for a solid product.
Now, don’t misunderstand. I’m not writing this to bash software developers. While not every developer I’ve worked with over the years has been a world-beater, I’ve had the fortune to work with quite a number whom I consider to be outstanding. But no matter how much thought, time and effort went into it, our projects always slipped. A lot. We usually ended up with a commercially successful product, but how much better we could have done, had we figured out a way to bring the product to market on time? The only saving grace was the competition had the same problem.
MORE ART THAN SCIENCE
The reason, I believe, is that writing software remains much more of an art than a science. This statement is a bit surprising, until you look a little deeper. There is certainly much methodology available to guide a team to use sound, time-tested practices in developing software. However, a software program is really just a document written in a foreign language. That’s why C++ and Java are called Programming Languages. In writing a novel you are essentially creating a unique work that has never been done quite the same way before. Also true for a software program. If you knew exactly how the writing of a novel or software program would go before you began, there would be no need to write it—it would have already been done. While there are plenty of rules (representing the science) to writing good software, at the end of the day it’s a unique, written creation (the art).
COMPLEXITY OVERWHELMS EXPERIENCE
Another key reason why conquering the software development process has appeared to be impossible, is the vastly increased complexity associated with software projects today. Let’s face it, the average piece of software today does a lot more, and is quite a larger in terms of the number of lines of code, than at the dawn of the PC era. The creation of graphical user interfaces really started the explosion in the size of software code. So much more code is needed, to bring the user-friendly products of today to life. And what enabled this, of course, was the dawn of the modern operating systems, especially the overcoming of the 640K limit that the original DOS operating system required PC programs to run in. Windows and other modern operating systems almost eliminated the need to write software efficiently, at least from a code size perspective. It’s interesting to speculate—if we were still writing in the 640K box, would software development have evolved to a more predictable science today? Maybe, but the world would be a less productive as a result.
WHAT TO DO FROM A BUSINESS PERSPECTIVE?
As you can tell from this discussion, I don’t have a great set of answers on how to bring software to market on time. It’s one of the great frustrations of my career. I still strongly believe that getting the best people you can get will make the problem better, even if it can’t be solved completely. I also believe in keeping development teams small, with the minimum of structure necessary to run the project. It’s also wise, in my opinion, to structure your product releases to be more frequent, while adding fewer new features per release. This should at least minimize the pain of each release slipping, since the slip time of each release should be less. And knowing what you’re going to be coding, developing a spec document and sticking to it (no feature creep!) is also sound practice, although I’ve found it to be no panacea. Beyond that, I’m at a loss. Maybe one of you has a strong opinion on how to bring projects out on time? If so, post a comment—this is a discussion worth having.
Monitoring the progress of a software project can be like peering into the darkness of a seemingly bottomless pit.
And why is this so? It seems that in such a typically high tech, yet now familiar activity, we would have long ago figured it out. You would think that the process of software development would, by now, amount to simply turning a crank—yet it seems it hasn’t advanced much since the dawn of the PC age.
I don’t mean to be overly dramatic here. But I have been in the high tech and software industries since 1983, and I have never been involved with—or even personally known of a software project—that came in on time and under budget. Never. Not even ONCE. That’s pretty incredible. Now, I realize that there are almost certainly examples of on-schedule projects out there, but they are in the overwhelming minority of all software that is developed.
One of the most puzzling things in high technology, especially for executives on the business side of things, is the software development process. It’s the high tech equivalent to the “Black Hole” phenomenon made famous in Astronomy. Endless resources can be poured into a software development project, yet there never seems to be an end in sight.
And why is this so? It seems that in such a typically high tech, yet now familiar activity, we would have long ago figured it out. You would think that the process of software development would, by now, amount to simply turning a crank—yet it seems it hasn’t advanced much since the dawn of the PC age.
I don’t mean to be overly dramatic here. But I have been in the high tech and software industries since 1983, and I have never been involved with—or even personally known of a software project—that came in on time and under budget. Never. Not even ONCE. That’s pretty incredible. Now, I realize that there are almost certainly examples of on-schedule projects out there, but they are in the overwhelming minority of all software that is developed.
THEY ALWAYS SLIP
It’s just accepted in the software business that projects will slip, particularly when the end result is an actual commercial product. The businesses I’ve been involved in have tried everything. We’ve tried an approach of “No upfront planning”-starting coding as soon as possible. We’ve tried “extensive and laborious upfront planning”-with a detailed spec, and a prototype, completed prior to initiating production coding. I’ve seen many projects that tried using intermediate steps, falling between the two extreme approaches above. We’ve tried to start projects by purchasing as many “pre-written” modules as possible, used various languages and platforms, hired dedicated debugging personnel, tried code-generators, assembled both small teams & large teams, you name it—we’ve tried it. Project schedules have been written with the utmost conservatism, at the insistence of senior management. No matter. Across a number of different companies, EVERY project has slipped out beyond the wildest nightmares or everyone involved.
ONE LINE OF CODE, TWO WEEK DELAY
Once I asked our lead programmer to change ONE LINE OF CODE in a well-established product. He estimated it would take just a few seconds to make the change, and a few hours to test it. The change would be final by the end of the day, at the latest. Two weeks later I was still waiting for a solid product.
Now, don’t misunderstand. I’m not writing this to bash software developers. While not every developer I’ve worked with over the years has been a world-beater, I’ve had the fortune to work with quite a number whom I consider to be outstanding. But no matter how much thought, time and effort went into it, our projects always slipped. A lot. We usually ended up with a commercially successful product, but how much better we could have done, had we figured out a way to bring the product to market on time? The only saving grace was the competition had the same problem.
MORE ART THAN SCIENCE
The reason, I believe, is that writing software remains much more of an art than a science. This statement is a bit surprising, until you look a little deeper. There is certainly much methodology available to guide a team to use sound, time-tested practices in developing software. However, a software program is really just a document written in a foreign language. That’s why C++ and Java are called Programming Languages. In writing a novel you are essentially creating a unique work that has never been done quite the same way before. Also true for a software program. If you knew exactly how the writing of a novel or software program would go before you began, there would be no need to write it—it would have already been done. While there are plenty of rules (representing the science) to writing good software, at the end of the day it’s a unique, written creation (the art).
COMPLEXITY OVERWHELMS EXPERIENCE
Another key reason why conquering the software development process has appeared to be impossible, is the vastly increased complexity associated with software projects today. Let’s face it, the average piece of software today does a lot more, and is quite a larger in terms of the number of lines of code, than at the dawn of the PC era. The creation of graphical user interfaces really started the explosion in the size of software code. So much more code is needed, to bring the user-friendly products of today to life. And what enabled this, of course, was the dawn of the modern operating systems, especially the overcoming of the 640K limit that the original DOS operating system required PC programs to run in. Windows and other modern operating systems almost eliminated the need to write software efficiently, at least from a code size perspective. It’s interesting to speculate—if we were still writing in the 640K box, would software development have evolved to a more predictable science today? Maybe, but the world would be a less productive as a result.
WHAT TO DO FROM A BUSINESS PERSPECTIVE?
As you can tell from this discussion, I don’t have a great set of answers on how to bring software to market on time. It’s one of the great frustrations of my career. I still strongly believe that getting the best people you can get will make the problem better, even if it can’t be solved completely. I also believe in keeping development teams small, with the minimum of structure necessary to run the project. It’s also wise, in my opinion, to structure your product releases to be more frequent, while adding fewer new features per release. This should at least minimize the pain of each release slipping, since the slip time of each release should be less. And knowing what you’re going to be coding, developing a spec document and sticking to it (no feature creep!) is also sound practice, although I’ve found it to be no panacea. Beyond that, I’m at a loss. Maybe one of you has a strong opinion on how to bring projects out on time? If so, post a comment—this is a discussion worth having.
Monitoring the progress of a software project can be like peering into the darkness of a seemingly bottomless pit.
And why is this so? It seems that in such a typically high tech, yet now familiar activity, we would have long ago figured it out. You would think that the process of software development would, by now, amount to simply turning a crank—yet it seems it hasn’t advanced much since the dawn of the PC age.
I don’t mean to be overly dramatic here. But I have been in the high tech and software industries since 1983, and I have never been involved with—or even personally known of a software project—that came in on time and under budget. Never. Not even ONCE. That’s pretty incredible. Now, I realize that there are almost certainly examples of on-schedule projects out there, but they are in the overwhelming minority of all software that is developed.
Life Science Companies and the Fda: How Change Control Software Transforms a Combatant Fist Fight Into a Picnic in the Park
Marci Lynn Crane asked:
When you consider the relationship a life science company shares with the FDA it’s easy to visualize the love/hate kinship of a parent and child. It’s also simple to deduce who the parent might be. After all, the FDA Mama or Papa Bear has never been shy when it comes to playing by regulatory rules.
The Goals of Life Science and the Goals of the FDA: Mutually Compatible or Mutually Exclusive?
The goals of a life science company are likely to include the development, manufacture and sales of the best and most useful biotech equipment, medical devices, drugs, etc. The goals of the FDA include safety, the transparent communication of possible dangers or uncomfortable side effects, and the viable effectiveness of products produced. Each set of goals is noble and in theory supports the goals of the other. In practice however, the goals of these separate institutions seem almost mutually exclusive.
Change Control Software: Middle Ground for Two Separate Powers
Like most solutions to apparently irreparable problems, the solution that allows both life science companies and the FDA to find a cheerful middle ground is the solution that benefits both organizations. This solution includes the implementation and use of change control software. Change control software is valuable for the simple reason that it allows life science companies to artfully manage (without spending ridiculous man-hours) those changes that occur at any stage of a product’s development or manufacturing processes. Documentation, routing, tasks and employee accountability are also concerted via a solid change control software solution.
The benefits of a change control software solution are many but to truly understand these benefits it’s beneficial to first understand the problems that often maintain many life science professionals in a state of “regulated” frustration.
The Problems
Among the problems that lie within the walls of many life science companies is the problem of change management processes that are controlled 1) manually or 2) with a hybrid/decentralized solution. Most companies simply want to save their hard earned ROI, but when considered carefully it’s easy to see that manual and/or hybrid solutions won’t save companies money in the long run, and in addition, present a variety of problems that might have managed to stump even Solomon the wise.
Manual and Hybrid Solutions: Problems that Contribute to Greater Problems
Listed below are some of the problems that life science companies can avoid with a digitized and centralized change control software solution.
High Costs—Life science professionals may think they save money with a paper-based system but in all reality they spend a great deal to pay for the tedious man-hours spent on the manual routing of documents, the document approval process and on manual search and document retrieval (if the document can actually be found). A change control software solution however manages to automatically route documents, seek the appropriate approvals/electronic signatures, search for documents and retrieve them.
Validation Procedures—The FDA requires that processes (this includes change processes) be validated. If a life science company has to validate their processes manually it can be a far greater challenge than some companies care to undertake. Electronic validation presents a far more effective alternative that saves time, money, and effort. In the long run, manual validation can also cost more than a centralized solution. When searching for change control software, ask the software sellers if automated validation services are available.
No Centralized Location—A paper-based system doesn’t allow life science professionals to maintain, protect and organize documents in one digitized and centralized location. A commendable change control software solution digitizes and centralizes information so that life science associates can find the organized information they need—when they need it. A good change control software solution will also manage almost any type of file required throughout any of the GxP processes.
Document Revision Processes—When change inevitably occurs the manual document revision process can be faulty in the extreme. Life science employees will unfortunately be subjected to manual searches for documents, and will be required to manually make changes on documents or SOPs. A good change control software solution however can provide revision control that is automatic and centralized so that once submitted, documents can be routed and approved quickly and effectually. Change tracking will also be automatic.
Also, when searching for the change control solution that will fit the needs of a life science company, professionals should look for a solution that is 1) customizable to their needs and that 2) require form explanations for changes that are being made. Accountability, after all should be an important aspect of change control.
A Picnic in the Park
Change control can take life science professionals one step further to a change management process that could potentially be compared to a picnic in the park.
When you consider the relationship a life science company shares with the FDA it’s easy to visualize the love/hate kinship of a parent and child. It’s also simple to deduce who the parent might be. After all, the FDA Mama or Papa Bear has never been shy when it comes to playing by regulatory rules.
The Goals of Life Science and the Goals of the FDA: Mutually Compatible or Mutually Exclusive?
The goals of a life science company are likely to include the development, manufacture and sales of the best and most useful biotech equipment, medical devices, drugs, etc. The goals of the FDA include safety, the transparent communication of possible dangers or uncomfortable side effects, and the viable effectiveness of products produced. Each set of goals is noble and in theory supports the goals of the other. In practice however, the goals of these separate institutions seem almost mutually exclusive.
Change Control Software: Middle Ground for Two Separate Powers
Like most solutions to apparently irreparable problems, the solution that allows both life science companies and the FDA to find a cheerful middle ground is the solution that benefits both organizations. This solution includes the implementation and use of change control software. Change control software is valuable for the simple reason that it allows life science companies to artfully manage (without spending ridiculous man-hours) those changes that occur at any stage of a product’s development or manufacturing processes. Documentation, routing, tasks and employee accountability are also concerted via a solid change control software solution.
The benefits of a change control software solution are many but to truly understand these benefits it’s beneficial to first understand the problems that often maintain many life science professionals in a state of “regulated” frustration.
The Problems
Among the problems that lie within the walls of many life science companies is the problem of change management processes that are controlled 1) manually or 2) with a hybrid/decentralized solution. Most companies simply want to save their hard earned ROI, but when considered carefully it’s easy to see that manual and/or hybrid solutions won’t save companies money in the long run, and in addition, present a variety of problems that might have managed to stump even Solomon the wise.
Manual and Hybrid Solutions: Problems that Contribute to Greater Problems
Listed below are some of the problems that life science companies can avoid with a digitized and centralized change control software solution.
High Costs—Life science professionals may think they save money with a paper-based system but in all reality they spend a great deal to pay for the tedious man-hours spent on the manual routing of documents, the document approval process and on manual search and document retrieval (if the document can actually be found). A change control software solution however manages to automatically route documents, seek the appropriate approvals/electronic signatures, search for documents and retrieve them.
Validation Procedures—The FDA requires that processes (this includes change processes) be validated. If a life science company has to validate their processes manually it can be a far greater challenge than some companies care to undertake. Electronic validation presents a far more effective alternative that saves time, money, and effort. In the long run, manual validation can also cost more than a centralized solution. When searching for change control software, ask the software sellers if automated validation services are available.
No Centralized Location—A paper-based system doesn’t allow life science professionals to maintain, protect and organize documents in one digitized and centralized location. A commendable change control software solution digitizes and centralizes information so that life science associates can find the organized information they need—when they need it. A good change control software solution will also manage almost any type of file required throughout any of the GxP processes.
Document Revision Processes—When change inevitably occurs the manual document revision process can be faulty in the extreme. Life science employees will unfortunately be subjected to manual searches for documents, and will be required to manually make changes on documents or SOPs. A good change control software solution however can provide revision control that is automatic and centralized so that once submitted, documents can be routed and approved quickly and effectually. Change tracking will also be automatic.
Also, when searching for the change control solution that will fit the needs of a life science company, professionals should look for a solution that is 1) customizable to their needs and that 2) require form explanations for changes that are being made. Accountability, after all should be an important aspect of change control.
A Picnic in the Park
Change control can take life science professionals one step further to a change management process that could potentially be compared to a picnic in the park.
