|Software development process|
|Paradigms and models|
|Methodologies and frameworks|
|Standards and BOKs|
Agile software development describes a set of principles for software development under which requirements and solutions evolve through the collaborative effort of self-organizing cross-functional teams. It advocates adaptive planning, evolutionary development, early delivery, and continuous improvement, and it encourages rapid and flexible response to change. These principles support the definition and continuing evolution of many software development methods.
The term agile was first coined for this in 2001, in the Manifesto for Agile Software Development (Agile Manifesto), and is usually written as Agile (with a capital A).
Incremental software development methods can be traced back to 1957. Evolutionary project management and adaptive software development[note 1] emerged in the early 1970s. During the 1990s, a number of lightweight software development methods evolved in reaction to the prevailing heavyweight methods that critics described as heavily regulated, regimented, and micro-managed. These included: from 1991, rapid application development; from 1994, unified process and dynamic systems development method (DSDM); from 1995, Scrum; from 1996, Crystal Clear and extreme programming (XP); and from 1997, feature-driven development. Although these originated before the publication of the Agile Manifesto in 2001, they are now collectively referred to as agile methods.
In February 2001, seventeen software developers met at the Snowbird resort in Utah to discuss lightweight development methods, among others Jeff Sutherland, Ken Schwaber, and Alistair Cockburn. Together the seventeen published the Manifesto for Agile Software Development, in which they shared that through their combined experience of developing software and helping others to do it they had come to value:
While the secondary concerns were important the primary concerns were more critical to success.
By these terms, they meant:
Some of the authors formed the Agile Alliance, a non-profit organization that promotes software development according to the manifesto's values and principles. Introducing the manifesto on behalf of the Agile Alliance, Jim Highsmith said,
The Agile movement is not anti-methodology, in fact many of us want to restore credibility to the word methodology. We want to restore a balance. We embrace modeling, but not in order to file some diagram in a dusty corporate repository. We embrace documentation, but not hundreds of pages of never-maintained and rarely-used tomes. We plan, but recognize the limits of planning in a turbulent environment. Those who would brand proponents of XP or SCRUM or any of the other Agile Methodologies as "hackers" are ignorant of both the methodologies and the original definition of the term hacker.-- Jim Highsmith, History: The Agile Manifesto
The Agile Manifesto is based on twelve principles:
Later, Ken Schwaber with others founded the Scrum Alliance and created the Certified Scrum Master programs and its derivatives. Schwaber left the Scrum Alliance in the fall of 2009, and founded Scrum.org.
In 2005, a group headed by Alistair Cockburn and Jim Highsmith wrote an addendum of project management principles, the Declaration of Interdependence, to guide software project management according to agile software development methods.
In 2009, a movement by Robert C Martin wrote an extension of software development principles, the Software Craftsmanship Manifesto, to guide agile software development according to professional conduct and mastery.
In 2011 the original Agile Alliance created the Guide to Agile Practices, an evolving open-source compendium of the working definitions of agile practices, terms, and elements, along with interpretations and experience guidelines from the worldwide community of agile practitioners.
Related disciplines, including project management (PRINCE2 and PMI) and business analysis (IIBA) have extended or updated their bodies of knowledge and certifications to embrace working with agile methods.
Most agile development methods break product development work into small increments that minimize the amount of up-front planning and design. Iterations are short time frames (timeboxes) that typically last from one to four weeks. Each iteration involves a cross-functional team working in all functions: planning, analysis, design, coding, unit testing, and acceptance testing. At the end of the iteration a working product is demonstrated to stakeholders. This minimizes overall risk and allows the product to adapt to changes quickly. An iteration might not add enough functionality to warrant a market release, but the goal is to have an available release (with minimal bugs) at the end of each iteration. Multiple iterations might be required to release a product or new features.
Working software is the primary measure of progress.
No matter which development method is followed, every team should include a customer representative (product owner in Scrum). This person is agreed by stakeholders to act on their behalf and makes a personal commitment to being available for developers to answer questions throughout the iteration. At the end of each iteration, stakeholders and the customer representative review progress and re-evaluate priorities with a view to optimizing the return on investment (ROI) and ensuring alignment with customer needs and company goals.
In agile software development, an information radiator is a (normally large) physical display located prominently near the development team, where passers-by can see it. It presents an up-to-date summary of the product development status. A build light indicator may also be used to inform a team about the current status of their product development.
A common characteristic in agile development is the daily stand-up (also known as the daily scrum). In a brief session, team members report to each other what they did the previous day toward their team's iteration goal, what they intend to do today toward the goal, and any roadblocks or impediments they can see to the goal.
Specific tools and techniques, such as continuous integration, automated unit testing, pair programming, test-driven development, design patterns, domain-driven design, code refactoring and other techniques are often used to improve quality and enhance product development agility.
Compared to traditional software engineering, agile software development mainly targets complex systems and product development with dynamic, non-deterministic and non-linear characteristics. Accurate estimates, stable plans, and predictions are often hard to get in early stages, and confidence in them is likely to be low. Agile practitioners will seek to reduce the "leap-of-faith" that is needed before any evidence of value can be obtained. Requirements and design are held to be emergent. Big up-front specifications would probably cause a lot of waste in such cases, i.e., are not economically sound. These basic arguments and previous industry experiences, learned from years of successes and failures, have helped shape agile development's favor of adaptive, iterative and evolutionary development.
Development methods exist on a continuum from adaptive to predictive. Agile methods lie on the adaptive side of this continuum. One key of adaptive development methods is a "Rolling Wave" approach to schedule planning, which identifies milestones but leaves flexibility in the path to reach them, and also allows for the milestones themselves to change. Adaptive methods focus on adapting quickly to changing realities. When the needs of a project change, an adaptive team changes as well. An adaptive team has difficulty describing exactly what will happen in the future. The further away a date is, the more vague an adaptive method is about what will happen on that date. An adaptive team cannot report exactly what tasks they will do next week, but only which features they plan for next month. When asked about a release six months from now, an adaptive team might be able to report only the mission statement for the release, or a statement of expected value vs. cost.
Predictive methods, in contrast, focus on analysing and planning the future in detail and cater for known risks. In the extremes, a predictive team can report exactly what features and tasks are planned for the entire length of the development process. Predictive methods rely on effective early phase analysis and if this goes very wrong, the project may have difficulty changing direction. Predictive teams often institute a change control board to ensure they consider only the most valuable changes.
Risk analysis can be used to choose between adaptive (agile or value-driven) and predictive (plan-driven) methods.Barry Boehm and Richard Turner suggest that each side of the continuum has its own home ground, as follows:
|Agile methods||Plan-driven methods||Formal methods|
|Low criticality||High criticality||Extreme criticality|
|Senior developers||Junior developers(?)||Senior developers|
|Requirements change often||Requirements do not change often||Limited requirements, limited features see Wirth's law[clarification needed]|
|Small number of developers||Large number of developers||Requirements that can be modeled|
|Culture that responds to change||Culture that demands order||Extreme quality|
One of the differences between agile and waterfall is the approach to quality and testing. In the waterfall model, there is always a separate testing phase after a build phase; however, in agile development testing is completed in the same iteration as programming.
Because testing is done in every iteration--which develops a small piece of the software--users can frequently use those new pieces of software and validate the value. After the users know the real value of the updated piece of software, they can make better decisions about the software's future. Having a value retrospective and software re-planning session in each iteration--Scrum typically has iterations of just two weeks--helps the team continuously adapt its plans so as to maximize the value it delivers.
This iterative approach supports a product rather than a project mindset. This provides greater flexibility throughout the development process; whereas on projects the requirements are defined and locked down from the very beginning, making it difficult to change them later. Iterative product development allows the software to evolve in response to changes in business environment or market requirements.
Because of the short iteration style of agile software development, it also has strong connections with the lean startup concept.
In a letter to IEEE Computer, Steven Rakitin expressed cynicism about agile development, calling it "yet another attempt to undermine the discipline of software engineering" and translating "Working software over comprehensive documentation" as "We want to spend all our time coding. Remember, real programmers don't write documentation."
This is disputed by proponents of agile software development, who state that developers should write documentation if that's the best way to achieve the relevant goals, but that there are often better ways to achieve those goals than writing static documentation.Scott Ambler states that documentation should be "Just Barely Good Enough" (JBGE), that too much or comprehensive documentation would usually cause waste, and developers rarely trust detailed documentation because it's usually out of sync with code, while too little documentation may also cause problems for maintenance, communication, learning and knowledge sharing. Alistair Cockburn wrote of the Crystal Clear method:
Crystal considers development a series of co-operative games, and intends that the documentation is enough to help the next win at the next game. The work products for Crystal include use cases, risk list, iteration plan, core domain models, and design notes to inform on choices...however there are no templates for these documents and descriptions are necessarily vague, but the objective is clear, just enough documentation for the next game. I always tend to characterize this to my team as: what would you want to know if you joined the team tomorrow.-- Alistair Cockburn.
Agile methods support a broad range of the software development life cycle. Some focus on the practices (e.g., XP, pragmatic programming, agile modeling), while some focus on managing the flow of work (e.g., Scrum, Kanban). Some support activities for requirements specification and development (e.g., FDD), while some seek to cover the full development life cycle (e.g., DSDM, RUP).
Popular agile software development frameworks include (but are not limited to):
Agile development is supported by a number of concrete practices, covering areas like requirements, design, modelling, coding, testing, planning, risk management, process, quality, etc. Some notable agile practices include:
The Agile Alliance has provided a comprehensive online guide to applying agile these and other practices.
In the literature, different terms refer to the notion of method adaptation, including 'method tailoring', 'method fragment adaptation' and 'situational method engineering'. Method tailoring is defined as:
A process or capability in which human agents determine a system development approach for a specific project situation through responsive changes in, and dynamic interplays between contexts, intentions, and method fragments.-- Mehmet Nafiz Aydin et al., An Agile Information Systems Development Method in use
Potentially, almost all agile methods are suitable for method tailoring. Even the DSDM method is being used for this purpose and has been successfully tailored in a CMM context. Situation-appropriateness can be considered as a distinguishing characteristic between agile methods and traditional software development methods, with the latter being relatively much more rigid and prescriptive. The practical implication is that agile methods allow product development teams to adapt working practices according to the needs of individual products. Practices are concrete activities and products that are part of a method framework. At a more extreme level, the philosophy behind the method, consisting of a number of principles, could be adapted (Aydin, 2004).
Some approaches, such as Scrum and extreme programming, make the need for method adaptation explicit. With these less-prescriptive frameworks, one of the principles is that no single process fits every product development, but rather that practices should be tailored to the needs of the product. Mehdi Mirakhorli proposes a tailoring practice that provides a sufficient road-map and guidelines for adapting all the practices. RDP Practice is designed for customizing XP. This practice, first proposed as a long research paper in the APSO workshop at the ICSE 2008 conference, is currently the only proposed and applicable method for customizing XP. Although it is specifically a solution for XP, this practice has the capability of extending to other methodologies. At first glance, this practice seems to be in the category of static method adaptation but experiences with RDP Practice says that it can be treated like dynamic method adaptation. The distinction between static method adaptation and dynamic method adaptation is subtle.
Scrum isn't designed for method tailoring. Schwaber notes that "Scrum is not a methodology that needs enhancing. That is how we got into trouble in the first place, thinking that the problem was not having a perfect methodology. Effort centers on the changes in the enterprise that is needed." Bas Vodde reinforces this statement, suggesting that Scrum isn't like traditional, large methodologies that require you to "pick and choose" elements. It is the basics on top of which you add additional elements to localise and contextualise its use.
Agile methods have much in common with the Rapid Application Development techniques from the 1980/90s as espoused by James Martin and others. In addition to technology-focused methods, customer-and-design-centered methods, such as Visualization-Driven Rapid Prototyping developed by Brian Willison, work to engage customers and end users to facilitate agile software development.
Further, James M. Kerr and Richard Hunter wrote a book on the subject that presented a day-by-day diary of a real RAD development. It covered work right from inception through to production and contains many of the techniques that forge the backbone of, and are very much present in, today's agile product development approaches.
In 2008 the Software Engineering Institute (SEI) published the technical report "CMMI or Agile: Why Not Embrace Both" to make clear that the Capability Maturity Model Integration and agile methods can co-exist. Modern CMMI-compatible development processes are also iterative. The CMMI Version 1.3 includes tips for implementing CMMI and agile process improvement together.
Organizations that adopt agile development see more frequent releases; which led to the concepts of continuous delivery and DevOps. While DevOps and agile software development both embody many Lean philosophies, such as collaboration and communication, they are distinct concepts. While agile principles represent a change in thinking, DevOps seeks to implement actual organizational cultural change.
Agile development has been widely seen as highly suited to certain types of environments, including small teams of experts working on greenfield projects,:157 and the challenges and limitations encountered in the adoption of agile methods in a large organization with legacy infrastructure are well-documented and understood.
In response, a range of strategies and patterns has evolved for overcoming challenges with large-scale development efforts (>20 developers) or distributed (non-colocated) development teams, amongst other challenges; and there are now several recognised frameworks that seek to mitigate or avoid these challenges.
When agile software development is applied in a distributed setting (with teams dispersed across multiple business locations), it is commonly referred to as distributed agile development. The goal is to leverage the unique benefits offered by each approach. Distributed development allow organizations to build software by strategically setting up teams in different parts of the globe, virtually building software round-the-clock (more commonly referred to as follow-the-sun model). On the other hand, agile development provides increased transparency, continuous feedback and more flexibility when responding to changes.
Agile methods were initially seen as best suitable for non-critical product developments, thereby excluded from use in regulated domains such as medical devices, pharmaceutical, financial, nuclear systems, automotive, and avionics sectors, etc. However, in the last several years, there have been several initiatives for the adaptation of agile methods for these domains.
There are numerous standards that may apply in regulated domains, including ISO 26262, ISO 9000, ISO 9001, and ISO/IEC 15504. A number of key concerns are of particular importance in regulated domains:
Teams and organizations choosing to adopt more agile ways of working typically undertake an agile transformation with the support of one or more agile coaches. The agile coach guides teams through the transformation. There are typically two styles of agile coaching: push based and pull-based agile coaching.
Although agile methods can be used with any programming paradigm or language in practice, they were originally closely associated with object-oriented environments such as Smalltalk and Lisp and later Java. The initial adopters of agile methods were usually small to medium-sized teams working on unprecedented systems with requirements that were difficult to finalize and likely to change as the system was being developed. This section describes common problems that organizations encounter when they try to adopt agile methods as well as various techniques to measure the quality and performance of agile teams.
Organizations and teams implementing agile development often face difficulties transitioning from more traditional methods such as waterfall development, such as teams having an agile process forced on them. These are often termed agile anti-patterns or more commonly agile smells. Below are some common examples:
A goal of agile software development is to focus more on producing working software and less on documentation. This is in contrast to waterfall models where the process is often highly controlled and minor changes to the system require significant revision of supporting documentation. However, this does not justify completely doing without any analysis or design at all. Failure to pay attention to design can cause a team to proceed rapidly at first but then to have significant rework required as they attempt to scale up the system. One of the key features of agile software development is that it is iterative. When done correctly design emerges as the system is developed and commonalities and opportunities for re-use are discovered.
In agile software development, stories (similar to use case descriptions) are typically used to define requirements and an iteration is a short period of time during which the team commits to specific goals. Adding stories to an iteration in progress is detrimental to a good flow of work. These should be added to the product backlog and prioritized for a subsequent iteration or in rare cases the iteration could be cancelled.
This does not mean that a story cannot expand. Teams must deal with new information, which may produce additional tasks for a story. If the new information prevents the story from being completed during the iteration, then it should be carried over to a subsequent iteration. However, it should be prioritized against all remaining stories, as the new information may have changed the story's original priority.
Agile software development is often implemented as a grassroots effort in organizations by software development teams trying to optimize their development processes and ensure consistency in the software development life cycle. By not having sponsor support, teams may face difficulties and resistance from business partners, other development teams and management. Additionally, they may suffer without appropriate funding and resources. This increases the likelihood of failure.
A survey performed by Version One found respondents cited insufficient training as the most significant cause for failed agile implementations Teams have fallen into the trap of assuming the reduced processes of agile development compared to other methodologies such as waterfall means that there are no actual rules for agile development. Agile development is a set of prescribed methodologies, and training/practice is a requirement.
A common mistake is to have the product owner role filled by someone from the development team. This requires the team to make its own decisions on prioritization without real feedback from the business. They try to solve business issues internally or delay work as they reach outside the team for direction. This often leads to distraction and a breakdown in collaboration.
The agile process requires teams to meet product commitments, which means they should focus only on work for that product. However, team members who appear to have spare capacity are often expected to take on other work, which makes it difficult for them to help complete the work to which their team had committed.
Teams may fall into the trap of spending too much time preparing or planning. This is a common trap for teams less familiar with the agile process where the teams feel obliged to have a complete understanding and specification of all stories. Teams should be prepared to move forward only with those stories in which they have confidence, then during the iteration continue to discover and prepare work for subsequent iterations (often referred to as backlog refinement or grooming).
A daily standup should be a focused, timely meeting where all team members disseminate information. If problem-solving occurs, it often can only involve certain team members and potentially is not the best use of the entire team's time. If during the daily standup the team starts diving into problem-solving, it should be tabled until a sub-team can discuss, usually immediately after the standup completes.
One of the intended benefits of agile development is to empower the team to make choices, as they are closest to the problem. Additionally, they should make choices as close to implementation as possible, to use more timely information in the decision. If team members are assigned tasks by others or too early in the process, the benefits of localized and timely decision making can be lost.
Being assigned work also constrains team members into certain roles (for example, team member A must always do the database work), which limits opportunities for cross-training. Team members themselves can choose to take on tasks that stretch their abilities and provide cross-training opportunities.
Another common pitfall is for a scrum master to act as a contributor. While not prohibited by the Scrum methodology, the scrum master needs to ensure they have the capacity to act in the role of scrum master first and not working on development tasks. A scrum master's role is to facilitate the process rather than create the product.
Having the scrum master also multitasking may result in too many context switches to be productive. Additionally, as a scrum master is responsible for ensuring roadblocks are removed so that the team can make forward progress, the benefit gained by individual tasks moving forward may not outweigh roadblocks that are deferred due to lack of capacity.
Due to the iterative nature of agile development, multiple rounds of testing are often needed. Automated testing helps reduce the impact of repeated unit, integration, and regression tests and frees developers and testers to focus on higher value work.
Test automation also supports continued refactoring required by iterative software development. Allowing a developer to quickly run tests to confirm refactoring has not modified the functionality of the application may reduce the workload and increase confidence that cleanup efforts have not introduced new defects.
Focusing on delivering new functionality may result in increased technical debt. The team must allow themselves time for defect remediation and refactoring. Technical debt hinders planning abilities by increasing the amount of unscheduled work as production defects distract the team from further progress.
A common misconception is that agile development allows continuous change, however an iteration backlog is an agreement of what work can be completed during an iteration. Having too much work-in-progress (WIP) results in inefficiencies such as context-switching and queueing. The team must avoid feeling pressured into taking on additional work.
Agile development fixes time (iteration duration), quality, and ideally resources in advance (though maintaining fixed resources may be difficult if developers are often pulled away from tasks to handle production incidents), while the scope remains variable. The customer or product owner often pushes for a fixed scope for an iteration. However, teams should be reluctant to commit to locked time, resources and scope (commonly known as the project management triangle). Efforts to add scope to the fixed time and resources of agile development may result in decreased quality.
The best agile practitioners have always emphasized sound engineering principles. As a result, there are a number of agile best practices and tools for measuring the performance of product development and teams.
The Agility Index Measurements (AIM) scores product development against a number of agility factors to achieve a total. The similarly named Agility Measurement Index, scores developments against five dimensions of product development (duration, risk, novelty, effort, and interaction).
One of the early studies reporting gains in quality, productivity, and business satisfaction by using agile methods was a survey conducted by Shine Technologies from November 2002 to January 2003.
A similar survey, the State of Agile, is conducted every year starting in 2006 with thousands of participants from around the software development community. This tracks trends on the benefits of agile, lessons learned, and good practices. Each survey has reported increasing numbers saying that agile development helps them deliver software faster; improves their ability to manage changing customer priorities; and increases their productivity. Surveys have also consistently shown better results with agile product development methods compared to classical project management. In balance, there are reports that some feel that agile development methods are still too young to enable extensive academic research of their success.
Agile methodologies can be inefficient in large organizations and certain types of developments. Many organizations believe that agile methodologies are too extreme and adopt a hybrid approach that mixes elements of agile and plan-driven approaches. Some methods, such as Dynamic systems development method (DSDM), have combined elements of agile and plan-driven approaches in a disciplined way, without sacrificing the fundamental principles that make agile work.
The term "agile" has also been criticized as being a management fad that simply describes existing good practices under new jargon, promotes a "one size fits all" mindset towards development strategies, and wrongly emphasizes method over results.
Alistair Cockburn organized a celebration of the 10th anniversary of the Agile Manifesto in Snowbird, Utah on 12 February 2011, gathering some 30+ people who had been involved at the original meeting and since. A list of about 20 elephants in the room ("undiscussable" agile topics/issues) were collected, including aspects: the alliances, failures and limitations of agile practices and context (possible causes: commercial interests, decontextualization, no obvious way to make progress based on failure, limited objective evidence, cognitive biases and reasoning fallacies), politics and culture. As Philippe Kruchten wrote:
The agile movement is in some ways a bit like a teenager: very self-conscious, checking constantly its appearance in a mirror, accepting few criticisms, only interested in being with its peers, rejecting en bloc all wisdom from the past, just because it is from the past, adopting fads and new jargon, at times cocky and arrogant. But I have no doubts that it will mature further, become more open to the outside world, more reflective, and also therefore more effective.-- Philippe Kruchten
Agile methods have been extensively used for development of software products and some of them use certain characteristics of software, such as object technologies. However, these techniques can be applied to the development of non-software products, such as computers, motor vehicles, medical devices, food, clothing, and music; see Flexible product development. Agile methods have been used in non-development IT infrastructure deployments and migrations. Some of the wider principles of agile have also found application in general management (e.g., strategy, governance, risk, finance) under the terms business agility or agile business management.
Under an agile business management model, agile techniques, practices, principles and values are expressed across five domains.
Agile development paradigms can be used in other areas of life such as raising children. Its success in child development might be founded on some basic management principles; communication, adaptation and awareness. Bruce Feiler has claimed that the basic agile development paradigms can be applied to household management and raising children. In his TED Talk "Agile programming - for your family", these paradigms brought significant changes to his household environment, such as the kids doing dishes, taking out the trash, and decreasing his children's emotional outbreaks, which inadvertently increased their emotional stability.
^ Note by Edmonds: I presented these ideas in London in 1970 and first submitted the paper to the Journal Computer Aided Design. It was rejected with the comment "If you don't know what you are going to do before you start you shouldn't start"! Only then did I submit it to General Systems.
What is a self-organizing team?
We were doing incremental development as early as 1957, in Los Angeles, under the direction of Bernie Dimsdale at IBM's Service Bureau Corporation. He was a colleague of John von Neumann, so perhaps he learned it there, or assumed it as totally natural. I do remember Herb Jacobs (primarily, though we all participated) developing a large simulation for Motorola, where the technique used was, as far as I can tell ... All of us, as far as I can remember, thought waterfalling of a huge project was rather stupid, or at least ignorant of the realities. I think what the waterfall description did for us was make us realize that we were doing something else, something unnamed except for 'software development.'
The article titled 'Agile Software Development: The Business of Innovation' . . . is yet another attempt to undermine the discipline of software engineering . . . We want to spend all our time coding. Remember, real programmers don't write documentation.
95% stated that there was either no effect or a cost reduction ... 93% stated that productivity was better or significantly better ... 88% stated that quality was better or significantly better ... 83% stated that business satisfaction was better or significantly better
Only 6% indicated that their productivity was lowered ... No change in productivity was reported by 34% of respondents and 60% reported increased productivity ... 66% [responded] that the quality is higher ... 58% of organizations report improved satisfaction, whereas only 3% report reduced satisfaction.
Manage research, learning and skills at defaultLogic. Create an account using LinkedIn or facebook to manage and organize your IT knowledge. defaultLogic works like a shopping cart for information -- helping you to save, discuss and share.