Decentralized Agile Development Using Scrum

This topic contains the drivers/rationale for adopting a decentralized, agile development approach using Scrum as the Project Management methodology. In addition to using Open Source tools which have achieved critical mass, we use aspects of the Open Source development method which are appropriate for an environment where speed of delivery of quality software for specific business customers is the objective (which is a bit of a different market than typical successful Open Source software efforts).

Background

In 2000, we decided to build software using decentralized agility for the following reasons:

• Use of Open Source software which has achieved critical mass often provided equal or superior quality to market leaders at the low low price of nothing. Given that we planned to use talented developers, the impact of having little documentation or support (unless paid for) was not believed to be an issue and in fact has not been an issue
• Demographics: Given the low cost of communication, the availability of free collaboration tools (instant messaging, wikis, free conference calls, email, Google, cvs, bugzilla etc), and a demographic that is more and more accustom to collaborating synchronously and asynchronously over the web (collaborative war games, Myspace, political blogs etc) it seemed logical to build a project culture that matched the societal trends
• "The World is Flat" - the reality of the playing field being leveled has it's advantages. If an organization has a culture which is optimized towards collaborating with great talent, regardless of time and space, the availability of resources and ability to make progress is increased significantly
• Lean/Agility - using the principle of Simplicity (Agile Manifesto says Simplicity is the art of maximizing what is NOT DONE), adopting an agile software development methodology has the potential of solving business problems while producing less software with higher quality. This requires disciplined execution of engineering practices and when coupled with Scrum as a project management approach, provides the visibility which allows for continuous improvement.
• Wikis - wiki technology is one of the key Open Source enabling platforms providing the following:
  • knowledge management capability
  • asynchronous and near synchronous collaboration
  • a platform where business process automation can be rapidly prototyped
  • built in social networking features useful in building and maintaining team chemistry

Statement of the Problem

• Build the right software for the right market using the right resources delivered at the right time. All of this done with HIGH quality.

Agile/Open Source Harmony and Dissonance

Characteristics of Agility and contributions to addressing The Problem

Scrum is a Project Management Approach for Agile projects. Xtreme Programming or XP is a collection of Engineering and extremely light-weight project management approaches. Scrum leverages XP to provide a robust, CMMI certifiable (there are examples of CMMI Level 5 companies using Scrum and XP which I will just call Scrum for most of this topic unless necessary for distinction)

• Face to Face interaction is highly valued in XP as it results in high bandwidth information transfer (decentralized communication takes more energy to communicate equivalent information from collaboration perspective).
• High Quality deliverables through Pair Programing and Test Driven Development and obsessive test automation
• Continuous Integration (practice where by software is checked in with high frequency and where automated tests are run which determine whether or not deployment occurs to a server for consumption by a broader audience), once it gets going, contributes greatly to the timely delivery of high quality software.
• Delivery of the Right software at the Right time is core to agility. Xtreme Programming approach is driven by Small Releases delivered to the Customer priority order. Scrum formalizes the process of determining and developing software (or any deliverable) in accordance with Customer priorities using the Role of Product Owner and the artifact of the Product Backlog. The Product Owner maintains the backlog in priority order and the Team pulls things from the backlog for implementation in priority order.

Characteristics of Open Source and contributions to addressing The Problem

• Environment optimized for decentralized developers who will never meet - Self selection and the judgement of "committer" role in determining the quality of code submitted by "coders" and the reputation of the coder submitting the change is important and on successful projects lead to high quality software. They determine priority of submission often via "Wisdom of Crowds" approach, leveraging bugzilla, blogs and patches written by users.
• Low barrier to entry for participation in terms of Quality of Life
  • Work from home or desired location
  • Work anytime desired (blow up notion of workday or workweek and contributions based upon value and timeliness vs. specifics of when and how produced)
  • Color-blind and Culture blind contributions enabled
• Low barrier to entry for participation in terms of tools (free, simple grep"able" text based) which include
  • bug tracking (often bugzilla)
  • source control (often cvs and more recently subversion)
  • text based ide
• Provide Total Visibility to all "stakeholders" (developers, users, maintainers, tool buiders)
• Users are often developers so, often times, successful projects (like eclipse, apache etc) often get developers from user community - the fact that users are often developers lead to HIGH quality deliverables because the requirements are known by those writing the software as well as contributers who want it changed.
• Developers tend to produce higher quality deliverables because given that there is a chance that their source will be read by others, there is pride and reputation at stake
• There is no formal requirements collection/specification approach. There is almost a hidden assumption that requirements begin to get collected as soon as it is in production... used by user. The defect list becomes a feature list. Also, Patches or Fixes provided by the user community end up representing strong votes in the prioritization of feature inclusions for future releases.
• Developers are energized by getting direct feedback from user community (vs. having intermediary like PM or QA). This enthusiasm also leads to higher quality software

Harmony

• Scrum/Agile and Open Source both can have high performing teams
• Scrum/Agile and Open Source when done well, deliver the right software to the right customer. Agility is driven more by timeboxes and priorities set by SPECIFIC customers where as Open Source is driven by priorities set by a "Wise Crowd" of users and maintainers.
• Scrum/Agile and Open Source are both empirical approaches in that they have strong aspects of inspection, adaptation and transparency.

Dissonance

• Agile Approaches value face to face interaction for high bandwidth communcation where as Open Source values decentralized communication as it allows for getting best, most enthusiastic volunteers independent of time and space
• Agility has great formality with respect to iterations and timeboxes where as open source is decentralized and deliveries arrive when they are ready and are integrated in accordance with multiple agendas
• Agility strives to have lots of face to face interactions with the "Customer/Stakeholder" where as Open Source is an environment where the interaction is between developers and users using bug reports and checkin's of code as communication/collaboration mechanism.
• Summary of Dissonance is that Agility is often targeted at concrete customer or Product Owner with concrete timeline requirements and Open Source is often targeted at End Users with less politics of stake holder management vs. intentional crowd dynamics

What We've Done (The BlueJean Platform)

As a company which provides software as internet subscribe-able Web Service with less concrete time-lines and priorities driven by intentional crowd dynamics (Open Source characteristics) and AND custom software development services to customers with more concrete time-lines and priorities driven by stakeholder politics AND in both cases a developer/contributor dynamic where we desire HIGH performance teams of motivated workers with low barrier to entry from both quality of life and tools perspective (Open Source) AND where interaction between each other and Customer is high bandwidth. So, we want to Deliver the Right SW, to the Right Customer, at the Right time with HIGH quality and based upon our assessment of demographics and a "Flat World" we want a development environment as described. We call our specific approach The BlueJean Platform Methodology (no cool acronym) which can be summarized as Decentralized Scrum where we use aspects from both Agility/XP and from Open Source to achieve our desired culture.

From Scrum/Agility we take the following: Disciplined execution of Scrum methodology

• Test Driven Development
• Use of Product Owners and Product Backlogs in the execution of all projects, software and operations... we are migrating to doing all that can be done using Scrum with Scrum (including Marketing, IT and aspects of HR)
• Continuous Integration
• Small and Frequent Releases
• Collective code ownership
• Continuous Improvement through Retrospectives and Kaizen moments (Lean Concept but, Lean is in the Scrum/Agile lineage so why not)
• Execution in Sprint/Iteration increments

From Open Source, we take the following

• High visibility of the state of a non-privacy information like defects, business plans etc (customers, users, developers, employees etc)... our belief is that the benefit of getting feedback/criticism/votes from employees, customers etc is WAAAAYYYY more valuable than the risk of members of the potentially wise crowd stealing an idea.
• We lower the barriers to entry for team participation from both quality of life (use wiki, phone conference, daily checkins to enable decentralized asynchronous and synchronous collaboration) and tools (extreme use of open source software which has achieved critical mass like twiki, tomcat/JBoss, eclipse, Mysql, Linux, cvs/subversion, cruise control... )
• Provide mechanism for end users to provide immediate feedback to developer community where feedback is visible to all stake and we are working on opening up interfaces to allow for software contributions from the user community
• Provide visibility to end user into code contributors of business services to achieve the reputation aspect of Open Source quality

Retrospective

What went well

• Mastered decentralized development approach - which has resulted in high quality, enthusiastic contributions to project successes
  • Have recruited experts living in multiple parts of the country contributing from management, software development, analysis, testing and management consulting perspectives
  • Management, employees and customers all have timely access of both the state of projects as well as contributions made by participants and relatedly, evidence of work by team members from an accountability perspective
• Agility has led to stronger customer relationship as well as an ability to make lightening fast changes in workflow and operations in response to changing priorities and external environment
• Team members are engaged, morale has improved
• Customers have been able to achieve with our team what was unachievable by others seeking to apply a define approach (waterfall) to a environment where requirements are either not known or rapidly changing (perfect for empirical approach).
• Speed to market has improved dramatically since adopting Scrum and adapting it to the decentralized team structure

Opportunities for Improvement

• More disciplined execution of engineering practices (Test Driven Development, Continuous Integration)
• More success with Self Organizing Teams (we've seen pockets of it, but, haven't been able to get teams to fully own the responsibility of successful delivery of Sprint objectives... usually requires a management type to make sure that all requirements are met for "doneness")
• So far, has been harder than desired to get new team members assimilated into the decentralized agile team culture

Conclusions

The approach that we are using represents an approach to leverage the current human capital demographic, tool sets and low communication cost to meet the needs of an environment where innovation occurs in months vs years and threats, more and more, are coming from individuals vs. organizations.

Reference Examples

The following are a subset of the projects which we've executing using the above approach and tools

• Banking Application for Federal Government Agency
• Web-based Time Management Software as a Subscribe-able Service
• Prince Georges County Housing Web Site development project
• Fairfax County Youth Football League Weigh In and Roster Software as a Subscribe-able Service
• BlueJean Platform Test and Simulation Tool Software as a Subscribe-able Service

Discussion

You must register in order to comment on this topic.	SolomonThompson	24 Sep 2008 - 13:42
All, please feel free to provide questions or feedback.	SolomonThompson	02 Sep 2008 - 12:54