I was recently asked this question and to be honest, I was flabbergasted. Understanding when to use agile and when to use six sigma (or some other highly structured method) is almost a decade old problem which is well known.
However, I was asked to write a post on this and so with much gnashing of teeth, I give you the stuff which you should already know and if you don't then please stop trying to write systems and go and do some basic learning instead.
First, every system of any reasonable scale consists of multiple components. Those components are all evolving (due to the effects of competition) and you can (and should) map out the components of any system before embarking on trying to build it. Below in figure 1 is a basic map from a heavy engineering project with a large IT component.
Now, all the components are evolving from left to right and as they do so their characteristics change - they move from an uncharted space (the novel, the chaotic, unpredictable, uncertain, potential differential) to the more industrialised (the common, appearance of linear order, the predictable, the certain, the cost of doing business). See figure 2.
Figure 2 - Basic Characteristics
Agile (e.g. XP, Scrum) and Structured (e.g. Six Sigma, Prince 2) methods are suited to different problem domains. Agile, through the use of test driven development attempts to reduce the cost of change throughout a projects life to a baseline but it does so at a slight cost compared to the cost of change in the requirements / design phase of a structured method. An approximate of this is given in figure 3.
Figure 3 - Cost of Change by Method
Hence for those uncharted activities where there is rapid change (due to their properties i.e. they're uncertain and we're discovering them) then Agile is the most appropriate method because whilst it might have a higher cost of change compared to the requirements / design phase of a structured method, there will however be change throughout the project's life.
Alternatively, those more industrialised acts that are defined, well understood and highly repeatable are more effectively managed by a highly structured method because change shouldn't exist and where it is necessary it should be eliminated in the requirement / design phase. This leads to a situation where different methods (e.g. Agile, Six Sigma) are suitable for different evolutionary stages - see figure 4.
Figure 4 - Method by Evolutionary Stage
So when examining your map, it should be fairly obvious that you need multiple methods with two extremes of Agile (e.g. XP) including development in-house and Six Sigma with potential for outsourcing. For those transitional activities (i.e. products) then you should be using products and minimising configuration / waste / unnecessary change where possible which requires another set of methods. Hence for our heavy engineering project we could use the following - see figure 5.
The anomaly in the above is 'web site' being a commodity but using Agile methods. This is because maps are imperfect and 'web site' is in fact a value chain of multiple activities, some of which are more commodity whilst others are novel and new. This needs to be broken down and in the case of the engineering project it was. One final note is that all the activities are evolving due to competition and hence the method you will use for one activity will change as it evolves.
Misapplication of methods will have costly effects. For example, if you outsource the entire value chain described by the map to a third party then you're likely to settle on a structured method because you'll want 'guarantees' of what is being delivered. The net effect of applying a single structured method (such as six sigma) across the entire value chain is that whilst some activities will be effectively treated, you'll also be applying structured methods to activities that are rapidly changing. The end result will always be cost overruns due to change control costs associated with structured methods i.e. you're applying a method designed to reduce deviation to an activity that is constantly deviating - see figure 6.
Figure 6 - Single Method.
At Fotango we went all agile in 2001 and then by 2003 had learned our lesson - multiple methods were needed. I first spoke about this need for multiple methods at a public conference in 2004 and have repeated this countless times. Since about 2007, the use of multiple methods and application of appropriate methods has increasingly become common. The above is of course a fairly basic interpretation.
It's 2013 now ... come on people get with the program, we've had almost a decade of this agile versus six sigma nonsense! It's really only the very extreme of the laggards who describe themselves either as an 'Agile' or 'Six Sigma' shop. There is no 'one size fits all' and you need to use methods where appropriate.
On a personal note
I'm extremely disappointed to be asked this 'Agile vs Six Sigma' question in 2013. The next person will get a boot up the backside from me. Either learn your techniques and how and when to apply them or find another job where you'll do less damage.