9

I'm very positive towards agile development, and have worked on agile projects on and off for about 13 years. But I have a concern that I've never really been able to address. It doesn't always seem to manifest, but it has bitten me a few times.

Agile seems to be in some sense a 'greedy algorithm'. Start with the highest value story, optimise the system to precisely fulfil that story, and repeat.

Actual greedy algorithms are prone to suffering from converging to locally optimal solutions, while missing a globally optimal solution.

Has this been people's experience?

Is it actually a problem?

If so, what techniques do you use to avoid such local optima and yet remain agile?

Burleigh Bear
  • 3,274
  • 22
  • 32
  • 1
    I'm voting to close this question as off-topic because project management is off-topic. Such questions should be asked at [https://pm.stackexchange.com/](https://pm.stackexchange.com/) – BDL Aug 28 '17 at 12:17
  • I'm voting to close this question as off-topic because such questions should be asked at https://pm.stackexchange.com/ – Olivier Pons Nov 04 '17 at 08:23

3 Answers3

4

Actual greedy algorithms are prone to suffering from converging to locally optimal solutions, while missing a globally optimal solution.

This holds true if EPIC technical User Story and guideline is not established, along with the normal business EPIC user story.

Has this been people's experience?

At times yes, it has been my experience. One instance was when the user stories we worked on were broken down too much, and the solution was to broaden them to get a more global outlook at our designs. And at times it was different enterprise scrum teams in the same projectt, conflicting with different technical framework uses and approaches.

Is it actually a problem?

It is only a problem, if you ignore the technical EPIC user story or guideline.

If so, what techniques do you use to avoid such local optima and yet remain agile?

Here is one Agile approach to solving this: During Agile Release planning, instead of just coming up with a Business EPIC User Story, also come up with a Technical EPIC User Story. The Technical EPIC User story would have the product vision from a technical stand point, in terms of technical architecture, application framework, quality standards, and global design considerations etc. These could be broken down into smaller technical user stories, and have a Scrum Team which works on getting those user stories working. An example of a user story could be: "As a Technical Project Manager, I want the whole enterprise project using A, B, C framework, and coding as per X,Y,Z coding standards, so that there is uniformity in project development work. If you don't want to form a scrum team separately for this, then just keep them as reminder cards next to backlogs for development teams to use as guidelines.

As a testing guideline, we used to have successful integration testing as a done criteria for each backlog. A global test was conducted in an integration environment, on all working software deployed from all enterprise teams, to deem it shippable. So right from inception to end of the backlog, the theme is set for global working software and not just local working software.

Finally, Agile development involves keeping a constant eye on quality, and one of the quality issues could be bad design or a too localized design. As and when this is discovered, it should be redesigned within that backlog itself, and followed going forward for other Backlogs.

sjt
  • 1,607
  • 11
  • 14
  • Thanks for taking the time to answer. I'm not sure I understand, though. If you take a technical epic, and solve it one story at a time, isn't that prone to exactly the type of local optima that a functionality epic is? I suppose what I'm trying to say is that it doesn't really matter what you're trying to optimise, if you attack it purely piecemeal, then you run the risk of not converging to a global optimum. – Burleigh Bear Oct 28 '10 at 05:09
  • I wish I could talk to you instead of typing my explanation, it would have been much easier to explain :) What I meant was not to tackle each technical story one at a time, I agree that that would not solve this issue. Let me give you an example - We had a Technical Architecture best practices team which included 4 members. They received user stories which mainly had the purpose of making sure global design is considered, and helping achieve that. There will always be a risk of going with a bad design irrespective of whether you follow agile or waterfall or RUP. – sjt Oct 28 '10 at 16:40
  • The key here is 'adapt' by finding ways to avoid those situations and following them. Can you tell me a specific instance where local optimization occurred in your organisation, I will be able to help you better by suggesting a solution to that specific example, while still being agile. – sjt Oct 28 '10 at 16:42
  • The team of 4 I was referring to mostly assisted development teams. – sjt Oct 28 '10 at 17:35
  • +1. I think I understand better now - thanks for bearing with me. So, the question I have is: would you then consider this approach, where you consider all (or many) of the technical stories at once, agile? (Regardless, I agree totally with your observation that there's always the danger of sub-optimal design). – Burleigh Bear Oct 28 '10 at 20:40
1

I've been on a project which has had this problem, and has not dealt with it effectively.

The local quality of the code - over the scale of a package, say - was not bad. But there were problems at larger scales; things like duplication of logic (but not code) between packages, use of batch recomputation jobs where we should be using event-driven approaches, splitting the system into separate services at the wrong place, etc.

None of these problems could be fixed by refactoring a single class or package. As a result, they never happened in the normal course of events. We did refactorings at smaller scale - when adding a feature, we'd refactor in that area before starting, and again after we finished (as well as making some effort to write good code as we were going). But that never led to refactoring larger, architectural concerns.

We were all conscious of the problems, we just didn't have anything in our process that let us fix them.

One notable victory we did have was where there was duplication between two distantly related module. Essentially, there was code to render a web page showing the results of some set of calculations, and also a background job to generate reports doing similar calculations. The calculation code was shared, but the code to set up the calculations was not; one was driven by a user's view preferences, whereas the other was driven by a configured reporting job. We had a feature to implement that would have involved adding a new aspect to the calculations, which would have meant adding more items to both kinds of configuration, and then adding business logic to both sets of calculation setup code. We managed to get the product manager (our customer proxy) to agree to budget enough time for the work that we could refactor to unite the ideas of user view preferences and configured reporting job, so throwing away one of the sides of the duplication, then implement the feature. This took longer than just implementing it twice, but the product manager was wise enough to realise that this would let us implement future features spanning both pages and reports more quickly.

The mechanism in the process by which we did this was writing stories for the job of refactoring. Essentially, something like "As a product manager, i want pages and reports to use common calculation setup code, so that i can get features added more quickly". This is absolutely not a proper kind of story, but it fitted in the system, and it did the job.

I think that if the running of this project had been a bit healthier, then there would have been a steady stream of stories like this. We would acknowledge that we had a lot of architectural debt, and that work to pay it off had value, and allocate a fixed fraction of our time to it, perhaps about 20% (which would really mean one pair at a time). We could then have generated features/epics, stories, and tasks just as we did for customer-oriented work. These would originate from the team themself, rather than the product managers.

Sadly, there wasn't quite enough communication and trust between the development and product management sides that this was feasible; we could say to product that we had a problem, it was important, and that it would take so long to fix, and they couldn't know if that was true or not. As such, they were generally unwilling to schedule time to do it. The sad thing was that everyone was in agreement that there were problems and it would be good to fix them, we just had an impasse over actually doing it.

Tom Anderson
  • 46,189
  • 17
  • 92
  • 133
0

in my experirence, if you´re working a project context with fixed time/requirements then yes, most of the times Agile leads to local optima.

But my point is that in a complex endeavour, the requirements, the team itself and even the goals change. Agile is also about embracing changes. Then, paradoxically, this greedy strategy arrises as a reasonable option for global optimization when dealing with moving targets.