Scaling Agile @ OTTO

At the beginning of OTTO's Agile journey from 2011 onwards, all the 'Feature teams' (FTs) involved in the development of the Webshop were agile in the sense that they worked based on Scrum. Maybe not exactly according to textbook Scrum, but quite close. For a company that had previously driven traditional project management as defined by ITIL, with a pool of developers from which the project teams were recruited and with project durations of 18 months, this was already major progress.

In a complex system, nobody can predict how a specific change will be received by the customer, so only constant testing on the customer will help. Of course, this entails that deviations from whatever the plan is become normal – and that far-reaching solution-level plans which stretch too far into the future make very little sense. Staying aware of the objectives is all the more important in preserving clarity on which parameters should be measured.

“Without data you are just another person with an opinion.“ -- W. E. Deming

In order to develop software further in a working approach like this, you need a diverse range of know-how. To be able to work at high speed in a perpetual cycle, you definitely need these skills in your team – otherwise you'll constantly be chewing your nails during the frequent waiting times in your feedback cycle.

Incidentally, the teams must also be permanently able to go into production independently of other teams in a largely automated manner, which is why many teams at OTTO now practice Continuous Deployment: practically every single 'git push' triggers a deployment pipeline and therefore leads to a go-live. Thousands of deployments per day are now complete normality.

In our company, this means that all Feature teams are cross-functional. Whether it's Frontend and Backend Development, Operations, Quality Assurance, Web Analysts, or User Experience (UX), all the expertise needed to develop and operate software is available within the team itself as often as possible.

Some roles are also shared. For example, Quality Assurance could be handled by 'the Developers' instead of by a single, dedicated Tester. In other cases (e.g. Analysts, UX), one expert is assigned to several teams. However, cross-team roles like this are often an unfortunate compromise: either they only participate in every second routine meeting, or they have two at the same time – and somehow they only partly 'belong' to the team. A clear 1:1 assignment would be better but sometimes it's impossible to implement this sensibly and we have to live with a compromise.

As a general rule, some team-members have several roles. Although there are also explicit 'Frontend Developers' and 'Backend Developers', most teams prefer Developers who are at least willing to slip into a different role – for instance to test and explore the new software's limits for a day, or to plunge into the abyss of JavaScript and different browser versions. I should tell you at this point that we work with microfrontends and don't have a monolithic frontend. This offers many advantages and is worth a blogpost in its own right! One of the advantages, however, is that it often makes continuous customer-effective Feature Development at OTTO possible without interacting with other teams.

Work in a Feature team is therefore very diverse – which implies that every experienced person will come across things they still need to learn. Paradise for curious spirits!

Long gone are the 'bad old days' when a pure Development team threw a new chunk of software developed under time-pressure over the fence to an understandably risk-averse Operations team... or even argued with them about whether it's actually necessary to migrate to a newer version of the JDK again.

Instead, today all Feature teams also hold responsibility for operating 'their' software. Obviously, this expands the number of roles that a cross-functional team has to take on. Some teams have one or more 'Ops Specialists', whose main task is the operational topics, while in other teams the Developers share these tasks.

Since our entire infrastructure has now moved to the cloud, the boundaries between Dev and Ops are becoming increasingly blurred anyway. Infrastructure as Code sounds like a good thing (it is), but it still means that a team has to master an additional tools 'zoo'.

A common pattern is the daily assignment of operational topics to a 'pair' – pair programming is very common with us and pair operations, pair testing or simply 'pairing' is therefore normal in most teams. The so-called "Daily Action Pair" is responsible for everything that's difficult to plan, including critical bugs, analysing log files, looking at the metrics, improving the deployment pipelines, updating service dependencies, bringing the next Java version into production, improving the automation of standard tasks, being the person in charge for other teams' ad hoc requests – whatever's needed!

Nobody expects everyone in the team to be able to handle everything. It's enough if the overall team can complete all tasks, which means that the individual (even if this hurts) somehow becomes dispensable. DevOps is not a role, DevOps is a mindset.

The Feature teams at OTTO are therefore Agile, cross-functional teams that are end-to-end responsible to a very large degree for their product (see also Products over Projects by Siriam Narajan). Unlike in a project organisation, at OTTO we develop software intended to be used by and for our business for several years. Software like this is never 'finished' and the initial development is not sufficient. Someone has to take responsibility for operation, bug-fixing, monitoring and of course further development.

We try to define and delineate the responsibility of the teams by business criteria as far as possible. For instance, Team X is responsible for the shopping cart, Team Y for the product recommendations. The goal is to define the teams so that they can deliver changes to the customer in most situations without needing to involve another team. With more than 40 teams currently working in parallel, this is our biggest challenge right now.

Wherever possible we try to avoid technical delineations (e.g. Team X does the frontend, Team Y the backend) or delineations by roles (e.g. Team X develops, Team Y runs the software). This is because delineations like this almost always lead in practice to tightly coupled teams which are hardly able to take action without involving other teams.

A team's product responsibility needs to be extensive because the team will gather both technical and business expert knowledge that can't be gathered to the same extent outside the team.

• The team should be able to further develop the product functions 'close to the customer' and continuously carry out A/B testing.
• Depending on the test results, further development should be adaptable. The fewer process hurdles there are, the better and more efficiently development will progress.
• Since the team is also responsible for Operations, no one knows better whether and when measures are required to restore or maintain quality. The question of whether one 'technical' measure is more important than the next technical topic can be taken quickly and responsibly in a team that holds product responsibility.

At OTTO, among other things 'product responsibility in the team' means that product management – and therefore technical responsibility – is part of the team's mandate. For us, Feature teams are not pure Development teams made up of a bunch of techies. In each team, at least one person is responsible for the business roadmap and, as we will see in a moment, not in a secondary role, but in a decisive one.

In Scrum there's the outstandingly important role of the Product Owner. This role is generally placed outside the Development teams – so to speak as a proxy of the departments interfacing with the Development team – and it is in the hands of one person.

We found this very impractical.

If you're dealing with numerous teams and have to minimize their external dependencies (to avoid waiting times in Development – e.g. Team A can only continue with project X when Team B has completed something), the Product Owner role belongs in the teams. Only then can we avoid:

• every tiny detail having to be worked out in advance, including all eventualities, and
• waiting times occurring for every change or query, because you have to schedule a meeting with the Product Owner first.

A further problem is that there are always different drivers for the required development efforts, at least business and technical factors. Differing competencies are also called for to assess the urgency of the specific development case: is it more important to build the next feature, or do we need to improve monitoring first? Both are important – so we must not only make the decision from a business perspective, and neither purely from a technical one. 

Almost nobody is able to assess a fairly complex product from both a technical and a business perspective equally competently. It's also tremendously difficult to assign this role competently – so we decided not to attempt it! We split Product Ownership in each team into a business and a technical facet, with a Business Designer responsible for the business side and a Technical Designer for the technical side (cool labels are unfortunately not our strong point). Both have to agree on what the team will tackle next and neither can simply outvote the other – at least theoretically.

When we then add the Project Lead, Business Designers and Technical Designers form a 'triad'. This term is a little bizarre, as the Project Lead is neither responsible for a project (rather for a Product team or Feature team), nor are they the 'Lead'. They are simply responsible for processes and methods in the team – the Agile Coach where necessary, the team assistant, the communicator, the good soul of the team. We could also say that this role is the 'centre of lateral leadership responsibility'.

We would be smart to rename these roles, perhaps like this:

• BD -> Business Lead
• TD -> Technical Lead
• PL -> People Lead

For us in the teams this no longer makes a difference, as every company has its own internal nomenclature. After a few years it's tough to get roles like this renamed anyhow – we might succeed in external communication at most.

No, this is not the path we're pursuing – some things you just have to leave behind! Cross-functional teams with product responsibility are large, and to match our teams' appetite we'd need XXL American pizzas and a team coming straight from the 'Elbe' (not the river but the OTTO 'Elbe' canteen).

However, our admittedly large teams do work in two groups: one group in the team fills the backlog pipeline, so to say, while the other group processes it. Smooth coordination is only possible because it is one team – but that still makes a team pretty big.

We've grown from 4 to well over 40 teams in around 10 years and yet we still work in the same team structure. You could call that a success story – and it is!

Nevertheless, we are slowly reaching the limits of growth. It's quite easy to imagine that overarching topics (business topics as well as technical ones) can be discussed and decided on quickly and unbureaucratically in a small group, while with 40 autonomous teams involved, discussing some questions takes a little longer. The transformation of otto.de into a marketplace also makes the situation trickier: there are now over 100 teams to coordinate, ranging from the Webshop to Logistics. Balancing alignment and autonomy is no small feat.