Technology innovation needs research and agility – have you ever been experienced

In creating innovative technology, I’ve often been confronted with the difficulty of coupling research with an agile approach to a problem. This article is a personal analysis of why marrying the two is such a challenge followed by some suggestions on how to overcome them. I’d be interested in hearing about your own experience.

Agility is how you ‘slice the problem’.

A concept widely used in engineering and business, agility has led to the emergence of a number of formal methodologies (Larman & Basili, 2003) (Cohen, Lindvall, & Costa, 2004). These formalized frameworks were usually developed with a specific type of work setting in mind and have since been rigorously and erroneously applied in many different, inappropriate contexts.

Figure 1

In software engineering, the traditional approach is to structure the steps of a project according to the competencies or groups of expertise that are going to be involved at some point or other. This vertical slicing, illustrated in Figure 1, is convenient because it often reflects how the organization is structured e.g. in specialized groups. But this approach neglects the fact that any focus on creating value to the end-user requires a lot of coordination amongst the groups. It’s like taking a football player whose aim is to score a goal and think they’ll succeed by combining a number of gestures and movements of different body parts (right leg, left leg, head, eyes, right arm, left arm, torso…). We know that all these parts are required and that they need to be coordinated, but it’s actually extremely difficult to describe the individual contribution of each of them. A simpler approach is to cut the objective up into actions e.g. run, control the ball, look at target, shoot, which involve the coordination of the body parts. This horizontal slicing of a problem is how you achieve agility.

Agile projects are organized into iterations where each iteration ideally involves all the different kinds of expertise required in the team. At the end of each iteration the end-user, or someone who represents them, (often called ‘the product owner’), is presented with a partial, if immature, solution so they can provide feedback about how well it answers the problem. The shorter the iterations, the more agile you are.

Being able to work in such a way makes planning projects that involve lots of different expertise more convenient. It can also make it easier to work faster although, in my opinion, that’s just a good side effect. The perception of speed arises because the end-user can see the value of what’s being produced at every iteration. It really does make it easier to control progress, to know when expectations have been met and when the project is complete. It also greatly reduces the risk of disappointment or wasting time in redoing aspects at late and costly phases.

And although that’s already a fair list of benefits for adopting agility, there’s one more that, for me, beats the others. You get better results – and that’s because the iterations let you not only plan the implementation of the solution in an iterative way, but at each and every iteration you can refine the understanding of the problem you’re trying to address and the requirements for the solution. That’s particularly useful in projects where, due to their innovative nature, it’s difficult to start with requirements never mind clear ones.

Is the traditional research/innovation process agile?

The traditional ‘innovation through research’ process which maps the organizational structure, is thought as a sequence of competencies that work independently of each other. A research team typically proposes a model to solve a theoretical problem and produces a proof of concept.  An innovation engineering team then steps in to make it robust and, finally, a product team integrates it as part of a product for a specific use. Each team will work on average for 2 or 3 years before they consider that they’re done with what they set out to do. That means anything between 6 to 9 years may pass before getting any feedback on the model first proposed.

Some researchers may claim they’re agile because their research was done in small iterations and that’s fine, but who provides the feedback each time? If the researcher is ‘self-guided’, one could assume that they may have redefined the problem to best demonstrate the advantages of the model. If peer-reviewed, more objective characteristics of the model would have been looked at but someone from the same discipline is not necessarily the best person to represent the understanding of user problems nor define the characteristics that matter in agility.

In no way do I wish to diminish the importance of research in technology. My intention is to share my experience to help explain why ‘Innovation transfer’ from research to industry has always been perceived as a challenge (Myers & Rosenbloom, 1996) and why some well-established scientific communities are, and can be, totally disrupted by the technology of startups.

What’s the answer?

An agile approach to innovation through research practiced by highly innovative companies e.g. (Spector, Norvig, & Petrov, 2012), is to think about a project as an integrated team. The main distinction with a traditional research project team is characterized by the co-existence of the two distinct responsibilities of ‘What to do’ and ‘How to do it’. This boils down to having someone play the role of product owner.

If the objective is to incrementally improve an existing product, the product owner already exists. The difficulty here is to conciliate the potentially short term driven objectives of a product team with the eagerness of a research team to tackle more ambitious problems.

If, on the other hand, the objective is to think about a new, more disruptive offering, not only is there no product owner but there are no existing end users. What you can do in this case, is dedicate part of the team to studying the user and their problem (in our LABS it’s the UX and ethnography team) while other members study solutions to the technical challenges. It’s critical to work as an integrated team from the start of the project despite not having a common asset to build upon at each iteration nor a clear understanding of the need. To address this, you need to quickly build a very simple but usable baseline of the technology.

Whilst this approach can be very powerful it has encountered resistance from many technology researchers because it represents a significant cultural shift from historical values and practices of the scientific community and in particular the following three.

The first one is the thought process. In natural sciences, conducting fundamental research to explain the laws of the universe, is an instinctive prerequisite to applications. In technology, where we try to build solutions to problems, researchers often apply the same reasoning and take it for granted that fundamental research on theoretical problems should be done before specific applications. However, in our case this approach is not a given. I do not doubt the intrinsic value of fundamental research in technology (and there are many examples where this approach has worked) but what I do question is how it diverges from agility in the principle of user feedback.

We need to turn the model around by first finding solutions to very specific, real problems then generalize them over time. In other words, applications lead to more fundamental findings in the long run following the model of “Use-inspired basic research” defined in Pasteur’s quadrant (Stokes, 1997). It’s good to be aware of people in a team who are tuned to thinking­­ ‘abstract first’ because they may have difficulty in adapting to this reverse thought process and might need some help.

The second aspect is transparency. Scientific communities collaboratively improve the state of knowledge of their field through peer-reviewed publications. This approach guarantees the solidity of the content as it’s usually the result of a significant amount of research before distribution. In an integrated and agile project team, transparency and continuous sharing is key. In addition to short iterations, co-located teams and daily ‘standup’ short status meetings are examples of popular practices that support this. This level of transparency is accepted because the aim is not to judge individual contributions but rather for peers to share their mistakes and issues to reach a common objective. Sharing very immature work or thoughts can be difficult for someone who is used to submitting for review only when their work has been completed.

Merits of metrics. Getting a doctorate means developing a highly specialized profile because that’s what the diploma requires. Young researchers therefore naturally measure how good they are according to the depth of their expertise. Working in an integrated and multidisciplinary team means having as much breadth as depth so there needs to be a desire to continuously learn about new things and to work on issues outside your comfort zone.

These cultural differences may refrain some people from adopting the agile methodology but the barriers diminish after (more or less) practice and are completely overcome with the experience of the potential benefits.

Innovation through research and agility is a major, necessary evolution in today’s technology environment. It is challenging the community and established practices but it has also opened up the way to solving many of the issues related to technology transfer in the traditional model. More importantly it has enabled greater integration and dialogue between communities of expertise and that’s something we all need more of.

Frederic Roulland leads the geospatial data group and projects in mobility in Europe. He would like to acknowledge the help of colleagues Jos Rozen, Jean-Yves Vion-Dury and Antonietta Grasso for their advice and feedback on this article.

References

Cohen, D., Lindvall, M., & Costa, P. (2004). An introduction to agile methods. In M. Zelkowitz, Advances in Software Engineering. Advances in Computers (pp. 1–66). Academic Press.

Larman, C., & Basili, V. R. (2003). Iterative and Incremental Development: A Brief History. Computer, 36(6), 47–56.

Myers, M. B., & Rosenbloom, R. S. (1996). Rethinking the role of industrial research. In R. Rosenbloom, & W. J. Spencer, Engines of innovation: U.S. industrial research at the end of an era (pp. 209-228). Boston: Harvard Business School Press.

Spector, A., Norvig, P., & Petrov, S. (2012). Google’s Hybrid Approach to Research. Communications of the ACM, 55(7), pp. 34-37.

Stokes, D. E. (1997). Pasteur’s Quadrant – Basic Science and Technological Innovation. Brookings Institution Press.