IT is not performing as well as it used to. That's the result from McKinsey's latest global survey of trends in IT:
The IT function in most organizations is under pressure like never before to justify its increasing budgets and reduced performance. The McKinsey report makes grim reading:
- IT is seen as less effective at enabling user goals
- Business people are less likely than 12 months ago to say that IT is performing effectively
- Even IT executives expect the reduction in performance to continue
Why is this? Information Technology has always suffered from a gap between expectations and delivery, but why is this trend becoming more pronounced? And what can be done to deal with it?
The juggernaut IT project
Many large organizations have at least one juggernaut IT project. Typically a huge, well-funded system integration exercise that's intended to make the right information available to the right people at the right time, and which will thereby speed up decision making. The teams are normally very large, possibly 100+ people, broken down hierarchically into smaller teams of 10, and the systems ordinarily depend upon large enterprise scale technologies. These days, the teams will often use an agile approach, recording requirements as user stories and delivering code in regular iterations. And yet, somehow, the project falls behind schedule and runs over budget. The promised features never quite make it into the promised releases. There are issues with source control and system integration. These juggernaut IT projects are one likely cause of much of the business disenchantment with IT. The projects are important. They have large numbers of staff. They are well-funded. They are behind schedule. If you're an IT director, how can you avoid your next project becoming a juggernaut? Here are some tips.
1. Start small
Don't be afraid to start small. Every large business problem probably has a small business problem at its heart. Create a small team to solve that problem in the simplest possible way. Treat the team like a small startup. Give them a small budget and the freedom to choose the simplest technology to solve the problem. Get the team to focus on solving the core problem and delivering quickly. It doesn't matter if the technology solution they deliver is simplistic. The very act of solving the core problem will provide invaluable knowledge, even if none of their code lasts longer than a few months.
2. Scale with more small teams
As the technology solution grows, break it down into smaller and smaller subsystems and then – importantly – break off smaller teams to work on the subsystems. Don't do this hierarchically. Make each team an autonomous unit with its own budget and the right to choose and change technology at will. This may feel very dangerous. What if different teams decide to use incompatible technologies? Consider using a microservice architecture. A microservice architecture has a couple of principles that will avoid many problems that arise from using disparate technologies:
- Create smart subsystems
- Use dumb pipes
What this means is that the different subsystems should communicate with each other using a single, simplistic technology. Given that the web is eating software you might consider using simple HTTP calls for your service APIs, which use some simple text-based format to transfer data, such as JSON. The HTTP/JSON connections are your dumb pipes. You want the API for each subsystem to be sufficiently simple that a tech-savvy teenager with a browser and a set of command line tools could interact with it. Then, once you have very simple APIs, you can bury the complexity within each subsystem. If a subsystem needs to implement queues, it can do so behind the API wall. Why do this? Why not just have an enterprise messaging system that integrates everything together? There are a couple of reasons:
- If you can't interact with a subsystem using text on the command line or via a browser, you won't be able to easily test it. Worse, your developers will never get a deep understanding of it.
- If you use complex software to connect subsystems, you will go through hell when you need to upgrade the middleware. Whereas, the time between HTTP upgrades is measured in decades.
3. Version your APIs, not your system
If you create a single version number for your entire system, the different subsystem teams will instantly need to coordinate their release cycles. If each subsystem team has their own versioning system at the API level, then subsystems can be upgraded and released separately. In the same way that Internet startups build systems that integrate with Twitter and Google and Facebook, you want your subsystem teams to integrate with each other via public APIs. Consider supporting three versions of the API:
- The deprecated version of the API to support teams currently unable to upgrade to the latest version
- The current version of the API
- The latest bleeding-edge version of the API, for teams that want to develop against the latest changes
Include the version numbers in the endpoint URLs.
4. Create your own Internet culture
Treat your IT function as a small collection of Internet startups. By starting small and growing organically, the software you produce is more likely to fit in with business needs. If one small team is working on a subsystem that is failing to deliver, you can close the team and use their budget elsewhere. Allowing each team to choose their own technology and control their own release cycles will reduce the coupling between subsystem components. A startup culture within your IT function will allow your organization to change and adapt to circumstances. The more it can adapt, the more it can fit in with the requirements of your business, the greater will be its value to your organization.