Everyone running a service is doing operation

NoOps and DevOps has been a hot topic on twitter in the last couple of months and a couple of days ago @adrianco has published a blog entry saying that Netflix are doing NoOps for the part of the service they run on AWS. I think this discussion is important how will the operation evolve today and in the future and what will the role be and what skill will be needed. There will always be operation, I can access the site at www.netflix.com so the site seams to be operated 😉 The question is which approach to take and what is the need for operation and development in the future.

Clearly as @lusis point out in this reply, http://blog.lusis.org/blog/2012/03/20/it-sucks-to-be-right Netflix does operation or the site as well as the business will be dead.

One of the issue I think @adrianco got wrong is the way you do operation today. Operation in businesses utilizing virtual environments and working with continues delivery is different compared to what you do in operation running on your own physical hardware and with scheduled releases. Of course you still do operation but you do it on another level and partly you need other skills.

(Maybe there are some improvements that could be done on the operation of netflix.com, earlier today I tried to access their site and got this:

This part of the site could of course be the stuff running on the physical hardware and operated by another team, but anyway Netflix is one company with a common goal to deliver services to their customers.)

Operation – the move from hardware to programming

Operations is getting more and more of a development approach and not just developing scripts, instead we see the movement that operations become developing, maintaining and operate 🙂 a system of itself utilizing most of the fancy things used in developing of applications like databases, messaging, indexing technologies and GUI development. Also operation needs to take a greater part in handling other the infrastructure of  development like build systems, version management, test tools etc as those tools are becoming more and more important in an agile world.

This is driven by the movements toward virtualization (public or private clouds), automation and the agile movement in development. This is a driver for a major change for the role for operations and also the skills needed for the operation girl or guy to do a great job.

How to handle the challenges

The new way of working put a lot of challenges on the organization and there are several ways you can handle this. Either as, I think, how Netflix handled it by separating the operation duty into two different teams one team with the “old” operation staff handling everything running on hardware owned by Netflix and then a new team of developers that operate everything that runs on hardware owned by Amazon. I think this is what happen if you have a company culture of us and them with no common goals. The development team get frustrated by the old way of running operation on physical hardware and they take the company credit card and sign up for a service, deploy their applications and take the responsibility for operation of the system. I don’t think Netflix is the only organization where you see this happen and when we see more of virtualization and automation we will see this happen in many organizations that can’t get operation and development together.

Instead successful organizations will handle this in another way and that is by deeply integrating the operation teams with the development teams and create a culture that makes the teams to work smoothly together or even create mixed teams. This creates new way of working, one good example is the cultural change at Nokia Entertainment UK, presented at the DevOps conference in Göteborg, by inclusion going from 6 releases/year and 50 persons working with releases to 246 releases/year with only 4 persons, see http://www.slideshare.net/pswartout/devopsorg-how-we-are-including-almost-everyone. That story was impressive.

Why are there so many operations persons involved in the DevOps movement, I think this is natural, the tools used by operations are the one that encounter the most dramatic changes and the evolution needs to come from this change of the reality. What is really the driver is the opportunity for the business to reduce time to market and lower costs. By more and more development organizations working with agile development this put pressure on the operation organization to become more agile and work in a different way. This different way involves a lot of development and therefore I think we already see a movement that we have more of persons with developer background working with operation. At Recorded Future we are hiring developers for operations stuff as 70 % of our work today is to do programming of a fairly complex systems.

So one of the major challenges is to transfer the operation team into teams doing development of the operation system. Those working with operation and understand this change and make the operation team more of a development team will be those that will success. There is a need of both developing operations people into more of programmers and when hiring new people focus on the programming skills and skills to interact with other persons and teams.

DevOps is about culture

DevOps is about emphasize a culture of common goals and get things done together. Even if you have a great development team and a skilled and great operation team you will not have success until those works close together and to have this to happen you must work with the organization, the values and the culture. As many has written before you need to break the silos, not building higher silos.

There is a movement started by introduction of new technologies and driven by business requirements to be faster on the market and to reduce the cost. We see a lot of services delivered on different levels of XAAS but still we will have to cope with how we operate our service in the best way and this will be even if we don’t handle the hardware or even the operating system anymore. We have to improve our way to meet the business need and to be competitive and for me one of the most important tools is DevOps.

(And if you have read so far you deserve not only a cake but a beer 😉 )


3 comments so far

  1. adrian cockcroft on

    Hi Ulf. www2.netflix.com is indeed the old datacenter code base. That page comes up whenever there is an Oracle schema update or other maintenance. There were some datacenter software issues yesterday after an update. The billing systems and parts of the members account page use www2. The streaming service runs on movies.netflix.com which is on AWS and didn’t have an outage.

    One reason why Cloud DevOps and PaaS are driven by the Dev org at Netflix, is that three years ago a significant number of people in the Ops org didn’t want to do DevOps or Cloud. The Dev org took over and just did it anyway. Those people have since left or changed their mind….

  2. ulfmansson on

    I think this is one of the main reason why we see this DevOps movement, we are many that see this happen in many organisations, the malfunctioning organisation and failing culture that can’t get operation and development to work towards common goals. In those organisations some day development give up and take care of operation by them self and let the operation guys take care of the old stuff.

    This still doesn’t mean that you don’t have any operation it’s just that the developers take care of the operation so I think NoOps is not a valid term, your development guys wear the ops hat even if they don’t have the title.

    I think we saw the same thing happening when PCs was introduced during the 80s and 90s, people just went away and bought PC by themselves as the IT department only focuses on their terminals and mainframes.

    I think it’s up to operations to prove that they can handle this new way of working, for developers to trust operation and for management to create a great culture.

  3. The Mechanic on

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s