How to mandate agility in software development, operations, and data science
- 16 October, 2020 11:09
Even when leaders proclaim in their townhalls that your organization needs to be more agile and nimble, they can’t mandate it. Your CIO and IT leaders may standardize on practices, metrics, and responsibilities that they describe as agile methodology standards, but they can’t dictate that everyone adopts agile cultures and mindsets.
You can select agile tools, automate more with devops practices, and enable citizen data science programs, but you can’t force adoption and demand employee happiness. IT operations may operate a hybrid multicloud architecture, but that doesn’t necessarily mean that costs are optimized or that infrastructure can scale up and down auto-magically.
So, if you were looking to quickly standardize your agile processes, or to miraculously address technical debt by shifting to agile architectures, or to instantly transform into an agile way of working, then I am sorry to disappoint you. Agility doesn’t come free, cheap, or easily. You can’t manage it on a Gantt chart with fixed timelines.
And while I believe that agility is largely a bottom-up transformation, that doesn’t mean that developers, engineers, testers, scrum masters, and other IT team members can drive agility independently. The team must work collaboratively, acknowledge tradeoffs, and define agile operating principles where there is consensus on the benefits.
So if agility can’t be mandated and requires everyone’s contributions, how do organizations become more agile? In the spirit of agile methodologies, data-driven practices, and adopting a devops culture, here are some ways everyone in the IT organization can drive agility collaboratively.
Make the case for agile methodologies
Chapter 2 of my book, Driving Digital, is all about going from basic scrum practices to a more comprehensive agile planning process that includes assigning roles and responsibilities, planning multi-sprint backlogs, and standardizing estimating practices. When I work with teams trying to adopt agile mindsets and cultures, we establish release management disciplines, architectural standards, agile principles, and other guidelines for driving agility.
But this is not rolled out prescriptively. Different organizations have different business strategies, organizational structures, organizational cultures, talents, compliance requirements, and mixes of legacy and modernized architectures. These contexts are incredibly important when considering when and where to apply different agile practices.
For example, a large organization may have teams working on APIs for mobile applications that leaders want rapidly developed and released to employees. A second group may be working to transition a complex legacy system central to the operations of a regulated, audited, and global business.
Should these two groups of teams be following identical, prescriptive, and regimented agile practices? That certainly would inhibit the API team, which would undoubtedly prefer (and likely excel) if the form of agile adopted was more democratic and self-organizing, and left many decisions to the team. On the flip side, giving too much freedom to teams working on complex, business-critical legacy systems has greater risks.
The disparity in goals and constraints is one reason why organizations striving for agility must foster a culture of asking and answering “why” questions when defining agile principles. When leaders dictate the how without explaining the why, people are less likely to adopt the underlying practices. Explaining agile principles — especially the why — helps teams make better decisions on when, where, and how to apply agile practices.
Accelerate machine learning with dataops and data governance
I love Spiderman’s famous quote, “With great power, there must also come great responsibility.” Every organization wants its data scientists, data visualization wizards, and citizen data analysts to produce ongoing insights that aid in decision-making. But this power also requires data, analytics, and machine learning teams to adopt proactive data governance and dataops practices that address the organization’s data quality, security, privacy, master data management, and data integration requirements.
So, while analytics teams strive to be more agile, to deliver results frequently, and to increase the number of data sets used in analytics, data teams must strengthen the underlying data processing foundations based on compliance requirements and evolving business expectations.
That agility doesn’t come for free or through mandates. Data and analytics processes evolve when multi-disciplinary teams recognize the importance of agility and work collaboratively to improve analytics delivery and the data processing foundations. Here are some examples:
- A citizen data science program requires participating departments to define and maintain the data catalog and definitions before releasing new data visualizations.
- The data science team documents their machine learning models, defines drift parameters, and maintains the production models based on a defined lifecycle.
- Data integration and quality teams view analytics teams as customers or stakeholders. They regularly review the data wrangling performed by analytics teams, evaluating and adjusting the data models and integrations to reduce downstream data processing.
- All teams given the license to work with data regularly review changes in data security, compliance, and privacy requirements. They capture gaps as security, data, or technical debt and assign priorities to remediation work.
- Dataops and cloud operations teams proactively increase the level of monitoring, capacity planning, and infrastructure automation to meet the growing performance requirements of data processing and analytics teams.
Agility comes through collaboration and balancing the work desired with the work required. Otherwise, this new generation of big data, machine learning, and self-service BI programs will easily generate a new mountain of data debt, data silos, and data security risks.
Apply a customer mindset when maturing devops practices
Organizations adopting devops cultures and practices are striving to resolve a decades-long IT paradox: How do you empower agile teams to deliver small, frequent, low-risk changes to production that satisfy users and improve the business, without compromising reliability, security, performance, and other operating service levels?
Devops practices and tools address the gaps in IT change management processes that lead to major incidents, complex problems that require root cause analysis, gnarly infrastructure dependencies that delay deployments, and chronic security issues. Some examples of devops success:
- Organizations that use private and public clouds automate the deployment and teardown of environments with secure infrastructure as code.
- Agile development teams automate testing and streamline builds and deployments with shift-left CI/CD pipelines. The more advanced teams include security validations in their pipelines and embrace devsecops.
- IT operations improve their ability to manage complex serverless deployments, microservice architectures, and hybrid multicloud networks by increasing monitoring and deploying AIOps platforms to improve visibility and incident response.
These are all strategic ingredients to address IT’s agile and operational paradox, but diving headfirst into these programs without a strategy can lead to IT results without business value. Worse, it can sometimes cause IT to overinvest in automations at the expense of delivering on business priorities.
For example, let’s say you’re modernizing a legacy three-tier application while moving it to a public cloud, and you must decide what level of automation to implement. How should you define what’s good enough? And how should you define the criteria for success of devops-related improvements?
There are questions and parameters to aid in answering this question. Some might call them service level requirements. Others might describe them as non-functional requirements. In some cases, highly engaged stakeholders will demand daily releases and five nines of reliability. In other cases, the stakeholder involvement needed to define requirements will be harder to come by.
Either scenario poses challenges, but the common denominator required for agility starts by defining customers, customer personas, and success criteria. When you have overly prescriptive stakeholders, it’s important to separate the requirements they request from the requirements that make rational business sense. And when their needs are ill-defined, it’s especially important to document the criteria for success.
Many organizations define product management or business relationship management responsibilities to capture and share the targeted personas, success criteria, and business requirements. Bringing this customer mindset to devops teams and practices is a best practice that will help the organization determine which automations to invest in and to what degree.
In summary, agility can’t be mandated. Agility is achieved only through a collaboration between leaders and contributors. Agile teams must operate with self-organizing principles and standards. They must balance delivering improvements required by the business with the work required to address the data, operational, and technical debt. Setting priorities, defining success criteria, and determining what’s minimally viable require defining customer personas and understanding their needs and values.
When organizations adopt these types of practices, they won’t have to demand agility. Agility becomes a shared value and the standard approach to getting the job done.