However, the report also found that code velocity has remained steady for the past two years, with about 15% of teams being considered top performers, meaning they have lead times of less than one day.

The CDF believes that while DevOps may be a help, it is likely the increasing complexity of projects that is slowing things down. 

Another finding in the report is that despite the increase in DevOps adoption, there hasn’t been an increase in the number of DevOps-related tools over the last year. The average number of tools sits at 4.5 currently. 

However, there is still a strong correlation between the number of tools in place and how likely a team is to be a top performer. These top performers were measured by three metrics: lead time for code changes, deployment frequency, and time to restore service.

The report also found that in general increasing CI/CD tools may increase performance, but interoperability concerns arise when multiple tools are used together. 

“We note that the proportion of top performers remains flat while that of low performers increases dramatically, with an increasing number of self-hosted CI/CD tools used. This suggests that there is a diminishing return from increasing the number of CI/CD tools a developer uses. The usage of an increasing number of tools may also be a response to increased complexity, which is having negative impacts on the performance of these developers. Similarly, the integration of multiple tools may not be optimally implemented, leading to function overlap that is impacting performance,” the report states. 

The report also shows a correlation between speed and stability metrics. 30% of the highest performers in code change lead time were also the highest performers when it came to service restoration. 

Interest in security is also clear from the survey, as testing applications for security measures was done by 37% of developers, rising up to the second most popular DevOps-related activity that teams engage in. 

“Developers who perform build-time security checks in an automated and continuous fashion are the most likely to be top performers, and the least likely to be low performers, across all

three metrics, of the types shown,” the report states. 

The report was conducted in partnership with SlashData, surveying over 125,000 respondents. It was released during the Linux Foundation’s Open Source Summit, happening this week in Vancouver, BC. At the event, the CDF also announced the addition of four new members: F5 NGINX, Prodvana, Salesforce, and Testkube.

The post Report: Adoption of DevOps practices increasing, while code velocity remains the same appeared first on SD Times.

This means that while organizations seek to increase software change velocity (as measured by the other DORA metrics in the report), a significant number of deployments result in degraded service (or service outage) in production and subsequently require remediation (including hotfix, rollback, fix forward, patch etc.). The frequent failures potentially impair revenue and customer experience, as well as incur significant costs to remediate. 

Most customers whom we speak to are unable to proactively predict the risk of a change going into production. In fact, the 2021 State of Testing in DevOps report also indicates that greater than 70% of organizations surveyed are not confident about the quality of their releases. A smaller, but still significant, proportion (15%) “Release and Pray” that their changes won’t degrade production. 

Reliability is a key product/service/system quality metric.  CFR is one of many reliability metrics. Other metrics include availability, latency, thruput, performance, scalability, mean time between failures, among others. While reliability engineering in software has been an established discipline, we clearly have a problem ensuring reliability.  

In order to ensure reliability for software systems, we need to establish practices that plan for, specify, engineer, measure and analyze reliability continuously along the DevOps life cycle. We call this “Continuous Reliability” (CR).  

Key Practices for Continuous Reliability 

Continuous Reliability derives from the principle of “Continuous Everything” in DevOps. The emergence (and adoption) of Site Reliability Engineering (SRE) principles has led to CR evolving to be a key practice in DevOps and Continuous Delivery. In CR, the focus is to take a continuous proactive approach at every step of the DevOps lifecycle to ensure that reliability goals will be met in production. 

This implies that we are able to understand and control the risks of changes (and deployments) before they make it to production. 

The key pillars of CR are shown in the figure below:

CR is not, however, the purview of site reliability engineers (SREs) alone. Like other DevOps practices, CR requires active collaboration among multiple personas such as SREs, product managers/owners, architects, developers, testers, release/deployment engineers and operations engineers. 

Some of the key practices for supporting CR (that are overlaid on top of the core SRE principles) are described below.

1)    Continuous Testing for Reliability

Continuous Testing (CT) is an established practice in Continuous Delivery. However, the use of CT for continuous reliability validation is less common. Specifically for validation of the key reliability metrics (such as availability, latency, throughput, performance, scalability), many organizations still use waterfall-style performance testing, where most of the testing is done in long duration tests before release. This not only slows down the deployment, but does an incomplete job of validation. 

Our recommended approach is to validate these reliability metrics progressively at every step of the CI/CD lifecycle. This is described in detail in my prior blog on Continuous Performance Testing. 

2)    Continuous Observability 

Observability is also an established practice in DevOps. However, most observability solutions (such as Business Services Reliability) focus on production data and events. 

What is needed for CR is to “shift-left” observability into all stages of the CI/CD lifecycle, so that reliability insights can be gleaned from pre-production data (in conjunction with production data). For example, it is possible to glean reliability insights from patterns of code changes (in source code management systems), test results and coverage, as well as performance monitoring by correlating such data with past failure/reliability history in production.   

Pre-production environments are more data rich than production environments (in terms of variety); however, most of the data is not correlated and mined for insights. Such observability requires us to set up “systems of intelligence” (SOI, see figure below) where we continuously collect and analyze pre-production data along the CI/CD lifecycle to generate a variety of reliability predictions as and when applications change (see next section). 

3)      Continuous Failure, Risk Insights and Prediction 

An observability system in pre-production allows us to continuously assess and monitor failure risk along the CI/CD lifecycle. This allows us to proactively assess (and even predict) the failure risk associated with changes. 

For example, we set up a simple SOI for an application (using Google Analytics) where I collected code change data (from the source code management system) as well as history of escaped defects (from past deployments to production). By correlating such data (gradient boosted tree algorithm), I was able to establish an understanding of what code change patterns resulted in higher levels of escaped defects. In this case, I found a significant correlation between code churn and defects leaked (see figure below).

We were then able to use the same analytics to predict how escaped defects would change based on code churn in my current deployment (see inset in the figure above). 

While this is a very simple example of reliability prediction using a limited data set, we can do continuous failure risk prediction by exploiting a broader set of data from pre-production, including testing and deployment data. 

For example, in my previous article on Continuous Performance Testing, I discussed various approaches for performance testing of component-based applications. Such testing generates a huge amount of data that is extremely difficult to process manually. An observability system can then be used to collect the data to establish baselines of component reliability and performance, and in turn used to generate insights in terms of how system reliability may be impacted by changes in individual application components (or other system components). 

4)    Continuous Feedback  

One of the key benefits of an observability system is to be able to provide quick and continuous feedback to the development/test/release/SRE teams on the risk associated with changes and provide helpful insights on how to address them. This would allow development teams to proactively address these risks before the changes are deployed to production. For example, development teams can be alerted as soon as performing a commit (or a pull request) of the failure risks associated with the changes they have made. Testers can get feedback on the tests that are the most important to run. Similarly, SREs can get early planning insights into the level of error budgets they need to plan for the next release cycle. 

Next up: Continuous Quality 

Reliability, however, is just one dimension of application/system quality. It does not, for example, fully address how we maximize customer experience that is influenced by other factors such as value to users, ease of use, and more. In order to get true value from DevOps and Continuous Delivery initiatives, we need to establish practices for predictively attaining quality – we call this “Continuous Quality.” I will discuss this in my next blog. 

The post Optimize continuous delivery with continuous reliability appeared first on SD Times.

Tracy Miranda, executive director of the Continuous Delivery Foundation said, “Our State of CD report focuses on developers, and the data certainly shows that DevOps is widespread in organizations of all sizes. However, despite a common perception that industry is accelerating software delivery adoption, for teams, the data shows no signs that velocity is increasing. Better onboarding materials, less complex systems and more will be needed to improve software delivery in 2022.”

The report found that while 44% of developers use either continuous integration or deployment (CD/CD), less than 18% employ both practices to fully automate their delivery processes. The report also found that continuous integration is currently the most widely used DevOps-related activity with 32% of developers surveyed reporting that their organizations use CI in practice. Furthermore, only 29% of respondents said that they are currently using continuous deployment as their main method of delivery automation. 

Additionally, it was found that DevOps is becoming mainstream not only for developers of internet-centric applications or complex distributed systems, but throughout all sectors of the software economy. This goes for organizations of vastly varying sizes. 80% of developers who are employed by companies with two or more workers reported that they are involved in DevOps practices. The report showed that the key difference between companies of differing sizes is the degree to which they have adopted these DevOps practices. 

The report found that nearly 60% of enterprise developers use CI/CD in their workflow. Comparatively, only 48% of developers working in small companies and 42% of freelancers said that CI/CD played a part in their development processes. 

There is still a way to go until the majority of organizations obtain a fully automated software delivery process. The study showed that as of Q3 2021, only 14% of developers surveyed took, on average, less than one day to go from code checked-in to having that code running successfully in production. In this same vein, only one in 10 developers reported that they deploy code into production multiple times per day, and the proportion of top performers in terms of lead time for code changes dropped from 17% in Q3 2020 to 14% in Q3 2021. 

This slowing of software delivery may be the result of the increased complexity and intricacy of modern-day software projects. In large-scale enterprises the report found that only 14% of developers showed lead times for changes of less than a day in Q3 2021. This is a decrease from last year where the study showed 18% performing at this level. This downward trend is steady among any size organization, with medium sized companies down from 16% to 14% and small organizations from 19% to 17%.

Over the past year, the amount of developers who are able to quickly restore an unpredicted, user-impacting incident also shows a deceleration. The proportion of those who need less than one hour to restore service dropped from 17% to 13%. However, for this metric, medium sized businesses showed the greatest drop in performance when compared to larger and smaller organizations. 

The study also revealed that the two strongest predictors of a high-quality software delivery performance are the areas that developers are involved in as well as their experience level. Developers with more than 16 years of experience behind them are almost twice as likely than beginners to be top performers in terms of lead time for code changes. With this, the likelihood of performing poorly showed a drop from 54% among those who are just starting out to 10% among seasoned developers.

As mentioned earlier, it is not just developers working on internet-centric applications that are taking advantage of DevOps practices, however, they are the ones with the highest concentration of top performers. Specifically, extensions of third-party ecosystems, back-end services, and web applications possess the highest number of top-performing developers for software delivery speed. 

To read the full report, visit here. 

The post Report: CI/CD still in early phase for much of software development community appeared first on SD Times.

Bitrise: is a cloud-based, secure, and scalable CI/CD solution, specifically designed for mobile app development. The company’s mission is to help their customers all around the world create better apps, faster. With customers ranging from centuries-old enterprises, to some of the youngest, most innovative new startups and mobile unicorns, applications built, tested, and deployed through Bitrise have made their way to billions of end users around the world.

HCL Software: Deploy anything, anywhere with HCL Launch, a versatile, enterprise-level continuous delivery solution. From mainframe to microservices and everything in between, HCL Launch is engineered to handle the most complex deployment situations with push-button automation and controlled auditing needed in regulated and technically diverse enterprises.

Atlassian: Bitbucket Pipelines is a modern cloud-based continuous delivery service that’s built right into Atlassian’s version control system, Bitbucket Cloud. Bitbucket Pipelines automates the code from test to production. Bamboo is Atlassian’s on-premises option with first-class support for the “delivery” aspect of Continuous Delivery, tying automated builds, tests and releases together in a single workflow. 

RELATED CONTENT:
Continuous delivery steps into the spotlight
The state of CI/CD
DevOps for mobile applications
How CI/CD providers help organizations deploy speedily, securely, and stably

CA Technologies, A Broadcom Company: CA Technologies’ solutions address the wide range of capabilities necessary to minimize friction in the pipeline to achieve business agility and compete in today’s marketplace. These solutions include everything from application life cycle management to release automation to continuous testing to application monitoring—and much more. 

Chef: Chef Automate, the leader in Continuous Automation, provides a platform that enables you to build, deploy and manage your infrastructure and applications collaboratively. Chef Automate works with Chef’s three open-source projects; Chef for infrastructure automation, Habitat for application automation, and Inspec for compliance automation, as well as associated tools. 

CircleCI: is the leading continuous integration and delivery platform for software innovation at scale. With intelligent automation and delivery tools, CircleCI is used by the world’s best engineering teams to radically reduce the time from idea to execution. The company has been recognized as an innovative leader in cloud-native continuous integration by independent research firms and industry awards like the DEVIES, Forbes’ Best Startup Employers of the Year, and Deloitte’s Technology Fast 500.

CloudBees: The CloudBees Suite builds on continuous integration and continuous delivery automation, adding a layer of governance, visibility and insights necessary to achieve optimum efficiency and control new risks. This automated software delivery system is becoming the most mission-critical business system in the modern enterprise.

Cloud Foundry: Cloud Foundry is promoted for continuous delivery as it supports the full application development lifecycle, from initial development through all testing stages to deployment. Cloud Foundry’s container-based architecture runs apps in any programming language over a variety of cloud service providers. This multi-cloud environment allows developers to use the cloud platform that suits specific application workloads and move those workloads as necessary within minutes with no changes to the application.

Digital.ai: The company’s Deploy product helps organizations automate and standardize complex, enterprise-scale application deployments to any environment — from mainframes and middleware to containers and the cloud. Speed up deployments with increased reliability. Enable self-service deployment while maintaining governance and control.

Liquibase: Liquibase, formerly Datical, brings Agile and DevOps to the database to radically improve and simplify the application release process. Liquibase solutions deliver the database release automation capabilities IT teams need to bring applications to market faster while eliminating the security vulnerabilities, costly errors and downtime often associated with today’s application release process.

GitLab: GitLab allows Product, Development, QA, Security, and Operations teams to work concurrently on the same project. GitLab’s built-in continuous integration and continuous deployment offerings enable developers to easily monitor the progress of tests and build pipelines, then deploy with the confidence across multiple environments — with minimal human interaction. 

IBM: UrbanCode Deploy accelerates delivery of software change to any platform – from containers on cloud to mainframe in data center. Manage build configurations and build infrastructures at scale. Release interdependent applications with pipelines of pipelines, plan release events, orchestrate simultaneous deployments of multiple applications. Improve DevOps performance with value stream analytics. Use as a stand-alone solution or integrate with other CI/CD tools such as Jenkins. 

JetBrains: TeamCity is a continuous integration and continuous delivery server that takes moments to set up, shows your build results on-the-fly, and works out of the box. TeamCity integrates with all major development frameworks, version control systems, issue trackers, IDEs, and cloud services.

Microsoft: Microsoft’s Azure DevOps Services solution is a suite of DevOps tools designed to help teams collaborate to deliver high-quality solutions faster. The solution features Azure Pipelines for CI/CD initiatives; Azure Boards for planning and tracking; Azure Artifacts for creating, hosting and sharing packages; Azure Repos for collaboration; and Azure Test Plans for testing and shipping.

Puppet: Puppet Pipelines provides developers with easy-to-use, self-service workflows to build containers, push them to any local or remote registries, build and deploy Helm charts, and deploy containers to Kubernetes in under 15 minutes, while providing governance and visibility into the entire software delivery pipeline and the status of every deployment. 

Octopus Deploy: Octopus Deploy is an automated release management tool for modern developers and DevOps teams. Features include the ability to promote releases between environments, repeatable and reliable deployments, ability to simplify the most complicated application deployments, an intuitive and easy-to-use dashboard, and first-class platform support.

Redgate Software: Including SQL Server databases in Continuous Integration and Continuous Delivery, and stopping them being the bottleneck in the process, is the mission at Redgate. Tools such as SQL Source Control, SQL Compare and SQL Change Automation integrate with Git or Team Foundation Server, Jenkins or TeamCity, Octopus Deploy or Bamboo, for example, allowing the database to be developed alongside the application.

VMware: With VMware Tanzu, you can make the most of these cloud native patterns, automate the delivery of containerized workloads, and proactively manage apps in production. It’s all about freeing developers to do their thing: build great apps. Enterprises that use Tanzu Advanced benefit from developer velocity, security from code to customer, and operator efficiency.

Weaveworks: Weaveworks is a container management and microservices solution provider that offers services for observing and monitoring solutions. Features of its Weave Cloud include ability to integration with preferred CI tools, the Prometheus service for gaining insights, troubleshooting and capabilities for understanding applications.

The post A guide to CI/CD tools appeared first on SD Times.

Because of this, it’s widely agreed upon that many if not most organizations have tried to embrace some DevOps thinking into their processes, but for many of them, it’s been kind of a slow go.

Tracy Miranda, executive director of the Continuous Delivery Foundation, explained that microservices and cloud-native technology have changed the game for creating and delivering software. “So even though we’d been delivering software for ages, all of a sudden, it was a different thing,” she said. “And you had to reconfigure your teams, adopt this new software, figure out these new best practices.” 

RELATED CONTENT:
Continuous delivery steps into the spotlight
DevOps for mobile applications
How CI/CD providers help organizations deploy speedily, securely, and stably
A guide to CI/CD tools

In the CDF’s recently released “State of CD” report, the organization found that only 10% of respondents are able to think about the new, advanced topics in today’s software delivery. The CDF also asked about using feature flags as part of progressive delivery and rolling back changes in a short time when things go wrong, and again, only 10% of the developers responding said they could do that effectively.  The report found that for nearly two-third of developers, it takes at least a week to move code from ‘committed’ to successfully running in production. 

Chaithanya Kolar, managing director in Deloitte Consulting LLP’s Quality Engineering practice, said it’s difficult to quantify adoption of these techniques, as some organizations do only a piece of DevOps but are happy with that. “There is not a single organization that hasn’t tried using Agile and DevOps practices in their IT development,” Kolar said. The 2021 Deloitte Quality Engineering Trends Report indicates that 73% of the organizations surveyed have adopted some form of DevOps. “As we all know,” he said, “CI/CD is a DevOps tactic. CI/CD emphasizes automation of software process while DevOps focuses on culture. Looking at CI/CD specifically, it is very hard to define the current single state of the adoption.” 

Kolar explained that one way to look at the current state of adoption is to break it down into two groups: high maturity organizations and low maturity organizations. “High maturity organizations have efficient development processes and standardized CI/CD frameworks across all value streams,” he said, adding that it is important to note that these organizations not only deploy applications into QA through CI/CD, but also deploy applications into production with minimal to no manual intervention. 

These organizations, he pointed out, also focus on automation across the entire life cycle and execute analytics-based tests, using customer feedback in regression and progression tests, for example.  On the other end of the spectrum, less mature organizations focus on implementing CI/CD in lower environments and/or across few value streams, he continued. “Releases in production still need release and change control processes that are traditional in nature,” Kolar said. “Automation is still being implemented in pockets and, most importantly, QA teams are not completely embedded into sprint teams.”  

As for the future of CI/CD? Kolar said, “A few years ago, CI/CD started off as a method to help continuous deployment, but that definition of CI/CD is long defunct. CI/CD now has QA and security elements to it. We may have seen people refer to the current trend as DevSecOps. In my mind, DevSecOps is changing to be Dev-Infra-Sec-Ops (infrastructure-as-a-service) and will soon be called “Dev-Infra-Sec-Analytics-Ops (including analytics-as-a-service). One day the trend of CI/CD will eventually lead to touchless software development and maintenance. We are on the brink of major efficiency shift in the industry now and AI/ML and LCNC [low code/no code] technologies are enabling this shift.”

The post The State of CI/CD appeared first on SD Times.

There is a problem, however. A recent “State of Continuous Delivery” report by the Continuous Delivery Foundation (CDF) found that adoption is low, and less than what was reported in earlier studies, according to Tracy Miranda, the foundation’s director.

Continuous delivery grew out of the DevOps concepts as a way to make changes quickly and get them out to customers, to take advantage of market changes or to outmaneuver competitors. But today, CD (as it is known) has grown to encompass testing, security and governance — basically, all the steps software always has gone through before it is released into the wild.

Taking a step back

As the twin notions of being able to make changes to software quickly and then deliver it to users safely and securely were fleshed out, the terms CI and CD were seen as a pipeline for software production, and were lumped together as CI/CD. But even early on, there was confusion over the meanings of ‘delivery’ and ‘deployment.’ According to Miranda, “For a long time, people saw them as connected. And now we’re starting to get really fine-grained. So you can deploy something into production, but you haven’t delivered it to customers.” 

RELATED CONTENT:
The state of CI/CD
DevOps for mobile applications
How CI/CD providers help organizations deploy speedily, securely, and stably
A guide to CI/CD tools

Viktor Benei, CTO and co-founder of mobile DevOps solution provider Bitrise, said continuous integration and delivery are intertwined. “I think it’s part of the same process and trend: automating as many of your processes as possible — anything that can be automated throughout the app development life cycle,” he said. “Reduce manual, error-prone steps via automation and save time for the things which can’t be automated.”

Yet, Miranda believes the term CI/CD is a bit of a misnomer, because it makes you think there are only two things you have to do to deliver software: integration and deployment. She said one of the first things that got discussed when the foundation was being created in 2019 was being able to agree on what is meant by those terms, so there would be a common language to build on. She said they now use the term continuous delivery to mean the set of practices teams need to use to deliver software safely, reliably and securely. 

“It’s pretty much in line with the “Accelerate” book Jez Humble and Nicole Forsgren (of the DevOps Research and Assessment team, or DORA) put together,” Miranda said. “We distinguish it from continuous deployment, which is that last stage of getting your code out into production and lots of people use continuous delivery and continuous deployment synonymously. But we prefer to use continuous delivery as the set of practices. So that includes continuous integration, continuous deployment, configuration management, security, and testing.”

Senthil Nathan, product manager, HCL Software DevOps, defined things this way: “I would say the answer is continuous delivery is getting to the end customer, and continuous deployment is more of automating the deployment process.”

CI, he said, can be viewed as a single-pipeline build/unit test environment. “Let’s say checking into master is your trigger. The moment that trigger happens, it goes through different things, but it all happens in, say, a span of minutes.” Organizations have done better with CI than CD, he opined, “because to an extent, that is just live work. Developers pick it up, and they do it.”

On the continuous deployment side, more than developers are involved — you have testers, security people, audit and governance people — and that makes the process more complex. And Nathan said automation is key to replacing manual and scripted deployments, to smooth out and speed up the process.

A manual approach requires changing scripts when moving from, say, a development environment to a staging environment to the production environment. In those non-automated organizations, Nathan explained, “they have a script and they run it out and that does the deployment. But for every new environment that comes up, they change the script to adapt to that environment. So from there, we would like to move them to somewhere where they have a single process that they use to deploy in the dev environment to your production environment. And anything that is different from that environment is more of a configuration and not an actual script change for you to do that.”

Nathan suggests that the first step companies need to take for continuous delivery is to write down their deployment process, which he said is more of a flow chart. “You drag and drop things and then configure it and say, ‘Okay, this is what I want to do as deployment.’ He went on to say organizations should test their deployment processes across environments before the code reaches production. “Then,” he said, “that’s the same thing that you use across all your environments and, if possible, template it and use it across teams, so that each team doesn’t have to reinvent the wheel.”

Still, adoption lags

Part of the reason that CD adoption lags is that many organizations believe what they’re doing today works for them. If scripted deployments work for them, they won’t feel an urgency to change. “It’s more of a denial mode at this point,” Nathan said. “Many of them really think that what they have today is good enough, and they don’t have to pick something new.”

Another piece to the adoption lag is that operations teams have trouble giving up control to the CD tools to do deployments, Nathan added. Where CI is mostly developer focused, CD involves more people, and thus requires more people to agree to doing it. “You need the test teams to come back here and say, I will do test. Like, once the deployment is over, the deployment tool will trigger the testing and all that you need, you need your operations teams to come and agree to whatever we are doing up here. And then you start talking about controlled environments like production and pre-production and things like that, then people say, like, ‘No, this works worked for us for decades, why change that.’ So that is the challenge.”

From continuous delivery emerges progressive delivery

Newcomers to continuous delivery have been confused by the idea, conflating it with continuous deployment, and that has led to people working in the space talking at cross-purposes.

The CDF’s Miranda said, “I think that’s a new evolution, where for a long time, people saw them as connected. And now we’re starting to get really fine-grained. So you can deploy something into production, but you haven’t delivered it to customers. And the term progressive delivery, I think, captures that movement to disconnect deployment, from something that goes to the end user.” 

This decoupling of deployment and delivery is largely due to the additional services required to rapidly get new software into the hands of customers. And that has led Miranda to characterize software delivery as a “team sport.”  

Delivery, she said, cannot just be left to operations people. Miranda said: “Developers need to have the awareness. Ops people need to be involved. And then you need these specialized skills, you need to bring in your security folks, you need to, if you’re in a highly regulated industry, bring in your compliance folks. The only way you’re going to get good understanding and fast feedback is if they understand what you’re trying to do. And so our approach now is saying, ‘How can we bring not just developers and ops people together, but everybody?’ And I think that will include support folks, and I think that will include business decision-makers, when, with feature flags, they can decide when to turn on features. And that becomes decoupled from the IT infrastructure, so they become part of the conversation in a more powerful way.”

The post Continuous delivery steps into the spotlight appeared first on SD Times.

Viktor Benei is CTO and co-founder of Bitrise, a company that offers solutions for mobile DevOps, and when asked if there were differences developing apps for mobile devices as opposed to on-premises servers or the cloud, he responded: “Lots of things — some big, some small.”

“You can’t fully automate the continuous deployment process for mobile, as releases have to go through the app stores,” he said. This means organizations should still optimize how frequently they deploy their applications. As Benei noted, “Users also expect frequent updates, but not too frequent, so around once per week.”

He went on to say that testing and monitoring also differ. For testing, for example, you have to run emulators for UI testing. You also inherently work with a “monolith” — you can’t just release parts of the app separately like you can with microservices.

RELATED CONTENT:
The state of CI/CD
Continuous delivery steps into the spotlight
How CI/CD providers help organizations deploy speedily, securely, and stably
A guide to CI/CD tools

The app stores themselves inhibit continuous deployment, Benei said. “You can employ some tricks like releasing only to certain countries, doing something like a canary release, but you can’t do full continuous deployment.”

Benei suggested still having iterations ready for deployment frequently, “but things like feature flagging are even more important for code integration from multiple teams and so you can release every week or so,” he said. “You also have to double down on automated testing, running tests on a wide variety of virtual and physical devices.”

Benei offered that automation is a key best practice for mobile DevOps. “Automate as much as possible — reduce manual steps and processes. Focus heavily on testing, as you can’t do rollbacks and you can’t just push a fix as quickly as you can for a web service. Also, benchmark your app as much as possible — for example, benchmark the size and the startup time of your app as you can lose a lot of customers if your app is too large or loads slowly.

The post DevOps for mobile applications appeared first on SD Times.

Modern dev teams strive for Continuous Delivery because the quicker a team can get the code out, the faster it can shorten the feedback cycle. The faster a team member is given quality feedback, the faster they can iterate to improve the product being delivered. This is what high-performing engineering teams strive to achieve and when possible, to incorporate automation into their continuous improvement systems.

Recently, I was joined by some of the most prolific new names in software engineering – Charity Majors, the CTO and founder of honeycomb.io; Dana Lawson, the VP of Engineering at GitHub; and Kathryn Koehler, the director of Productivity Engineering at Netflix — for a discussion on continuous improvement. You can watch the entire discussion here: https://youtu.be/5sAJ3N6KNdQ.  

During our discussion, we talked about creating continuous improvement in engineering organizations. We dove deep into feedback loops, the paved path methodology, and more. As an engineering leader, I heard a few key takeaways where all engineering teams can benefit:

The power of a 15-minute feedback loop

Short feedback loops are often the cornerstone that many great engineering teams are built on. While the goal of a 15-minute feedback loop may seem excessive, it’s important to remember that continuous improvement relies on ownership cycles and shipping code quickly and efficiently. 

With a 15-minute feedback loop, the idea and context of code is fresh in an engineer’s mind when a problem arises, making it easier to identify the problem and find a solution. Without returning to a problem quickly, you run the risk of losing ownership within the release cycle. As you move through the release cycle, you start batching up other’s changes and shipping them together, resulting in the loss of an ownership cycle. 

“If you have that rapid feedback loop, you have that original intent that’s fresh in your head, you know exactly what you’re trying to do, and why, and all the trade-offs you had to make. You can never get that back again. It starts to decay immediately as soon as you’ve switched your focus away from it,” said Majors. “If you know your code is going out within 15 minutes or less, you’re going to go look at it. If you know that your code and half a dozen to a dozen others might go out at some point in the next hour, day, or week, you’re not going to go look at it, right? The longer the feedback loop gets, the more stuff snowballs. The longer the interval, the more time you spend waiting on people.”

The paved path methodology

Similar to the desired outcomes of a 15-minute feedback loop, the paved path methodology can be the golden path to fast feedback. With the paved path methodology, engineers’ jobs and roles are streamlined to their subject domain expertise.

Koehler reiterated this when discussing engineering teams at Netflix: “Our whole goal with the paved path is to maximize developer efficiency. Can we build out curated components? Can we build out end-to-end experiences? Can we get people onto this superhighway that lets them move away from all of the different Netflix systems and everything that’s under the hood?”

As Koehler said, the paved path methodology is all about formulating a workflow enabling people to focus on their domain. Through this methodology, you rely on the experts and trust that each engineer takes ownership over their given domain. 

“We have UI developers. We have back-end service developers. We want them to focus on what they know. We want to simplify,” said Koehler. “Enabling engineers to focus on what they do best not only ensures those who know how to get the job done are doing the work, but delights your engineers along the way. They get to do what they love and do it well.” 

Learn from what other engineering teams are doing

Successful continuous improvement often stems from learning from others. However, it’s important to find a workflow that works best for your engineering teams. Sometimes a 15-minute feedback loop is possible but oftentimes it isn’t. “I dream of the 15-minute cycle, if that tells you anything. The reality of it is we have a huge, major scale, and we are a polyglot environment,” said Lawson of her engineering teams at GitHub. 

What’s important is that you find a deployment and feedback cycle that works best for your teams, while enabling each engineer to own their work. 

“You have to find what works for your systems and continuously improve it like you continuously deploy. You need to meet people where they’re at. You need to observe and look back and say, ‘Okay, what friction is really happening there?’ And then assess the risk of change.”

Long feedback loops can ruin everything

The truth of the matter is that long feedback loops can ruin someone’s day, even their lives. “Engineering should be a creative, fulfilling activity. If you’re on a high-performing team, you spend most of your time solving new and interesting puzzles that move the business forward every day,” said Majors. “The longer that feedback loop is, the more you can’t live up to that, and it just becomes a bunch of toil.”

One thing that these engineering leaders and I can all agree on is that engineers don’t get burnt out by shipping too much, they get burnt out by shipping too little. Creating systems that enable fast, quality feedback not only improves the lives of engineering teams but has them own the process and get excited about improving the quality of their work along the way.

The post Feedback loops are a prerequisite for Continuous Improvement appeared first on SD Times.

Progressive delivery uses experimentation techniques such as feature flags, blue-green rollouts and canary releases to show new features or bug fixes to a small cohort of users, and takes feedback from those experiments to make a decision to go big with it or roll it back to its original state for more work. These experiments enable organizations to decouple deployment from release.

In a recent conversation I had with Dave Karow, evangelist at feature flag platform provider Split Software, he discussed something he called layered progressive delivery.

This approach, he explained, begins with finding consensus with developers and SREs. “There’s nobody that’s not going to want better cycle time, shorter cycles. There’s nobody that’s not going to want automating the ability to detect when things go awry that you didn’t expect,” he said. “There’s probably — hopefully — not too many people that aren’t going to want to know whether the thing they just did had an effect.” 

He went on to say that this new approach to progressive delivery builds layer upon layer of richness to get more out of the experiments, and strongly debunked the notion that experimentation is both hardcode rigorous and that it requires building two versions of the code.

Savvy experimenters, Karow said, do dynamic config, which he explained allows development teams to send data along with a flag that sets different parameters for different users. He said the parameters of a recommendation engine, for example, “could dictate, do I want to give David a lot of answers, or just a handful of answers? And if you’re deciding whether you’re going to expose people to this new thing, you could also create two or three cohorts that each have different parameters. Now you’ve got people on your legacy engine, and you’re got two or three cohorts in the new one, and you’re trying different things — like lots of answers, not very many answers, ranked by popularity versus ranked by relevance.” The key point he made is that you can change the value in the flags and what those parameters are without having to create new versions of the code. 

“So now David is in cohort three that gets this, but we’ve just changed that he’s going to see results ranked by popularity instead of ranked by relevance in the engine. And we’re going to run that for a week and see what happens. That’s not three copies of code.”

When Karow talks about a layered approach, it simply describes a way to implement progressive delivery in progressively more value-rich ways, starting with the one that’s least threatening and not a point of debate with a developer.

A hidden benefit of using a feature flag platform to deliver the variations is that it also is capturing telemetry from each of those cohorts separately and processing the data separately, to quickly compare how each cohort behaves.

Karow gave an example from LinkedIn, which he said has been doing experimentation for a long time. They had an experiment on which version of an application would cause people to do more job listings. The developers didn’t monitor the application for speed, but got an alert from the platform that said the changes made the application slower. Automating guardrails, such as always monitoring for speed, can provide insights you might not have expected.  “When the thing that’s rolling it out also is the thing that’s keeping track of how it’s going, it becomes really easy to know what’s happening,” he said.

The next layer is measuring release impact. “If you achieve shorter lead times, and you’re shipping a lot, you might be like a hamster on a wheel, like you’re in a feature factory, and it sucks,” Karow said. “It’s demotivating. But if you have direct evidence of your efforts, it leads to pride of ownership.”

The top layer is test to learn, an area Karow said can help organizations take bigger risks but in a safe way. He gave the example of a food delivery service that wanted to ask customers questions about their eating and shopping habits to fine-tune their service, but didn’t want to ask too many questions for fear of turning off their users. So, he said, they did a status quo, a modest release, and a “go for it” release — which also increased onboarding time by two or three minutes. And right away, he said, they saw more money from every customer.

So instead of the usual pre-release hand-wringing — Do it. Don’t do it. We’ll lose everything. We’ll miss our quarter. — they tried these changes out in a safe way that gave them hard data from real customers.

The post Layered progressive delivery appeared first on SD Times.

Continuous delivery is a key part of how organizations deliver value to their customers, providing the ability to make small changes to software reliably and at any time, according to the report. 

The study, completed by research firm SlashData from data collected from more than 19,000 developers, asked for information on lead time for changes, deployment frequency, time to restore services, and change fail rates. 

Two-thirds of respondents indicated it takes at least a week for code to go from committed to successfully run in production. Only about 6 percent said they can move code from commit to production in under an hour.

The programming languages that rank highest for speed and stability in software delivery performance are the shell scripting languages (Bash and PowerShell), with Go/Golang and JavaScript being the next best choices. Interestingly, Ruby, Python and Java — three of the most popular languages based on Tiobe rankings, were poorest in terms of software delivery performance.

“Having access to new relevant data is critical to assessing and making the right decisions. By building this State of CD Report, we have produced an exclusive dashboard for use by our community that shows the current state of continuous delivery for deployment frequency, lead time for changes, time to restore service, and more,” said Tracy Miranda, Continuous Delivery Foundation executive director. “Moving forward, we will publish this report annually in order to track the world’s capacity to deliver software with speed and security.”

The post CDF report: Developers slow to adopt continuous delivery appeared first on SD Times.