“Historically, you’d deploy a new version of software and everybody would see it. For a lot of reasons, this ends up being less than optimal,” said John Kodumal, CTO and co-founder of LaunchDarkly, a feature flag management company. “One of the biggest reasons is risk because if there’s a problem, error or suboptimal behavior, that’s going to be exposed to your entire user base.”

What Is Progressive Delivery?

Progressive Delivery refines the release step, separating it from deployment so the potential damage caused by a bug or poor application design can be limited to a much smaller user base than the entire population. What’s more, teams can decide just how granular they want the level of control to be. 

Regardless of how granular they choose to get, the concept is the same: start small and then roll out to progressively larger cohorts. That way, the impact of a change can be assessed before it becomes a liability.

RELATED CONTENT: Progressive delivery: Testing software through limited releases

“The two core ideas driving Progressive Delivery are release progression and delegation,” said Kodumal. “Release progression allows you to adjust the number of users and what they are able to see. Delegation progressively delegates the control of a feature to the owner who is most responsible for an outcome.”

For example:

• Developers can decouple deploys from releases and test in production
• Data scientists can experiment and run A/B tests
• Sales and customer success can ensure appropriate entitlement and manage plans
• Operations can invoke kill switches and safety valves
• Security can ensure role-based access control and compliance
• Product owners can do dark launches and beta testing
• Marketing can synchronize launches and target markets

There are two ways to affect Progressive Delivery: using feature flags or canary deployments.

“With feature flags, you can observe a change over a longer period of time and compartmentalize the change. Feature flags also provide the smallest level of granularity such as an individual commit, an individual developer’s work or the most engaged customers,” said Kodumal. “Canary releases are really just the aggregation of all the changes from the last deployment to the new deployment, but with either mechanism, you can do something as simple as a percentage rollout.”

Feature management is essential to achieve Progressive Delivery because canary releases alone can be too constraining. It’s almost impossible to do several canary releases simultaneously, but with feature management, changes can be segregated at any level of granularity, such as squad-based, team member-based, or commit-based, for example. The isolated changes can then be measured independently.

“Feature management essentially gives you the benefit of a canary process, but on a per-change basis as opposed to a per-deploy basis,” said Kodumal. “It gives you more ability to parallelize more availability in a bigger team and pinpoint what changes connect to a positive or negative impact.”

With a canary release, it also may not be possible to identify the cause of a negative impact. However, if the change is guarded by a feature flag, then the feature flag can simply be turned off with the rest of the deployment kept live. Organizations with hundreds of engineers working in parallel find the level of precision greatly beneficial because they can control releases, very precisely, without adding an unwieldy level of complexity that’s hard to manage. Product quality also tends to improve.

Continuous Delivery and Progressive Delivery Go Hand in Hand

Many organizations have adopted CI/CD for competitiveness reasons. Typically, their industry has been disrupted by cloud-native companies that deliver software orders of magnitude faster. In fact, elite companies ship software 106 times faster and their applications fail seven times less often than slower-moving companies. They’re also doing 208 times more code deployments and can recover from incidents 2,604 times faster. 

While continuous deployment remains out of reach for most organizations, they’re still able to hone their DevOps practices and CI/CD pipelines. Progressive Delivery is simply the next step.

In fact, Continuous Delivery and Progressive Delivery are not mutually exclusive. Continuous Delivery accelerates release frequency and it can help improve product quality. However, in the absence of Progressive Delivery, Continuous Delivery can’t guarantee that the most recent updates will resonate with customers.

“Most people think of Continuous Delivery as the steps up to deployment. Progressive Delivery expands the life cycle by allowing you to minimize changes and react quickly to change beyond the deployment phase and into the runtime phase,” said Kodumal. “Continuous Delivery allows you to achieve a faster cycle time to production by reducing the size of changes that are pushed to production. Progressive Delivery is a dynamic version of that.”

In fact, the organizations that are in the best position to take advantage of Progressive Delivery are those that are already doing Continuous Delivery, but it isn’t an absolute requirement. For example, if an organization releases software every quarter, they can minimize the risk associated with code changes if the changes are protected by feature flags.

“Regardless of where teams are in their journey, they all tend to want the same thing,” said Kodumal. “They want to deploy when they want and release when their customers are ready.”

Adding Progressive Delivery to Continuous Delivery enables teams to push a change to production quickly and measure the impacts of that change right away. The rapid feedback cycles allow teams to have more confidence in the changes they’re making.

Benefits of Progressive Delivery

A surprising benefit of Progressive Delivery is that it enables teams who have not adopted CI/CD yet to adopt it safely. Progressive Delivery also enables DevOps because it provides collaborative capabilities and helps increase release velocity without interfering with system reliability.

“When software teams adopt DevOps and CI/CD, they often see the speed benefit, but they’re less confident about code quality, user perceptions and whether they’ve done enough to minimize the risk of security vulnerabilities,” said Kodumal. “Progressive Delivery enables you deploy new code faster and with an unprecedented level of confidence because it gives you the safeguards you need to minimize risk potential.”

Risk is the main reason why highly regulated companies have been slow to adopt even Agile practices, let alone DevOps or CI/CD. They want assurances that if their release velocity increases, the price of that speed won’t be incidents, outages, lawsuits or regulatory fines. Using Progressive Delivery, they’re able to limit risk exposure in a manner that’s auditable while delivering value faster to customers. 

The operative word when it comes to Progressive Delivery is control. Using feature flags, organizations can control:

• Which capabilities groups or even individual users can access and experience
• When those changes are delivered to different customers
• Who can release features
• The impact of less-than-optimal code

In fact, Microsoft reduces its launch risks using ring deployments, which is a Progressive Delivery technique. Specifically, it starts with a core group (which is actually a canary deployment). If the deployment meets the target performance criteria, then it moves to the next ring, which involves more users, and so on.

GitHub starts with “staff ships,” which are also canary deployments. That way if a deployment misses the mark, it can be fixed before any customers see it. If the deployment is successful, then the rollout continues among customers.

Software teams deploying microservices applications to Kubernetes clusters can take advantage of Progressive Delivery using service meshes or feature management. If using a service mesh, the process starts with a blue-green deployment in which an old version of the software and a new version of the software run on separate servers. The service mesh runs a canary test by routing a subset of users to the new application and the rest of the users to the old application. If the canary test fails, then all traffic is routed to the old version. Feature management provides more control.

The Best Way to Adopt Progressive Delivery

Feature flagging software is readily available, but as with any tool, the results depend on a combination of people, processes and tools.

“If you’re in the planning phase of a Progressive Delivery practice, make sure it’s a collaborative exercise with the team,” said Kodumal. “Your PM, your designer, your engineering – collectively the trio that is in charge of delivering a new change or a new piece of functionality – should talk about how to expose that feature to users.”

For example, they might decide to roll a change out gradually over a 10-day period in 5% increments on a daily basis. If any of the metrics indicate a negative impact, then the change could simply be rolled back.

Alternatively, if the situation involves a rebrand or other initiative involving non-software assets such as documentation or a marketing site, they might choose a rollout strategy.

“You want to bring culture and process change thinking into your planning phase so you’re clear about why you’re releasing a change and what would be considered a positive impact. You also want to consider the metrics so you can understand whether the release is working the way you expect it to,” said Kodumal. “A feature management tool enables you to do all that quickly and efficiently.”

As with many things, teams can decide whether to use open-source tools, build their own tool or license a tool from a company such as LaunchDarkly. The teams using open source or homegrown tools tend to discover they need enterprise capabilities such as a collaboration layer, security and permissions.

“Ultimately, Progressive Delivery software controls how users experience your product, what features they’re seeing and what features they’re not seeing. It’s a mission critical piece of your stack,” said Kodumal. 

Transform Your Application Portfolio

Companies that have become adept at Progressive Delivery aren’t just using feature flags to minimize risks. They’re also using it as a means of controlling software over the long term. That way, they can have one codebase that delivers multiple product experiences.

For example, companies offering different subscription or on-premises product levels might have gold, silver, and bronze plans, each of which offers different functionality or capabilities that can easily be controlled using feature flags. Without feature flags, there may be multiple code branches or arcane controls that determine what an individual user sees.

“Customers are now running longer-term experiments, doing A/B testing, optimization and personalization. It’s amazing how many capabilities you can unlock once you have a good feature management practice in place,” said Kodumal. “Once you begin to practice feature management, it fundamentally transforms the way you build software for the better.”

Learn more at www.launchdarkly.com.

Content provided by SD Times and LaunchDarkly

The post Introducing Progressive Delivery appeared first on SD Times.

The term progressive delivery was created in mid-2018 by Adam Zimman, the VP of Platform at LaunchDarkly, and James Governor, analyst and cofounder at RedMonk, to expand on continuous delivery’s notion of separating deployments and releases for organizations. 

Organizations that adopted continuous delivery early on were primarily software-first  organizations and their main delivery of value was through some sort of software package. Companies that didn’t have software as their only source of value faced challenges that weren’t really addressed by continuous delivery. 

RELATED CONTENT: 
Industry Watch: What follows CD? Progressive delivery
Learning about your software progressively

“When you start talking to the business, continuous deployment and continuous delivery tend to sound a little bit scary. If you talk to the business and say, look, we aren’t going to decouple these things. You decide when the business activation happens and you can do that because something is very well tested and you can test in production, you could be confident about when the services are rolled out and this will de-risk what you’re doing, then it sounds like they’re back in control,” Governor said. 

All of the core testing concepts of progressive delivery existed in continuous delivery. Now, it’s a matter of what’s actually getting the focus since there are a lot more things organizations can do while utilizing the cloud. 

Progressive delivery is a term that can be applied to a set of disciplines that people are already using now, whether that’s delivery and production excellence or organizations that are effectively testing and have a high level of confidence in their operations with a culture of troubleshooting and observability. 

“If you look at Google, Amazon, and Microsoft from a public cloud perspective, they are all doing stuff like this even though they don’t always call it progressive delivery,” Governor said. “Once you start getting into banks and telcos, then it’s becoming a more generally applicable set of approaches and technologies.” 

Progressive delivery really boils down to two core tenets: release progression and delegation, according to Zimman.

Release progression is all about adjusting the number of users that are able to see or interact with new features and new code at a pace that is appropriate for one’s business. It’s also about expanding it out only to the appropriate parties at any given time as part of the testing. That could mean only offering the feature to early access beta users first and then expanding it out to a trusted user group before expanding it out to everyone. Or maybe, the end state is to only give access to the people who are on the premium plan. 

“The thing that [continuous delivery] stopped short of was it was more of a binary mentality,” Zimman said. “So it was either on or off for everyone, as opposed to this notion that we’re really focused on this ability for increasing your blast radius.” 

Practicing release progression helps with the testing aspect of software delivery because the individual or team that built a new feature or a new widget can choose to deploy it and be the only ones that can interact with it. 

“Everybody is testing in production. Some people do it on purpose, but if you’re not testing in production on purpose, chances are that you are going to be burned by a bad release or a lack of consistency between your test environment and your production environment.”

The other core aspect, release delegation, focuses on shifting release control from the engineering and operations organization out to the business owner.

“As soon as you move out of the realm of pure software organizations, in which their only value is through their software, you start recognizing that the business owners are actually looking for greater control and greater ability to impart change on digital experiences,” Zimman said. 

Business owners can then customize what features they want to release to certain customers and even give the end users the ability to toggle certain features on and off, all while having guardrails put in place to make sure that the releases meet an industry’s compliance requirements. 

A lot of companies are looking to do that autonomously and not have to go back to the engineering or operations team for the ability to control features, especially when it comes to things like beta testing, A/B testing or experimentation, according to Zimman. 

Ravi Lachhman, an evangelist at Harness, said that progressive delivery comes from getting feedback and this is especially important in the software development model of today where a lot of the time you’re doing the unknown and you don’t know what the impact is going to be. One of the quintessential firms that has relied on feedback for progressive delivery is Facebook. 

“If you take it back 10 years ago, and you and I were downloading Facebook from the App Store, you and I would have two different download sizes and there’d be a reason for that. They’d be shipping different features for you and I,” Lachhman said. “For example, I really like fried chicken and I’m on several fried chicken groups on Facebook.They might say, you know what, target him with cook-specific things and so how they started doing it was with the concept of progressive delivery. We’re not going to give all the users the same thing, and we want to be able to make sure that we can retract those features if they’re not performing well, or we can roll those features out if they are doing well and determine how we provide feedback and how we choose to deploy across our entire user base or our entire infrastructure.”

One common way that organizations are going about progressive delivery is by using feature flags. Feature flags give users fine-grained control over their deployments and remove the need to change config files, do blue-green deployments and perform rollbacks.

A new functionality would be wrapped up in a feature flag and then deployed to a new version of the application to a single production environment, allowing only users from the designated canary group to access the new functionality.

However, having too many feature flags at once can lead to sprawl and a difficulty in keeping track of what feature flags are out there. This prompted a demand for feature flag management solutions, which serve as a central spot for the management of the flags with a common API that tracks the whole feature flag life cycle — for example, what was the logic? How do you turn it on? How do you turn it off? Where did it go? 

Progressive delivery is maturing 

Progressive delivery is starting to become a more mature practice as vendors are coming and coalescing around it. 

Governor said that this is the stage when it gets interesting because if you have a set of practices and then package them as a platform, it becomes something that a broader set of constituents can use. 

In addition to new tooling, it’s also about shifting the delivery side of the equation mostly from the context of engineering readiness to business readiness. 

“We don’t want to make any changes whatsoever to the deployment side of that equation because we want engineers to continue to develop at the pace of innovation, however fast they are comfortable with creating new technologies, features and code. They should continue to have that flexibility to do that creation and deployment into a production environment so that it is something where they’re able to test,” Zimman said. 

Now, the release side of the equation is really the delivery of value, Zimman noted. In the context of engineering readiness, something is released when it’s ready. On the other hand, business readiness puts the business in charge of when and how to release new feature functionality or release when customers are actually ready to adopt this new feature functionality.

This might be great for a company running a deal-a-day site because their value is changing on a daily cadence, Zimman said. 

Getting started with progressive delivery really requires getting all aspects of the business on board. 

One has to talk to product management about experimentation with progressive delivery, talk to the business about delegating the service activation to the business and having delegated users, and then talk to software developers and say that this technology won’t slow them down and will just enable you to move more rapidly and with higher quality, Governor explained. 

“The question that I like to ask enterprises is are you comfortable shipping code on a Friday afternoon?,” Governor said. “There are some people that will be like, no, the last thing I want to do is roll something out at 5PM on a Friday, because if something goes wrong, then there goes the weekend. Some organizations are like ‘well, yeah, that’s where we’re getting to, we do enough testing’ and really begin to say, yeah, we can ship a new service whenever. We have that confidence because we’ve done the engineering work and the cultural work in order to be able to do this. That’s progressive delivery.” 

The post Progressive delivery: Testing software through limited releases appeared first on SD Times.

TDD
TDD has become popular with Agile and DevOps teams because it saves time. Tests are written from requirements in the form of use cases and user stories and then code is written to pass those tests. TDD further advances the concept of building smaller pieces of code, and the little code quality successes along the way add up to big ones. TDD builds on the older concept of extreme programming (XP).

RELATED CONTENT: There’s more to testing than simply testing

“Test-driven development helps drive quality from the beginning and [helps developers] find defects in the requirements before they need to write code,” said Thomas Murphy, senior director analyst at Gartner.

Todd Lemmonds, QA architect at health benefits company Anthem, said his team is having a hard time with it because they’re stuck in an interim phase.

“TDD is the first step to kind of move in the Agile direction,” said Lemmonds. “How I explain it to people is you’re basically focusing all your attention on [validating] these acceptance criteria based on this one story. And then they’re like, OK what tests do I need to create and pass before this thing can move to the next level? They’re validating technical specifications whereas [acceptance test driven development] is validating business specifications and that’s what’s presented to the stakeholders at the end of the day.”

Progressive Software Delivery
Progressive software delivery is often misdefined by parsing the words. The thinking is if testing is moving forward (becoming more modern or maturing), then it’s “progressive.” Progressive delivery is something Agile and DevOps teams with a CI/CD pipeline use to further their mission of delivering higher-quality applications faster that users actually like. It can involve a variety of tests and deployments including A/B and multivariate testing using feature flags, blue-green and canary deployments as well as observability. The “progressive” part is rolling out a feature to progressively larger audiences.

“Progressive software delivery is an effective strategy to mitigate the risk to business operations caused by product changes,” said Nancy Kastl, executive director of testing services at digital transformation agency SPR. “The purpose is to learn from the experiences of the pilot group, quickly resolve any issues that may arise and plan improvements for the full rollout.”

Other benefits Kastl perceives include:

• Verification of correctness of permissions setup for business users
• Discovery of business workflow issues or data inaccuracy not detected during testing activities
• Effective training on the software product
• The ability to provide responsive support during first-time product usage
• The ability to monitor performance and stability of the software product under actual production conditions including servers and networks

“Global companies with a very large software product user base and custom configurations by country or region often use this approach for planning rollout of software products,” Kastl said.

Chaos Engineering
Chaos engineering is literally testing the effects of chaos (infrastructure, network and application failures) as it relates to an application’s resiliency. The idea originated at Netflix with a program called “Chaos Monkey,” which randomly chooses a server and disables it. Eventually, Netflix created an entire suite of open-source tools called the “Simian Army” to test for more types of failures, such as a network failure or an AWS region or availability zone drop. 

The Simian Army project is no longer actively maintained but some of its functionality has been moved to other Netflix projects. Chaos engineering lives on. In fact, Gartner is seeing a lot of interest in it.

“Now what you’re starting to see are a couple of commercial implementations. For chaos to be accepted more broadly, often you need something more commercial,” said Gartner’s Murphy. “It’s not that you need commercial software, it’s going to be a community around it so if I need something, someone can help me understand how to do it safely.”

Chaos engineering is not something teams suddenly just do. It usually takes a couple of years because they’ll experiment in phases, such as lab testing, application testing and pre-production. 

Chris Lewis, engineering director at technology consulting firm DMW Group, said his firm has tried chaos engineering on a small scale, introducing the concept to DMW’s rather conservative clientele.

“We’ve introduced it in a pilot sense showing them it can be used to get under the hood of non-functional requirements and showing that they’re actually being met,” said Lewis. “I think very few of them would be willing to push the button on it in production because they’re still nervous. People in leadership positions [at those client organizations] have come from a much more traditional background.”

Chaos engineering is more common among digital disruptors and smaller innovative companies that distinguish themselves using the latest technologies and techniques.

H2: Proceed with caution

Expanding more testing techniques can be beneficial when organizations are actually prepared to do that. One common mistake is trying to take on too much too soon and then failing to reap the intended benefits. Raj Kanuparthi, founder and CEO of custom software development company Narwal, said in some cases, people need to be more realistic. 

“If I don’t have anything in place, then I get my basics right, [create] a road map, then step-by-step instrument. You can do it really fast, but you have to know how you’re approaching it,” said Kanuparthi, who is a big proponent of Tricentis. “So many take on too much and try 10 things but don’t make meaningful progress on anything and then say, ‘It doesn’t work.”

The post Engineering practices that advance testing appeared first on SD Times.

“Just because it works doesn’t mean it’s a good experience,” said Thomas Murphy, senior director analyst at Gartner. “If it’s my employee, sometimes I make them suffer but that means I’m going to lose productivity and it may impact employee retention. If it’s my customers, I can lose retention because I did not meet the objectives in the first place.”

Today’s applications should help facilitate the organization’s business goals while providing the kind of experience end users expect. To accomplish that, software teams must take a more holistic approach to testing than they have done traditionally, which involves more types of tests and more roles involved in testing.

“The patterns of practice come from architecture and the whole idea of designing patterns,” said Murphy. “The best practices 10 years ago are not best practices today and the best practices three years ago are probably not the best practices today. The leading practices are the things Google, Facebook and Netflix were doing three to five years ago.”

Chris Lewis, engineering director at technology consulting firm DMW Group, said his enterprise clients are seeing the positive impact a test-first mindset has had over the past couple of years.

“The things I’ve seen [are] particularly in the security and infrastructure world where historically testing hasn’t been something that’s been on the agenda. Those people tend to come from more traditional, typically full-stack software development backgrounds and they’re now wanting more control of the development processes end to end,” said Lewis. “They started to inject testing thinking across the life cycle.”

Nancy Kastl, executive director of testing services at digital transformation agency SPR, said a philosophical evolution is occurring regarding what to test, when to test and who does the testing. 

“Regarding what to test, the movement continues away from both manual [and] automated UI testing methods and toward API and unit-level testing. This allows testing to be done sooner, more efficiently and fosters better test coverage,” said Kastl.

“When” means testing earlier and throughout the SDLC.

“Companies are continuing to adopt Agile or improve the way they are using Agile to achieve its benefits of continuous delivery,” said Kastl. “With the current movement to continuous integration and delivery, the ‘shift-left’ philosophy is now embedded in continuous testing.”

However, when everyone’s responsible for testing, arguably nobody’s responsible, unless it’s clear how testing should be done by whom, when, and how. Testing can no longer be the sole domain of testers and QA engineers because finding and fixing bugs late in the SDLC is inadequate, unnecessarily costly and untenable as application teams continue to shrink their delivery cycles. As a result, testing must necessarily shift left to developers and right to production, involving more roles.

“This continues to be a matter of debate. Is it the developers, testers, business analysts, product owners, business users, project managers [or]  someone else?” said Kastl. “With an emphasis on test automation requiring coding skills, some argue for developers to do the testing beyond just unit tests.”

Meanwhile, the scope of tests continues to expand beyond unit, integration, system and user acceptance testing (UAT) to include security, performance, UX, smoke, and regression testing. Feature flags, progressive software delivery, chaos engineering and test-driven development are also considered part of the testing mix today.

Security goes beyond penetration testing
Organizations irrespective of industry are prioritizing security testing to minimize vulnerabilities and manage threats more effectively.

“Threat modeling would be a starting point. The other thing is that AI and machine learning are giving me more informed views of both code and code quality,” said Gartner’s Murphy. “There are so many different kinds of attacks that occur and sometimes we think we’ve taken these precautions but the problem is that while you were able to stop [an attack] one way, they’re going to find different ways to launch it, different ways it’s going to behave, different ways that it will be hidden so you don’t detect it.”

In addition to penetration testing, organizations may use a combination of tools and services that can vary based on the application. Some of the more common ones are static and dynamic application security testing, mobile application security testing, database security testing, software composition analysis and appsec testing as a service.

DMW Group’s Lewis said his organization helps clients improve the way they define their compliance and security rules as code, typically working with people in conventional security architecture and compliance functions.

“We get them to think about what the outcomes are that they really want to achieve and then provide them with expertise to actually turn those into code,” said Lewis.

SPR’s Kastl said continuous delivery requires continuous security verification to provide early insight into potential security vulnerabilities.

“Security, like quality, is hard to build in at the end of a software project and should be prioritized through the project life cycle,” said Kastl. “The Application Security Verification Standard (ASVS) is a framework of security requirements and controls that define a secure application with developing and testing modern applications.”

Kastl said that includes:

• adding security requirements to the product backlog with the same attention to coverage as the application’s functionality;
• a standards-based test repository that includes reusable test cases for manual testing and to build automated tests for Level 1 requirements in the ASVS categories, which include authentication, session management, and function-level access control;
• in-sprint security testing that’s integrated into the development process while leveraging existing approaches such as Agile, CI/CD and DevOps;
• post-production security testing that surfaces vulnerabilities requiring immediate attention before opting for a full penetration test;
• and, penetration testing to find and exploit vulnerabilities and to determine if previously detected vulnerabilities have been fixed. 

“The OWASP Top 10 is a list of the most common security vulnerabilities,” said Kastl. It’s based on data gathered from hundreds of organizations and over 100,000 real world applications and APIs.”

Performance testing beyond load testing
Load testing ensures that the application continues to operate as intended as the workload increases with emphasis on the upper limit. By comparison, scalability testing considers both minimum and maximum loads. In addition, it’s wise to test outside of normal workloads (stress testing), to see how the application performs when workloads suddenly spike (spike testing) and how well a normal workload endures over time (endurance testing).

“Performance really impacts people from a usability perspective. It used to be if your page didn’t load within this amount of time, they’d click away and then it wasn’t just about the page, it was about the performance of specific elements that could be mapped to shopping cart behavior,” said Gartner’s Murphy.

For example, GPS navigation and wearable technology company Garmin suffered a multi-day outage when it was hit by a ransomware attack in July 2020. Its devices were unable to upload activity to Strava’s mobile app and website for runners and cyclists. The situation underscores the fact that cybersecurity breaches can have downstream effects.

“I think Strava had a 40% drop in data uploads. Pretty soon, all this data in the last three or four days is going to start uploading to them so they’re going to get hit with a spike of data, so those types of things can happen,” said Murphy.

To prepare for that sort of thing, one could run performance and stress tests on every build or use feature flags to compare performance with the prior build.

Instead of waiting for a load test at the end of a project to detect potential performance issues, performance tests can be used to baseline the performance of an application under development.

“By measuring the response time for a single user performing specific functions, these metrics can be gathered and compared for each build of the application,” said Kastl. “This provides an early warning of potential performance issues. These baseline performance tests can be integrated with your CI/CD pipeline for continuous monitoring of the application’s performance.”

Mobile and IoT devices, such as wearables, have increased the need for more comprehensive performance testing and there’s still a lot of room for improvement.

“As the industry has moved more to cloud-based technology, performance testing has become more paramount,” said Todd Lemmonds, QA architect at health benefits company Anthem, a Sauce Labs customer. “One of my current initiatives is to integrate performance testing into the CI/CD pipeline. It’s always done more toward UAT which, in my mind, is too late.”

To affect that change, the developers need to think about performance and how the analytics need to be structured in a way that allows the business to make decisions. The artifacts can be used later during a full systems performance test.

“We’ve migrated three channels on to cloud, [but] we’ve never done a performance test of all three channels working at capacity,” said Lemmonds. “We need to think about that stuff and predict the growth pattern over the next five years. We need to make sure that not only can our cloud technologies handle that but what the full system performance is going to look like. Then, you run into issues like all of our subsystems are not able to handle the database connections so we have to come up with all kinds of ways to virtualize the services, which is nothing new to Google and Amazon, but [for] a company like Anthem, it’s very difficult.”

DMW Group’s Lewis said some of his clients have ignored performance testing in cloud environments since cloud environments are elastic.

“We have to bring them back to reality and say, ‘Look, there is an art form here that has significantly changed and you really need to start thinking about it in more detail,” said Lewis.

UX testing beyond UI and UAT
While UI and UAT testing remain important, UI testing is only a subset of what needs to be done for UX testing, while traditional UAT happens late in the cycle. Feature flagging helps by providing early insight into what’s resonating and not resonating with users while generating valuable data. There’s also usability testing including focus groups, session recording, eye tracking and quick one-question in-app surveys that ask whether the user “loves” the app or not.

One area that tends to lack adequate focus is accessibility testing, however. 

“More than 54 million U.S. consumers have disabilities and face unique challenges accessing products, services and information on the web and mobile devices,” said SPR’s Kastl. “Accessibility must be addressed throughout the development of a project to ensure applications are accessible to individuals with vision loss, low vision, color blindness or learning loss, and to those otherwise challenged by motor skills.”

The main issue is a lack of awareness, especially among people who lack firsthand or secondhand experience with disabilities. While there are no regulations to enforce, accessibility-related lawsuits are growing exponentially. 

“The first step to ensuring an application’s accessibility is to include ADA Section 508 or WCAG 2.1 Accessibility standards as requirements in the product’s backlog along with functional requirements,” said Kastl.

Non-compliance to an accessibility standard on one web page tends to be repeated on all web pages or throughout a mobile application. To detect non-compliant practices as early as possible, wireframes and templates for web and mobile applications should be reviewed for potential non-compliant designed components, Kastl said. In addition to the design review, there should be a code review in which development teams perform self-assessments using tools and practices to identify standards that have not been followed in coding practices. Corrective action should be taken by the team prior to the start of application testing. Then, during in-sprint testing activities, assistive technologies and tools such as screen readers, screen magnification and speed recognition software should be used to test web pages and mobile applications against accessibility standards. Automated tools can detect and report non-compliance.

Gartner’s Murphy said organizations should be monitoring app ratings and reviews as well as social media sentiment on an ongoing basis.

“You have to monitor those things, and you should. You’re feeding stuff like that into a system such as Statuspage or PagerDuty so that you know something’s gone wrong,” said Murphy. “It may not just be monitoring your site. It’s also monitoring those external sources because they may be the leading indicator.”

The post There’s more to testing than simply testing appeared first on SD Times.

There is another way, according to Jon Noronha, senior vice president of product at Optimizely, a progressive delivery and experimentation platform provider. Feature experimentation, he said, allows businesses to go to market quicker while improving product quality and minimizing the fear of failure.  

“I like to think of feature experimentation as a safety net. It’s something that gives people the confidence to do something bold or risky,” he said. “Imagine you are jumping on a trapeze with no net. You’re going to be really scared to take even the smallest step because if you fall, you’re going to really hurt yourself. When there is a net, you know the worst thing that can happen is you land on the net and bounce a little bit.”

RELATED CONTENT:
Waving the flag for feature experimentation
Speed releases with feature flags

Feature experimentation is that net that allows you to leap, but catches you if you fall, Noronha explained. It enables businesses to take small risks, roll it out to a few users, and measure the impact of changes before releasing it to 100% of the user base. 

Christopher Condo, a principal analyst at the research firm Forrester, said, “In order to be innovative, you need to really understand what your customers want and be willing to try new experiences. Using feature experimentation allows businesses to be more Agile, more willing to put out smaller pieces of functionality, test it with users and continue to iterate and grow.” 

However, there are still some steps businesses need to take before they can squeeze out the benefits of feature experiment. They need to learn to walk before they can run.

Progressive Delivery: Walk
Progressive delivery is the walk that comes before the run (feature experimentation), according to Dave Karow, continuous delivery evangelist at Spilt, a feature flag, experimentation and CD solution provider. Progressive delivery assumes you have the “crawl” part already in place, which is continuous delivery and continuous integration. For instance, teams need to have a  centralized source of information in place where developers can check in code and have it automatically tested for basic sanity with no human intervention, Karow explained. 

Without that, you won’t see the true promise of progressive delivery, John Kodumal, CTO and co-founder of LaunchDarkly, a feature flag and toggle management company, added.

“Imagine a developer is going to work on a feature, take a copy of the source code and take a copy of their plan and work on it for some time. When they are done, they have to merge their code back into the source code that is going to go out into production,” Karow explained. “In the meantime, other developers have been making other changes. What happens is literally referred to in the community as ‘merge hell.’ You get to a point where you think you finished your work and you have to merge back in and then you discover all these conflicts. That’s the crawl stuff. It’s about making changes to the software faster and synchronizing with coworkers to find problems in near real-time.” 

Once you have the crawl part situated, the progressive delivery part leverages feature flags (also known as feature toggles, bits or flippers) to get features into production faster without breaking the application. According to Optmizely’s Noronha, feature flags are one layer off the safety net that feature experimentation offers. It allows the development teams to try things at lower risks and roll out by slowly and gradually enabling developers to expose key functionalities with the goal of catching bugs or errors before they become widespread. “It’s making it easier to roll things out faster, but be able to stop rollouts without a lot of drama,” Karow said. 

Some examples of feature flags

Feature flags come in several different flavors. Among them are: 

• Release flags that enable trunk-based development. “Release Toggles allow incomplete and un-tested codepaths to be shipped to production as latent code which may never be turned on,” Pete Hodgson, an independent software delivery consultant, wrote in a post on MartinFowler.com.
• Experiment flags that leverage A/B testing to make data-driven optimizations. “By their nature Experiment Toggles are highly dynamic – each incoming request is likely on behalf of a different user and thus might be routed differently than the last,” Hodgson wrote.
• Ops flags, which enable teams to control operational aspects of their solution’s behavior. Hodgson explained “We might introduce an Ops Toggle when rolling out a new feature which has unclear performance implications so that system operators can disable or degrade that feature quickly in production if needed.”
• Permission flags that can change the features or experience for certain users. “For example we may have a set of ‘premium’ features which we only toggle on for our paying customers. Or perhaps we have a set of “alpha” features which are only available to internal users and another set of “beta” features which are only available to internal users plus beta users,” Hodgson wrote.

One way to look at it is through the concept of canary releases, according to Kodumal, which is the idea of being able to release some change and controlling the exposure of that change to a smaller audience to validate that change before rolling it out more broadly. 

These flags help minimize the blast radius of possible messy situations, according to Forrester’s Condo. “You’re slowly gauging the success of your application based on: Is it working as planned? Do customers find it useful? Are they complaining? Has the call value gone up or stayed steady? Are the error logs growing?” As developers implement progressive delivery, they will become better at detecting when things are broken, Condo explained.

“The first thing is to get the hygiene right so you can build software more often with less drama. Implement progressive delivery so you can get that all the way to production. Then dip your toes into experimentation by making sure you have that data automated,” said Split’s Karow. 

Feature experimentation: Run
Feature experimentation is similar to progessive delivery, but with better data, according to Karow. 

“Feature experimentation takes progressive delivery further by looking at the data and not just learning whether or not something blew up, but why it did,” he said. 

By being able to consume the data and understand why things happen, it enables businesses to make better data-driven decisions. The whole reason you do smaller releases is to actually confirm they were having the impact you were looking for, that there were no bugs, and you are meeting users’ expectations, according to Optmizely’s Noronha. 

It does that through A/B testing, multi-armed bandits, and chaos experiments, according to LaunchDarkly’s Kodumal. A/B testing tests multiple versions of a feature to see how it is accepted. Multi-armed bandits is a variation of an A/B test, but instead of waiting for a test to complete it uses algorithms to increase traffic allocations to see how features work. And chaos experiments refer to finding out what doesn’t work rather than looking for what does work. 

“You might drive a feature experiment that is intended to do something like improve engagement around a specific feature you are building,” said Kodumal. “You define the metric, build the experiment, and validate whether or not the change being made is being received positively.”

The reason why feature experimentation is becoming so popular is because it enables development teams to deploy code without actually turning it on right away. You can deploy it into production, test it in production, without the general user base seeing it, and either release it or keep it hidden until it’s ready, Forrester’s Condo explained. 

In some cases, a business may decide to release the feature or new solution to its users, but give them the ability to turn it on or off themselves and see how many people like the enhanced experience. “Feature experimentation makes that feature a system of record. It becomes part of how you deliver experiences to your customers in a varied experience,” said Condo. “It’s like the idea of Google. How many times on Google or Gmail has it said ‘here is a brand new experience, do you want to use it?’ And you said ‘no I’m not ready.’ It is allowing companies to modernize in smaller pieces rather than all at once.”

What feature experimentation does is it focuses on the measurement side, while progressive delivery focused on just releasing smaller pieces. “Now you are comparing the 10% release against the other 90% to see what the difference is, measuring that, understanding the impact, quantifying it, and learning what’s actually working,” said Opitmizely’s Noronha.

While it does reduce risks for businesses, it doesn’t eliminate the chance for failure. Karow explained businesses have to be willing to accept failure or they are not going to get very far. “At the end of the day, what really matters is whether a feature is going to help a user or make them want to use it or not. What a lot of these techniques are about is how do I get hard data to prove what actually works,” Karow explained. 

To get started, Noronha recommends to look for parts of the user experience that drive traffic and make simple changes to experiment with. Once they prove it out and get it entrenched in one area, then it can be quickly spread out to other areas more easily. 

“It’s sort of addictive. Once people get used to working in this way, they don’t want to go back to just launching things. They start to resent not knowing what the adoption of their product is,” he said. 

Noronha expects progressive delivery and feature experimentation will eventually merge. “Everyone’s going to roll out into small pieces, and everyone’s going to measure how those things are doing against the control,” he said. 

“What both progressive delivery and feature experimentation do is provide the ability to de-risk your investment in new software and R&D. They give you the tooling you need to think about decomposing those big risky things into smaller, achievable things where you have faster feedback loops from customers,” LaunchDarkly’s Kodumal added.

Experimenting with A/B testing
A/B testing is one of the most common types of experiments, according to John Kodumal, CTO and co-founder of LaunchDarkly, a feature flag and toggle management company

It is the method of comparing two versions of an application or functionality. Previously, it was more commonly used for front-end or visual aesthetic changes done to a website rather than a product. For instance, one could take a button that was blue and make it red, and see if that drives more clicks, Jon Noronha, senior vice president of product at Optimizely, a progressive delivery and experimentation platform provider, explained. “In the past several years, we’ve really transitioned to focusing more on what I would call feature experimentation, which is really building technology that helps people test the core logic of how their product is actually built,” he said. 

A/B testing is used in feature experimentation to test out two competing theories and see which one achieves the result the team is looking for. Christopher Condo, a principal analyst at the research firm Forrester, explained that “It requires someone to know and say ‘I think if we alter this experience to the end user, we can improve the value.’ You as a developer want to get a deeper understanding of what kind of changes can improve the UX and so A/B testing comes into play now to show different experiences from different people and how they are being used.”

According to Dave Karow, continuous delivery evangelist at Spilt, a feature flag, experimentation and CD solution provider, this is especially useful in environments where a “very important person” within the business has an opinion or the “highest paid person” on the team wants you to do something and a majority of the team members don’t agree. He explained normally what someone thinks is going to work, doesn’t work 8 or 9 times out of 10. But with A/B testing, developers can still test out that theory, and if it fails they can provide metrics and data on why it didn’t work without having to release it to all their customers. 

A good A/B test statistical engine should be able to tell you within a few days which experience or feature is better. Once you know which version is performing better, you can slowly replace it and continue to iterate to see if you can make it work even better, Condo explained. 

Kodumal explained A/B testing works better with feature experimentation because in progressive delivery the customer base you are gradually delivering to is too small to run full experiments on and achieve the statistical significance of a fully rigorous experiment.

“We often find that teams get value out of some of the simpler use cases in progressive delivery before moving onto full experimentation,” he said.  

Feature experimentation is for any company with user-facing technology
Feature experimentation has already been used among industry leaders like eBay, LinkedIn and Netflix for years. 

“Major redesigns…improve your service by allowing members to find the content they want to watch faster. However, they are too risky to roll out without extensive A/B testing, which enables us to prove that the new experience is preferred over the old,” Netflix wrote in a 2016 blog post explaining its experimentation platform. 

Up until recently it was only available to those large companies because it was expensive. The alternative was to build your own product, with the time and costs associated with that.  “Now there is a growing marketplace of solutions that allow anyone to do the same amount of rigor without having to spend years and millions of dollars building it in-house,” said Dave Karow, continuous delivery evangelist at Spilt, a feature flag, experimentation and CD solution provider

Additionally,  feature experimentation used to be a hard process to get started with, with no real guidelines to follow. What has started to happen is the large companies are getting to share how their engineering teams operate and provide more information on what goes on behind the scenes, according to Christopher Condo, a principal analyst at the research firm Forrester. “In the past, you never gave away the recipe or what you were doing. It was always considered intellectual property. But today, sharing information, people realize that it’s really helping the whole industry for everybody to get better education about how these things work,” Condo said. 

Today, the practice has expanded into something that every major company with some kind of user-facing technology can and should take advantage of, according to Jon Noronha, senior vice president of product at Optimizely, a progressive delivery and experimentation platform provider.

Norona predicts feature experimentation “will eventually grow to be adopted the same way we see things like source control and branching. It’s going to go from something that just big technology companies do to something that every business has to have to keep up.”

“Companies that are able to provide that innovation faster and bring that functionality that consumers are demanding, they are the ones that are succeeding, and the ones that aren’t are the ones that are left behind and that consumers are starting to move away from,” John Kodumal, CTO and co-founder of LaunchDarkly, a feature flag and toggle management company, added.

The post <span class="sdt-premium">premium</span> Feature experimentation: Walk before you run appeared first on SD Times.

RedMonk founder James Governor, who coined the term progressive delivery in 2018 to describe a basket of related skills and technologies for gradual releases that reduce risk in application delivery, announced his 2020 research topics recently and at the top of his list was progressive delivery. 

It’s great to experiment to ensure your software works as intended, yet progressive delivery is motivated by business factors — if the company releases software and business metrics go down, it’s better to know that before a wider release. “I want to watch it to limit the blast radius, so as I’m ramping it up, I want to be able to know whether it’s going well or not,” explained Dave Karow, Continuous Delivery evangelist at feature delivery platform provider Split Software. “I want to learn before I get to 100%. You know, one of our senior engineering leaders used to be at a very large file-sharing provider and he admitted that even when they used gradual rollouts, they didn’t tend to learn about [issues] until they were past the 50% mark. That’s painful compared to finding problems earlier in a gradual rollout. You want to set yourself up to learn about unforeseen issues as quickly as possible in production, and it would be nice if you didn’t have to hurt most of your users — if not all of them — before you figured out you had an incident.”

These patterns have been used by the largest e-commerce leaders for years. At Walmart, they use Progressive Delivery for two main purposes: one is called “test to learn,” and the other is “test to launch.” Test to learn is essentially A/B testing; which version of the software yields more of the desired user behaviors, such as buying more items or signing up for premium services. Test to launch, on the other hand, is more like application monitoring, where you’re watching a gradual rollout of the application to see that not only systems metrics, but also business metrics, don’t decline as the software is given to more users.

The ability to effectively roll out software to small cohorts before wide release to assess impact on the business comes with the assumption that business and development people are on the same page. Karow said he’s seeing the greatest success with progressive delivery in Agile and DevOps teams.

“Those who most embrace this and get the most value out of it are already sort of in a two-pizza team,” Karow said, referring to Amazon CEO Jeff Bezos’ belief that if a team is so large that you can’t feed it with two pizzas, if won’t work effectively. “I have everybody related to this project within shouting distance of each other in a room or at least that tight in terms of a remote teaming thing, so that there’s no specification being written by one group and coding being done by another group and testing being done by another group and deployment being done by another group. Everybody knows what we’re building this week, and what we are trying to accomplish.”

These teams, he added, understand they might not get it exactly right the first time, so they’re looking to see how they can iterate quickly to improve it. “We don’t want to mark it ‘done’ because we shipped; we want to mark it ‘done’ because we moved the number.”

Split’s feature delivery management software is the means by which people work together to experiment with and deliver quality software that meets business and user requirements. “If [companies] are still solving fundamental problems to get people talking to each other and get the business and the developers on the same page, we’re not a silver bullet for that. We’re the means by which to actually do work together, not to get people to work together.”

With progressive delivery, there is a natural progression of cohorts you want to expose to new code in production, Karow said. “The first cohort you want to expose in production to the new code is your developers and your testers. This gives them one last chance to be sure everything works as expected in the actual production environment, without any risk to users. Then there’s dogfooding. If I use my own product, then I’m going to go from my developers and testers to my non-developer users. Then I might go out to my friendlies, or my freebies.  Finally, I’ll begin rolling it out to the rest of my general population.”

Learn more at: www.split.io

Content provided by SD Times and Split Software

The post Learning about your software progressively appeared first on SD Times.

One area involves value streams, the creation of which allow organizations to see waste in their organization and eliminate for better efficiency and, ultimately, quality. 

Another is CI/CD. The practice of continually introducing changes to the codebase and deploying those changes out for testing and feedback prior to wide release is well understood. So, how does the industry improve on continuous delivery?

RELATED CONTENT:
Waving the flag for feature experimentation
Feature flags simplify feature development and testing 

According to Adam Zimman, VP of platform at feature experimentation software provider LaunchDarkly, the future is through ‘progressive delivery.’ 

As defined by Redmonk analyst James Governor and Zimman in mid-2018, progressive delivery allows organizations to roll out changes while being mindful of the users’ experience. In a nutshell, Zimman said it’s about having increasing control over who gets to see what, when.  “The idea of being able to garner feedback from specific cohorts prior to broad release of features or products has been a thing since people started selling stuff,” Zimman said. “In the past five years, as the ideas around more continuous delivery — a faster cadence of release cycles — has shifted what tools people look to to be able to do this type of experimentation or controlled rollout.”

A key tool to creating control points in software is a feature flag.  Developers add these flags into their code that can be turned on or off for certain cohorts. It can be as simple as turning a feature on or off, or giving access in certain contexts but not in others. For instance, you might want to allow access to features in a staging environment for testing but not in production. And those types of access controls ultimately enable an organization to delegate who gets to manage them.

“If it’s something the developer is providing the feature flag mechanism themselves, or they’re storing the values in a simple database, then the ownership of that control resides with developers,” Zimman said. “You’re talking about accessing a database directly; you’re talking about changing value in that database, so you need to have some level of engineering background to be able to do that without shooting yourself in the foot, basically.”

Then, he added, as you start to roll that into production, “you may look to transition that ownership to an operations team, commensurate with a DevOps model where you want to have equal visibility and shared responsibility.” Finally, as you start to look at this functionality as it starts to impact users, you want to include the individual business owners that are closest to those business outcomes. “That could be product management, that could be product marketing, that could be sales, or customer success,” he continued. “All of these teams that are closer to business outcomes, we have seen in our customers being given responsibility for these control points.”

The use of feature flags has been associated with feature experimentation — A/B testing, blue-green deployments — but Zimman sees distinctions between experimentation and progressive delivery. “Often times, you think of an experiment, I want to put something out there and test if it’s better than what I had before. Or, if it’s something net new, then I want to see if people like it. But the goal of the experiment often times implies that you’re going to have an outcome that either is going to roll it back for everyone, so that no one gets that new thing because it was worse, or you’re going to roll it out to everyone.

“One of the key distinctions with progressive delivery,” he continued, “is the idea that you actually want to have an end state of this control point that is being thoughtful of the user base that it’s applicable to. I talk about this in the context of B2B. That makes a lot of sense for anybody who has kind of looked at the idea of mult-tiers of service, where you have a premium tier, and entry-level tier, and so on. You actually want new features to potentially to only go to one of those groups. You’re not actually rolling it out to everyone.” 

In a post from Aug. 2018, Redmonk’s Governor wrote: “A great deal of our thinking in application delivery has been about consistency between development and deployment targets – see the promise of Docker and then Kubernetes. A core aspect of cattle vs pets as an analogy is that the cattle are all the same. The fleet is homogeneous. But we’re moving into a multicloud, multiplatform, hybrid world, where deployment targets are vary and we may want to route deployment and make it more, well, progressive.”

And that’s progress.

The post Industry Watch: What follows CD? Progressive delivery appeared first on SD Times.