TDD
TDD has become popular with Agile and DevOps teams because it saves time. Tests are written from requirements in the form of use cases and user stories and then code is written to pass those tests. TDD further advances the concept of building smaller pieces of code, and the little code quality successes along the way add up to big ones. TDD builds on the older concept of extreme programming (XP).

RELATED CONTENT: There’s more to testing than simply testing

“Test-driven development helps drive quality from the beginning and [helps developers] find defects in the requirements before they need to write code,” said Thomas Murphy, senior director analyst at Gartner.

Todd Lemmonds, QA architect at health benefits company Anthem, said his team is having a hard time with it because they’re stuck in an interim phase.

“TDD is the first step to kind of move in the Agile direction,” said Lemmonds. “How I explain it to people is you’re basically focusing all your attention on [validating] these acceptance criteria based on this one story. And then they’re like, OK what tests do I need to create and pass before this thing can move to the next level? They’re validating technical specifications whereas [acceptance test driven development] is validating business specifications and that’s what’s presented to the stakeholders at the end of the day.”

Progressive Software Delivery
Progressive software delivery is often misdefined by parsing the words. The thinking is if testing is moving forward (becoming more modern or maturing), then it’s “progressive.” Progressive delivery is something Agile and DevOps teams with a CI/CD pipeline use to further their mission of delivering higher-quality applications faster that users actually like. It can involve a variety of tests and deployments including A/B and multivariate testing using feature flags, blue-green and canary deployments as well as observability. The “progressive” part is rolling out a feature to progressively larger audiences.

“Progressive software delivery is an effective strategy to mitigate the risk to business operations caused by product changes,” said Nancy Kastl, executive director of testing services at digital transformation agency SPR. “The purpose is to learn from the experiences of the pilot group, quickly resolve any issues that may arise and plan improvements for the full rollout.”

Other benefits Kastl perceives include:

• Verification of correctness of permissions setup for business users
• Discovery of business workflow issues or data inaccuracy not detected during testing activities
• Effective training on the software product
• The ability to provide responsive support during first-time product usage
• The ability to monitor performance and stability of the software product under actual production conditions including servers and networks

“Global companies with a very large software product user base and custom configurations by country or region often use this approach for planning rollout of software products,” Kastl said.

Chaos Engineering
Chaos engineering is literally testing the effects of chaos (infrastructure, network and application failures) as it relates to an application’s resiliency. The idea originated at Netflix with a program called “Chaos Monkey,” which randomly chooses a server and disables it. Eventually, Netflix created an entire suite of open-source tools called the “Simian Army” to test for more types of failures, such as a network failure or an AWS region or availability zone drop. 

The Simian Army project is no longer actively maintained but some of its functionality has been moved to other Netflix projects. Chaos engineering lives on. In fact, Gartner is seeing a lot of interest in it.

“Now what you’re starting to see are a couple of commercial implementations. For chaos to be accepted more broadly, often you need something more commercial,” said Gartner’s Murphy. “It’s not that you need commercial software, it’s going to be a community around it so if I need something, someone can help me understand how to do it safely.”

Chaos engineering is not something teams suddenly just do. It usually takes a couple of years because they’ll experiment in phases, such as lab testing, application testing and pre-production. 

Chris Lewis, engineering director at technology consulting firm DMW Group, said his firm has tried chaos engineering on a small scale, introducing the concept to DMW’s rather conservative clientele.

“We’ve introduced it in a pilot sense showing them it can be used to get under the hood of non-functional requirements and showing that they’re actually being met,” said Lewis. “I think very few of them would be willing to push the button on it in production because they’re still nervous. People in leadership positions [at those client organizations] have come from a much more traditional background.”

Chaos engineering is more common among digital disruptors and smaller innovative companies that distinguish themselves using the latest technologies and techniques.

H2: Proceed with caution

Expanding more testing techniques can be beneficial when organizations are actually prepared to do that. One common mistake is trying to take on too much too soon and then failing to reap the intended benefits. Raj Kanuparthi, founder and CEO of custom software development company Narwal, said in some cases, people need to be more realistic. 

“If I don’t have anything in place, then I get my basics right, [create] a road map, then step-by-step instrument. You can do it really fast, but you have to know how you’re approaching it,” said Kanuparthi, who is a big proponent of Tricentis. “So many take on too much and try 10 things but don’t make meaningful progress on anything and then say, ‘It doesn’t work.”

The post Engineering practices that advance testing appeared first on SD Times.

“Just because it works doesn’t mean it’s a good experience,” said Thomas Murphy, senior director analyst at Gartner. “If it’s my employee, sometimes I make them suffer but that means I’m going to lose productivity and it may impact employee retention. If it’s my customers, I can lose retention because I did not meet the objectives in the first place.”

Today’s applications should help facilitate the organization’s business goals while providing the kind of experience end users expect. To accomplish that, software teams must take a more holistic approach to testing than they have done traditionally, which involves more types of tests and more roles involved in testing.

“The patterns of practice come from architecture and the whole idea of designing patterns,” said Murphy. “The best practices 10 years ago are not best practices today and the best practices three years ago are probably not the best practices today. The leading practices are the things Google, Facebook and Netflix were doing three to five years ago.”

Chris Lewis, engineering director at technology consulting firm DMW Group, said his enterprise clients are seeing the positive impact a test-first mindset has had over the past couple of years.

“The things I’ve seen [are] particularly in the security and infrastructure world where historically testing hasn’t been something that’s been on the agenda. Those people tend to come from more traditional, typically full-stack software development backgrounds and they’re now wanting more control of the development processes end to end,” said Lewis. “They started to inject testing thinking across the life cycle.”

Nancy Kastl, executive director of testing services at digital transformation agency SPR, said a philosophical evolution is occurring regarding what to test, when to test and who does the testing. 

“Regarding what to test, the movement continues away from both manual [and] automated UI testing methods and toward API and unit-level testing. This allows testing to be done sooner, more efficiently and fosters better test coverage,” said Kastl.

“When” means testing earlier and throughout the SDLC.

“Companies are continuing to adopt Agile or improve the way they are using Agile to achieve its benefits of continuous delivery,” said Kastl. “With the current movement to continuous integration and delivery, the ‘shift-left’ philosophy is now embedded in continuous testing.”

However, when everyone’s responsible for testing, arguably nobody’s responsible, unless it’s clear how testing should be done by whom, when, and how. Testing can no longer be the sole domain of testers and QA engineers because finding and fixing bugs late in the SDLC is inadequate, unnecessarily costly and untenable as application teams continue to shrink their delivery cycles. As a result, testing must necessarily shift left to developers and right to production, involving more roles.

“This continues to be a matter of debate. Is it the developers, testers, business analysts, product owners, business users, project managers [or]  someone else?” said Kastl. “With an emphasis on test automation requiring coding skills, some argue for developers to do the testing beyond just unit tests.”

Meanwhile, the scope of tests continues to expand beyond unit, integration, system and user acceptance testing (UAT) to include security, performance, UX, smoke, and regression testing. Feature flags, progressive software delivery, chaos engineering and test-driven development are also considered part of the testing mix today.

Security goes beyond penetration testing
Organizations irrespective of industry are prioritizing security testing to minimize vulnerabilities and manage threats more effectively.

“Threat modeling would be a starting point. The other thing is that AI and machine learning are giving me more informed views of both code and code quality,” said Gartner’s Murphy. “There are so many different kinds of attacks that occur and sometimes we think we’ve taken these precautions but the problem is that while you were able to stop [an attack] one way, they’re going to find different ways to launch it, different ways it’s going to behave, different ways that it will be hidden so you don’t detect it.”

In addition to penetration testing, organizations may use a combination of tools and services that can vary based on the application. Some of the more common ones are static and dynamic application security testing, mobile application security testing, database security testing, software composition analysis and appsec testing as a service.

DMW Group’s Lewis said his organization helps clients improve the way they define their compliance and security rules as code, typically working with people in conventional security architecture and compliance functions.

“We get them to think about what the outcomes are that they really want to achieve and then provide them with expertise to actually turn those into code,” said Lewis.

SPR’s Kastl said continuous delivery requires continuous security verification to provide early insight into potential security vulnerabilities.

“Security, like quality, is hard to build in at the end of a software project and should be prioritized through the project life cycle,” said Kastl. “The Application Security Verification Standard (ASVS) is a framework of security requirements and controls that define a secure application with developing and testing modern applications.”

Kastl said that includes:

• adding security requirements to the product backlog with the same attention to coverage as the application’s functionality;
• a standards-based test repository that includes reusable test cases for manual testing and to build automated tests for Level 1 requirements in the ASVS categories, which include authentication, session management, and function-level access control;
• in-sprint security testing that’s integrated into the development process while leveraging existing approaches such as Agile, CI/CD and DevOps;
• post-production security testing that surfaces vulnerabilities requiring immediate attention before opting for a full penetration test;
• and, penetration testing to find and exploit vulnerabilities and to determine if previously detected vulnerabilities have been fixed. 

“The OWASP Top 10 is a list of the most common security vulnerabilities,” said Kastl. It’s based on data gathered from hundreds of organizations and over 100,000 real world applications and APIs.”

Performance testing beyond load testing
Load testing ensures that the application continues to operate as intended as the workload increases with emphasis on the upper limit. By comparison, scalability testing considers both minimum and maximum loads. In addition, it’s wise to test outside of normal workloads (stress testing), to see how the application performs when workloads suddenly spike (spike testing) and how well a normal workload endures over time (endurance testing).

“Performance really impacts people from a usability perspective. It used to be if your page didn’t load within this amount of time, they’d click away and then it wasn’t just about the page, it was about the performance of specific elements that could be mapped to shopping cart behavior,” said Gartner’s Murphy.

For example, GPS navigation and wearable technology company Garmin suffered a multi-day outage when it was hit by a ransomware attack in July 2020. Its devices were unable to upload activity to Strava’s mobile app and website for runners and cyclists. The situation underscores the fact that cybersecurity breaches can have downstream effects.

“I think Strava had a 40% drop in data uploads. Pretty soon, all this data in the last three or four days is going to start uploading to them so they’re going to get hit with a spike of data, so those types of things can happen,” said Murphy.

To prepare for that sort of thing, one could run performance and stress tests on every build or use feature flags to compare performance with the prior build.

Instead of waiting for a load test at the end of a project to detect potential performance issues, performance tests can be used to baseline the performance of an application under development.

“By measuring the response time for a single user performing specific functions, these metrics can be gathered and compared for each build of the application,” said Kastl. “This provides an early warning of potential performance issues. These baseline performance tests can be integrated with your CI/CD pipeline for continuous monitoring of the application’s performance.”

Mobile and IoT devices, such as wearables, have increased the need for more comprehensive performance testing and there’s still a lot of room for improvement.

“As the industry has moved more to cloud-based technology, performance testing has become more paramount,” said Todd Lemmonds, QA architect at health benefits company Anthem, a Sauce Labs customer. “One of my current initiatives is to integrate performance testing into the CI/CD pipeline. It’s always done more toward UAT which, in my mind, is too late.”

To affect that change, the developers need to think about performance and how the analytics need to be structured in a way that allows the business to make decisions. The artifacts can be used later during a full systems performance test.

“We’ve migrated three channels on to cloud, [but] we’ve never done a performance test of all three channels working at capacity,” said Lemmonds. “We need to think about that stuff and predict the growth pattern over the next five years. We need to make sure that not only can our cloud technologies handle that but what the full system performance is going to look like. Then, you run into issues like all of our subsystems are not able to handle the database connections so we have to come up with all kinds of ways to virtualize the services, which is nothing new to Google and Amazon, but [for] a company like Anthem, it’s very difficult.”

DMW Group’s Lewis said some of his clients have ignored performance testing in cloud environments since cloud environments are elastic.

“We have to bring them back to reality and say, ‘Look, there is an art form here that has significantly changed and you really need to start thinking about it in more detail,” said Lewis.

UX testing beyond UI and UAT
While UI and UAT testing remain important, UI testing is only a subset of what needs to be done for UX testing, while traditional UAT happens late in the cycle. Feature flagging helps by providing early insight into what’s resonating and not resonating with users while generating valuable data. There’s also usability testing including focus groups, session recording, eye tracking and quick one-question in-app surveys that ask whether the user “loves” the app or not.

One area that tends to lack adequate focus is accessibility testing, however. 

“More than 54 million U.S. consumers have disabilities and face unique challenges accessing products, services and information on the web and mobile devices,” said SPR’s Kastl. “Accessibility must be addressed throughout the development of a project to ensure applications are accessible to individuals with vision loss, low vision, color blindness or learning loss, and to those otherwise challenged by motor skills.”

The main issue is a lack of awareness, especially among people who lack firsthand or secondhand experience with disabilities. While there are no regulations to enforce, accessibility-related lawsuits are growing exponentially. 

“The first step to ensuring an application’s accessibility is to include ADA Section 508 or WCAG 2.1 Accessibility standards as requirements in the product’s backlog along with functional requirements,” said Kastl.

Non-compliance to an accessibility standard on one web page tends to be repeated on all web pages or throughout a mobile application. To detect non-compliant practices as early as possible, wireframes and templates for web and mobile applications should be reviewed for potential non-compliant designed components, Kastl said. In addition to the design review, there should be a code review in which development teams perform self-assessments using tools and practices to identify standards that have not been followed in coding practices. Corrective action should be taken by the team prior to the start of application testing. Then, during in-sprint testing activities, assistive technologies and tools such as screen readers, screen magnification and speed recognition software should be used to test web pages and mobile applications against accessibility standards. Automated tools can detect and report non-compliance.

Gartner’s Murphy said organizations should be monitoring app ratings and reviews as well as social media sentiment on an ongoing basis.

“You have to monitor those things, and you should. You’re feeding stuff like that into a system such as Statuspage or PagerDuty so that you know something’s gone wrong,” said Murphy. “It may not just be monitoring your site. It’s also monitoring those external sources because they may be the leading indicator.”

The post There’s more to testing than simply testing appeared first on SD Times.

For example, Nate Custer, senior manager at testing automation consultancy TTC Global, has been researching autonomous testing tools for about a year. When he started the project, he was new to the company and a client had recently inquired about options. The first goal was to build a technique for evaluating how effective the tools were in testing. 

“The number one issue in testing is test maintenance. That’s what people struggle with the most. The basic idea is that you automate tests to save a little bit of time over and over again. When you test lots of times, you only run tests if the software’s changed, because if the software changes, the test may need to change,” said Custer. “So, when I first evaluate stuff, I care about how fast I can create tests, how much can I automate and the maintenance of those testing projects.”

RELATED CONTENT:
AI and ML make testing smarter… but autonomous tools are a long way from being enterprise-ready
What to look for in a web and mobile test automation tool
Continuous testing isn’t optional anymore

Custer’s job was to show how and where different tools could and could not make an impact. The result of his research is that he’s optimistic, but skeptical.

There’s a lot of potential, but…
Based on first-hand research, Custer believes that there are several areas where AI and ML could have a positive impact. At the top of the list is test selection. Specifically, the ability to test all of what’s in an enterprise, not just web and mobile apps.

“If I want to change my tools from this to that, the new tool has to handle everything in the environment. That’s the first hurdle,” said Custer. “But what tests to run based on this change can be independent from the platform you use to execute your test automation, and so I think that’s the first place where you’re going to see a breakthrough of AI in the enterprise. Here’s what’s changed, which tests should I run? Because if I can run 10% of my tests and get the same benefit in terms of risk management, that’s a huge win.”

The second area of promise is surfacing log differences, so if a test that should take 30 seconds to run suddenly took 90 seconds, the tool might suggest that the delay was caused by a performance issue. 

“Testing creates a lot of information and logs and AI/ML tools are pretty good at spotting things that are out of the ordinary,” said Custer. 

The third area is test generation using synthetic test data because synthetic data can be more practical (faster, cheaper and less risky) to use than production data. 

“I’m at a company right now that does a lot of credit card processing. I need profiles of customers doing the same number of transactions, the same number of cards per household that I would see in production. But I don’t want a copy of the production data because that’s a lot of important information,” said Custer.

Self-healing capabilities showed potential, although Custer wasn’t impressed with the results.

“Everything it healed already worked. So, you haven’t really changed maintenance. When a change is big enough to break my automation, the AI tool had a hard time fixing it,” said Custer. “It would surface really weird things. So, that to me is a little longer-term work for most enterprise applications.”

Are we there yet?
“Are We There Yet?” was the title of Custer’s research project and his conclusion is that autonomous testing isn’t ready for prime time in an enterprise environment.

“I’m not seeing anything I would recommend using for an enterprise customer yet. And the tools that I’ve tested didn’t perform any better. My method was to start with a three-year-old version of software, write some test cases, automate them, go through three years of upgrades and pay attention to the maintenance it took to do those upgrades,” said Custer. “When I did that, I found it didn’t save any maintenance time at all. Everybody’s talking about [AI], everyone’s working on it but there are some of them I’m suspicious about,” said Custer.

For example, one company requested the test script so they could parse it in order to understand it. When Custer asked how long it would take, the company said two or three hours. Another company said it would take two or three months to generate a logical map of a program.

“[T]hat doesn’t sound different from hiring a consultant to write your testing. AI/ML stuff has to actually make life easier and better,” said Custer.

Another disappointment was the lack of support for enterprise applications such as SAP and Oracle eBusiness Suite. 

“There are serious limitations on what technologies they support. If I were writing my own little startup web application, I would look at these tools. But if I were a Fortune 500 company, I think it’s going to take them a couple of years to get there,” said Custer. “The challenge is most of these companies aren’t selling a little add-on that you can add into your existing system. They’re saying change everything from one tool that works to my thing and that’s a huge risk.”

The post Autonomous testing: Are we there yet? appeared first on SD Times.

“Currently, AI/ML works great for testing server-side glitches and, if implemented correctly, it can greatly enhance the accuracy and quantity of testing over time,” said Nate Nead, CEO of custom software development services company Dev.co. “Unfortunately, where AI/ML currently fails is in connecting to the full stack, including UX/UI interfaces with database testing. While that is improving, humans are still best at telling a DevOps engineer what looks best, performs best and feels best.”

RELATED CONTENT:
What to look for in a web and mobile test automation tool
Continuous testing isn’t optional anymore
Forrester’s recommendations for building a successful continuous testing capability

Dev.co has tried solutions from TextCraft.io and BMC, and attempted some custom internal processes, but the true “intelligence” is not where imaginations might lead yet, Nead said.

It’s early days
Gartner Senior Director Analyst Thomas Murphy said autonomous testing is “still on the left-hand side of the Gartner Hype cycle.” (That’s the early adopter stage characterized by inflated expectations.)

The good news is there are lots of places to go for help including industry research firms, consulting firms, and vendors’ services teams. Forrester VP and Principal Analyst Diego Lo Giudice created a five-level maturity model inspired by SAE International’s “Levels of Driving Automation” model. Level 5 (the most advanced level) of Lo Giudice’s model, explained in a report, is fully autonomous, but that won’t be possible anytime soon, he said. Levels one through four represent increasing levels of human augmentation, from minimal to maximum. 

The most recent Gartner Magic Quadrant for Software Test Automation included a section about emerging autonomous testing tools. The topic will be covered more in the future, Murphy said.

“We feel at this point in time that the current market is relatively mature, so we’ve retired that Magic Quadrant and our intent is to start writing more about autonomous capabilities and potentially launch a new market next year,” said Murphy. “But first, we’re trying to get the pieces down to talk about the space and how it works.”

Forrester’s Lo Giudice said AI was included in most of the criteria covered in this year’s Continuous Functional Test Automation Wave.

“There was always the question of, tell me if you’re using AI, what for and what are the benefits,” said Lo Giudice. “Most of the tools in the Wave are using AI, machine learning and automation at varying levels of degree, so it’s becoming mainstream of who’s using AI and machine learning.”

How AI and ML are being used in testing
AI and ML are available for use at different points in the SDLC and for different types of testing. The most popular and mature area is UI testing. 

“Applitools allows you to create a baseline of how tolerant you want to be on the differences. If something moved from the upper right-hand corner to the lower left-hand corner, is that a mistake or are you OK with accepting that as the tests should pass?” said Forrester’s Lo Giudice.  

There’s also log file analysis that can identify patterns and outliers. Gartner’s Murphy said some vendors are using log files and/or a web crawling technique to understand an application and how it’s used.

“I’ll look at the UI and just start exercising it and then figure out all the paths just like you used to have in the early days of web applications, so it’s just recursively building a map by talking through the applications,” said Murphy. “It’s useful when you have a very dynamic application that’s content-oriented [like] ecommerce catalogs, news and feeds.”

If the tool understands the most frequently used features of an application it may also be capable of comparing its findings with the tests that have been run.

“What’s the intersection between the use of the features and the test case that you’ve generated? If that intersection is empty, then you have a concern,” said Forrester’s Lo Giudice. “Am I designing and automating tests for the right features? If there’s a change in that space I want to create tests for those applications. This is an optimization strategy, starting from production.”

Natural language processing (NLP) is another AI technique that’s used in some of the testing tools, albeit to bring autonomous testing capabilities to less technical testers. For example, the Gherkin domain specific language (DSL) for Cucumber has a relatively simple syntax :”Given, When, Then,” but natural language is even easier to use.

“There’s a [free and open source] tool called Gauge created by ThoughtWorks [that] combines NLP together with the concept of BDD so now we can start to say you can write requirements using a relatively normal language and from that the tool can figure out what tests you need, when you met the requirement,” said Gartner’s Murphy. “[T]hen, they connect that up to a couple of different tools that create those [tests] for you and run them.”

Parasoft uses AI to simplify API testing by allowing a user to run the record-and-play tool and from that it generates APIs.

“It would tell you which APIs you need to test if you want to go beyond the UI,” said Forrester’s Lo Giudice. 

Some tools claim to be “self-healing,” such as noticing that a path changed based on a UI change. Instead of making the entire test fail, the tool may recognize that although a field moved, the URL is the same and that the test should pass instead of fail.

“Very often when you’re doing Selenium tests you get a bug, [but] you don’t know whether it’s a real bug of the UI or if it’s just the test that fails because of the locator,” said Lo Giudice. “AI and machine learning can help them get over those sorts of things.”

AI and ML can also be used to identify similar tests that have been created over time so the unnecessary tests can be eliminated. 

Dev.co uses AI and ML to find and fix runtime errors faster.

“The speed improvements of AI/ML allow for runtime errors to be navigated more quickly, typically by binding and rebinding elements in real time, and moving on to later errors that may surface in a particular batch of code,” said Dev.co’s Nead. “Currently, the machine augmentation typically occurs in the binding of the elements, real-time alerts and restarts of testing tools without typically long lags between test runtime.”

Do autonomous testing tools require special skills?
The target audience for autonomous software testing products are technical testers, business testers and developers, generally speaking. While it’s never a bad idea to understand the basics of AI and ML, one does not have to be a data scientist to use the products because the vendor is responsible for ensuring the ongoing accuracy of the algorithms and models used in their products. 

“In most cases, you’re not writing the algorithm, you’re just utilizing it. Being able to understand where it might go wrong and what the strengths or weaknesses of that style are can be useful. It’s not like you have to learn to write in Python,” said Gartner’s Murphy.

Dev.co’s Nead said his QA testing leads and DevOps managers are the ones using autonomous testing tools and that the use of the tools differs based on the role and the project in which the person is engaged.

If you want to build your own autonomous testing capabilities, then data scientists and testers should work together. For example, Capgemini explained in a webinar with Forrester that it had developed an ML model for optimizing Dell server testing. Before Dell introduces a new server, it tests all the possible hardware and software configurations, which exceed over one trillion tests.

“They said the 1.3 trillion possible test cases would take a year to test, so they sat down with smart testers and built a machine learning model that looked at the most frequent business configurations used in the last 3, 4, 5 years,” said Forrester’s Lo Giudice. “They used that data and basically leveraging that data, they identified the test cases they had to test for maximum coverage with a machine learning model that tells you this is the minimum number of test cases [you need to run].”

Instead of needing a year to run 1.3 trillion tests, they were able to run a subset of tests in 15 days. 

Benefits
The Dell example and the use cases outlined above show that autonomous testing can save time and money.

“Speed comes in two ways.  One is how quickly can I create tests? The other is how quickly can I maintain those tests?” said Gartner’s Murphy. “One of the issues people run into when they build automation is that they get swamped with maintenance. I’ve created tons of tests and now how do I run them in the amount of time I have to run them?”

For example, if a DevOps organization completes three builds per hour but testing a build takes an hour, the choices are to wait for the tests to run in sequence or run them in parallel.

“One of the things in CI is don’t break the build. If you start one build, you shouldn’t start another build until you know you have a good build, so if the tests [for three builds] are running [in parallel] I’m breaking the way DevOps works. If we’ve got to wait, then people are laying around before they can test their changes. So if you can say based on the changes you need, you don’t need to run 10,000 tests, just run these 500, that means I can get through a build much faster,” said Murphy.

Similarly, it may be that only 20 tests need to be created instead of 100. Creating fewer tests takes less time and a smaller number of tests takes less time to automate and execute. The savings also extend out to cloud resource usage and testing services.

“The more you can shift the use of AI to the left, the greater your benefits will be,” said Forrester’s Lo Giudice. 

Limitations
The use of AI and ML in testing is relatively new, with a lot of progress being made in the last 12 to 18 months. However, there is always room for improvement, expansion and innovation.

Perhaps the biggest limitation has to do with the tools themselves. While there’s a tendency to think of AI in general terms, there is no general AI one can apply to everything. Instead, the most successful applications of AI and ML are narrow, since artificial narrow intelligence (ANI) is the state of the art. So, no one tool will handle all types of tests on code regardless of how it was built.

“It’s not just the fact that it’s web or not. It’s this tool works on these frameworks or it works for Node.js but it doesn’t work for the website you built in Java, so we’re focused on JavaScript or PHP or Python,” said Gartner’s Murphy. “Worksoft is focused on traditional legacy things, but the way the tool works, I couldn’t just drop it in and test a generic website.”

Dev.co’s Nead considers a human in the loop a limitation.

“Fixes still require an understanding of the underlying code, [because one needs to] react and make notes when errors appear. The biggest boons to testing are the speed improvements offered over existing systems. It may not be huge yet as much of the testing still requires restarting and review from a DevOps engineer, but taken in the aggregate, the savings do go up over time,” said Nead.

Autonomous testing will continue to become more commonplace because it helps testers do a better job of testing faster and cheaper than they have done in the past. The best way to understand how the tools can help is to experiment with them to determine how they fit with existing processes and technologies.

Over time, some teams may find themselves adopting autonomous testing solutions by default, because their favorite tools have simply evolved.

The post AI and ML make testing smarter… but autonomous tools are a long way from being enterprise-ready appeared first on SD Times.

Forrester Research VP and principal analyst Diego Lo Giudice and some of his colleagues, recently published a report that includes 12 essential must-dos that span people, practices and technology. The following is based on a recent interview with Lo Giudice in which he shared insights that are explained in greater detail in the report.

People
Continuous testing requires testing team transformation. Instead of having a centralized test center where all the testers reside, executing and managing all the tests, there’s now a hub-and-spoke structure which includes a small center of excellence and testers that are assigned to different teams.

RELATED CONTENT: Continuous testing isn’t optional anymore

“The traditional way, you had a development team that would write the code and throw it over to the test center to do the testing to find bugs. That’s not the way we operate today because testers are in the Agile teams and what’s in the central team is a small team that’s focusing on best practices,” said Lo Giudice. “The central team is maybe recommending tools, harvesting the good practices from different teams and formalizing and sharing them among the teams. So, there’s a shift from a centralized test center to a federated test center.

The testers working in Agile teams need Agile skills including the ability to talk with developers and product owners from the business.

“That’s a different testing persona,” said Lo Giudice. “The testing persona of the past would look for bugs and be happy he found a lot of bugs. Now [that developers and testers are] on the same team, they have shared goals. Quality is one of them. The tester helps prevent bugs from happening so the tester gets involved earlier on in designing the test cases, helping the developers formalize the unit testing, making sure that developers are doing their unit testing and that they’re covering as much code as possible. [The testers are] helping developers build better quality code from the beginning.”

Also, to align their efforts and jointly produce better quality code, developers and testers also need to share common metrics.

“In the past, we never measured if the level of automation is improving. We never measured how long automation takes because when these teams measure the execution of automation, they check in code in a CI tool and execution kicks off. If it’s suddenly taking longer, then something is going on,” said Lo Giudice. “That’s an indication that the release will be stopped, that the code that was checked in will go back to the team to figure out what the problem was.”

Practices
Behavior-driven development (BDD) is one of the practices teams are adopting. Many of them are using Cucumber, a BDD development tool and Gherkin, its ordinary language parser because when test cases and test scenarios are written in ordinary language, everyone can understand them.

“It helps the collaboration between the product owner from the business, the tester and the developers. The product owner will write what he wants in terms of the behavior of the application together with the test cases and then people will understand that language. He can start thinking about how to write the automation for it and depending on the tools that might be generated from the DSL,” said Lo Giudice.

Other teams have adopted test-driven development (TDD), which differs from BDD.

“TDD is different because it impacts the life cycle. It’s writing the test cases and then the code that passes the test cases,” said Lo Giudice.

Shifting left is another popular practice that involves testing as soon as a new sprint or product development starts. More types of testing have been shifting left over time, and that will continue to be the case. Right now, a lot of organizations are focused on shifting performance testing left because leaving it to the end is too late.

“Testers are part of the team and involved early on. It’s about starting testing, and unit testing is one way of shifting left, but it’s about the testers working with the product owners and the team defining test cases or user acceptance criteria right away when we start writing the user stories in the background,” said Lo Giudice.

Service virtualization is also essential for shifting testing left because developers and testers can mock up resources instead of filing a ticket and then waiting for operations to make a resource available or competing with others to access a resource.

Forrester stopped covering service virtualization separately because it doesn’t have its own market, so it’s now included as part of continuous testing.

“You don’t need the full service virtualization capabilities that the tools three to five years ago were offering, but simplified versions that help you do a stub very quickly,” said Lo Giudice.

Teams also need to shift testing right as well as left.

“It’s monitoring the view into production. If you’re deploying your features frequently in production and the developer can monitor some of the code that they’re deploying, they can prevent performance issues from happening,” said Lo Giudice.

Finally, exploratory testing is replacing the old way of manual testing. Manual testing isn’t going away, but its uses are diminishing.

Technology
The tech stack is more focused on smart automation than traditional test automation. Smart automation uses AI and machine learning to help developers focus on what matters, which simplifies and speeds testing.

“Smart automation tools leverage machine learning and AI to generate from requirements more precise test cases that would optimize the business coverage, so that’s at the design level,” said Lo Giudice. “But there’s also automation of test execution. When code gets checked in, do I have to run all my regression tests or based on the change can I figure out the ones that need to be run and shorten the execution?”

API testing is also important because developers are writing more API and microservice-based applications. Beyond that, there should be fully layered testing and version control with all assets stored centrally.

“All the testing assets necessary to speed up the automation end up being stored together with the code, so you store the code that’s being tested, the code that we use for writing the test cases, and all other assets so I can version all that,” said Lo Giudice. “If I find a bug and need to review and update the test automation, I can do that very quickly, so in the technology stack, CI/CD integration with application life-cycle management remains fundamental.”

For advanced performance testing, test data management is recommended.

“You can’t use the old way of doing test data generation when we’re cycling fast on testing continuously,” said Lo Giudice. “You have to have something that integrates into the sprint or the life cycle and updates the data all the way through.”

Self-service provisioning of test environments is also essential. That’s accomplished in the cloud, spinning up and spinning down test environments.

Expect AI and machine learning to impact vendor rankings
At the time of this writing, Forrester is about to release its Forrester Wave on Continuous Test Automation. Of the 26 criteria used in the wave, more than half of criteria (15 or 16) focus on functionality.

“A very large portion of those had a question around how and why are you using any ML or AI in this capability,” said Lo Giudice. “The response was the vendors have finally moved onto this, so this year you’re already seeing the use of AI in the tools and the way they’re using it. They’re using it to make things smarter.”

How, exactly, will be covered in the August issue of SD Times, which will include a deep dive on machine learning and AI in the context of CT.

The post Forrester’s recommendations for building a successful continuous testing capability appeared first on SD Times.

Forrester Research VP and principal analyst Diego Lo Giudice estimates that only 20% to 25% of organizations are doing continuous testing (CT) at this time, and even their teams may not have attained the level of automation they want.

“I have very large U.S. organizations saying, ‘We’re doing continuous delivery, we’ve automated unit testing, we’ve automated functional testing, we shifted those parts of the testing to the left, but we can’t leave performance testing to the end because it breaks the cycle,” said Lo Giudice.

The entire point of shifting left is to minimize the number of bugs that flow through to QA and production. However, achieving that is not just a matter of developers doing more types of tests. It’s also about benefiting from testers’ expertise throughout the life cycle.

“The old way of doing QA is broken and ineffective. They simply focus on quality control, which is just detecting bugs after they’ve already been written. That’s not good enough and it’s too late. You must focus on preventing defects,” said Tim Harrison, VP of QA Services at software quality assurance consultancy  SQA².  “QA 2.0 extends beyond quality control and into seven other areas: requirements quality, design quality, code quality, process quality, infrastructure quality, domain knowledge and resource management.”

What’s holding companies back
Achieving CT is a matter of people, processes and technology. While some teams developing new applications have the benefit of baking CT in from the beginning, teams in a state of transition may struggle with change management issues.

“Unfortunately, a lot of organizations that hire their QA directly don’t invest in them. Whatever experience and skills they’re gaining is whatever they happen to come across in the regular course of business,” said SQA2‘s Harrison.

Companies tend to invest more heavily in development talent and training than testing. Yet, application quality is also a competitive issue.

“Testing has to become more of the stewardship that involves broader accountability and broader responsibility, so it’s not just the testers or the quality center, or the test center, but also a goal in the teams,” said Forrester’s Lo Giudice.

Also holding companies back are legacy systems and their associated technical debt.

“If you’ve got a legacy application and let’s say there are 100 or more test cases that you run on that application, just in terms of doing regression testing, you’ve got to take all those test cases, automate them and then as you do future releases, you need to build the test cases for the new functionality or enhancements,” said Alan Zucker, founding principal of project management consultancy Project Management Essentials. “If the test cases that you wrote for the prior version of the application now are changed because we’ve modified something, you need to keep that stuff current.”

Perhaps the biggest obstacle to achieving CT is the unwillingness of some team members to adapt to change because they’re comfortable with the status quo. However, as Forrester’s Lo Giudice and some of his colleagues warn in a recent report, “Traditional software testing has no place in modern app delivery.”

Deliver value faster to customers
CT accelerates software delivery because code is no longer bouncing back and forth between developers and testers. Instead, team members are working together to facilitate faster processes by eliminating traditional cross-functional friction and automating more of the pipeline.

Manish Mathuria, founder and COO of digital engineering services company Infostretch, said that engineering teams benefit from instant feedback on code and functional quality, greater productivity and higher velocity, metrics that measure team and deployment effectiveness, and increased confidence about application quality at any point in time.

The faster internal cycles coupled with a relentless software quality focus translate to faster and greater value delivery to customers.

“We think QA should be embedded with a team, being part of the ceremony for Agile and Scrum, being part of planning, asking questions and getting clarification,” said SQA2‘s Harrison. “It’s critical for QA to be involved from the beginning and providing that valuable feedback because it prevents bugs down the line.”

Automation plays a bigger role
Testing teams have been automating tests for decades, but the digital era requires even more automation to ensure faster release cycles without sacrificing application quality.

“It takes time to invest in it, but [automation] reduces costs because as you go through the various cycles, being promoted from dev to QA to staging to prod, rather than having to run those regression cycles manually, which can be very expensive, you can invest in some man-hours in automation and then just run the automation scripts,” said SQA2‘s Harrison. “It’s definitely super valuable not just for the immediate cycle but for down the road. You have to know that a feature doesn’t just work well now but also in the future as you change other areas of functionality.”

However, one cannot just “set and forget” test automation, especially given the dynamic nature of modern applications. Quite often, organizations find that pass rates degrade over time, and if corrective action isn’t taken, the pass rate eventually becomes unacceptable.

To avoid that, SQA2 has a process it calls “behavior-based testing,” or BBT, which is kind of like behavior-driven development (BDD) but focused on quality assurance. It’s a way of developing test cases that ensures comprehensive quantitative coverage of requirements. If a requirement is included in a Gherkin-type test base, the different permutations of test cases can be extrapolated out. For example, to test a log-in form, one must test for combinations of valid and invalid username, valid and invalid password, and user submissions of valid and/or invalid data.

“Once you have this set up, you’re able to have a living document of test cases and this enables you to be very quick and Agile as things change in the application,” said SQA2‘s Harrison. “This also then leads to automation because you can draw up automation directly from these contexts, events, and outcomes.”

If something needed to be added to the fictional log-in form mentioned above, one could simply add another context within the given statement and then write a small code snippet that automates that portion. All the test cases in automation get updated with the new addition, which simplifies automation maintenance.

“QA is not falling behind because they’re actually able to keep up with the pace of development and provide that automation on a continuous basis while keeping the pass rates high,” said Harrison.

Service virtualization saves time
Service virtualization is another speed enhancer because one no longer waits for resources to be provisioned or competes with other teams for access to resources. One can simply mock up what’s needed in a service virtualization tool.

“I remember working on a critical application one time where everything had gone great in test and then when we moved the application changes to prod, things ground to a halt because the configurations in the upper and lower environment differed,” said Project Management Essential’s Zucker. “With service virtualization that goes away.”
Within the context of CT, service virtualization can kick off automatically, triggered by a developer pushing a feature out to a branch.

“If you’re doing some integration testing on a feature and you change something in the API, you’re able to know that a new bug is affected by the feature change that was submitted. It makes testing both faster and more reliable,” said SQA2’s Harrison. “You’re able to pinpoint where the problems are, understand they are affected by the new feature, and be able to give that feedback to developers much quicker.”

Infostretch’s Mathuria considers service virtualization a “key requirement.”

“Service virtualization plays a key role in eliminating the direct dependency and helps the team members move forward with their tasks,” said Mathuria. “Software automation engineers start the process of automation of the application by mocking the back-end systems whether UI, API, end points or database interaction. Service virtualization also automates some of the edge scenarios.”

AI and machine learning are the future
Vendors have already started embedding AI and machine learning into their products in order to facilitate more effective continuous testing and to speed application delivery cycles even faster. The greatest value comes from the pattern recognition pinpointing problem areas and providing recommendations for improving testing effectiveness and efficiency.

For example, Infostretch’s Mathuria has observed that AI and machine learning help with test optimization, recommendations on reusability of the code base and test execution analysis.

“As the test suites are increasing day by day, it is important to achieve the right level of coverage with a minimum regression suite, so it’s very critical to ensure that there are no redundant test scenarios,” said Mathuria of test optimization.
Since test execution produces a large set of log files, AI and machine learning can be used to analyze them and make sense out of the different logs. Mathuria said this helps with error categorization, setup and configuration issues, recommendations and deducing any specific patterns.

SQA2’s Harrison has been impressed with webpage structure analysis capabilities that learn a website and can detect a breaking change versus an intended change. However, he warned if XPaths have been used, such as to refer to a button that has just moved, the tool may automatically update the automation based on the change, creating more brittle XPaths than were intended.

The use cases for AI and machine learning are virtually limitless, but they are not a wholesale replacement for quality control personnel. They’re “assistive” capabilities that help minimize speed-quality tradeoffs.

The post Continuous testing isn’t optional anymore appeared first on SD Times.

According to Sridharan, an event log is a record of events that contains both a timestamp and payload of content. Event logs come in three forms:

• Plaintext: A log record stored in plaintext is the most commonly used type of log
• Structured: A log record that is typically stored in JSON format and highly advocated for as the form to use
• Binary: Examples of binary event logs include Protobuf formatted logs, MySQL binlogs, systemd journal logs, etc.

RELATED CONTENT:
A practical guide to observability for developers
Observability: It’s all about the data
Monitoring applications in modern software architectures

Logs can be useful in identifying unpredictable behavior in a system. Sridharan explained that often distributed systems experience failures not because of one specific event happening, but because of a series of possible triggers. In order to pin down the cause of an event, operations teams need to be able start with a symptom pinpointed by a metric or log, infer the life cycle of a request across various system components, and iteratively ask questions about interactions between parts of that system. 

Logs are the base of the three pillars, and both metrics and traces are built on top of them, Sridharan wrote. 

Sridharan defines metrics as numeric representations of data measured across time intervals. They are useful in observability because they can be used by machine learning algorithms to gain insights on the behavior of a system over time. According to Sridharan, their numerical nature also allows for longer retention of data and easier querying, making them well suited for building dashboards that show historical trends. 

Traces are the final pillar of observability. According to Sridharan, a trace is “a representation of a series of causally related distributed events that encode the end-to-end request flow through a distributed system.” They can provide visibility into the path that a request took and the structure of that request. Traces can help uncover the unintentional effects of a request, making them particularly well-suited for complex environments, like microservices

The post Three pillars of observability appeared first on SD Times.

But if you’re a developer, and you’re writing code to deliver to your customers in the form of an application or a service, you’d likely want to know after you deliver it that it’s working the way you intended.

RELATED CONTENT:
Observability: It’s all about the data
Monitoring applications in modern software architectures

This engineering-centric view of performance management has taken on the name “application stability management.” James Smith, co-founder of ASM solution provider Bugsnag, said his company and another, Sentry, are the first two to raise the banner for application stability.

So what’s the real difference between APM and ASM? Smith explained: “There’s this big gap in the APM space — figuring out when to promote builds from data to production, figuring out when to roll out an A/B test from 5% to 100%. You need to know when you’re making these rapid iterative changes, ‘Are the changes we’re delivering actually working?’ And this is just not something that the APM providers are thinking about. It’s an afterthought for them.”

It’s this focus on this persona of product and engineering teams that is making a difference. Smith said that when used alongside a traditional APM solution, his company found that less than 5% of the engineers were logging into the APM, while 70% of the engineering team was logging into Bugsnag on a weekly basis. “That’s meant that we’ve built what essentially is a daily dashboard for the engineering and product teams,” Smith said, “instead of waiting from the monitoring team to tell the software engineer that he screwed up and needs to fix it.  It’s a tool those people are using every day to hone their craft and get better at being a software engineer.”

Large enterprises today are realizing that their brand impression comes more from the web and mobile experiences than it does from their stores or offices. So focusing on the customer experience first, Smith said client-side monitoring — JavaScript and mobile monitoring — is where “the rubber meets the road when it comes to customers touching your software.” 

The post Application performance management vs. application stability management appeared first on SD Times.

While being able to have this view is important, handling all the data these systems throw off can be a huge challenge for organizations. In terms of observability, the three pillars of performance data are logs (for recording events), metrics (what data you decide gives you the most important measures of performance) and traces (views into how software is performing).

Those data sources are important, but if that is where you stop in terms of what you do with the data, your organization is being passive and not proactive. All you’ve done is collect data. According to Gartner research director Charley Rich,  “We think the definition of observability should be expanded in a couple of ways. Certainly, that’s the data you need — logs, metrics and traces. But all of this needs to be placed and correlated into a topology so that we see the relationships between everything, because that’s how you know if it can impact something else.” 

RELATED CONTENT: 
A practical guide to observability for developers
Monitoring applications in modern software architectures

Bob Friday, who leads the AIOps working group at the Open Networking User Group (ONUG) and is CTO at wireless network provider Mist Systems (a Juniper Networks company), said from a network perspective, it’s important to start with the question, “Why is the user having a problem?” and work back from that. That, he said, all starts with the data. “I would say the fundamental change I’ve seen from 15 years ago, when we were in the game of helping enterprises deal with network stuff, is that this time around, the paradigm is we’re trying to manage end-to-end user experience. [Customers] really don’t care if it’s a Juniper box or a Cisco box.”

Part of this need is driven by software development, which has taken services and distributed deployment environments to a whole other level, by deploying more frequently and achieving higher engineering productivity.  And, as things speed up, performance and availability management become more critical than ever. “Infrastructure and ops, these app support teams, have to understand that if more applications are coming out of the factory, we better move fast,” said Stephen Elliot, program vice president for I&O at analysis firm IDC. “The key thing is recognizing what type of analytics are the proper ones to the different data sets; what kinds of answers do they want to get out of these analytics.” 

But with that, it’s very important to recognize what type of analytics are the proper ones to the different data sets; what kinds of answers do organizations want to get out of these analytics. 

Elliot explained that enterprises today understand the value of monitoring. “Enterprises are beginning to recognize that with the vast amount of different types of data sources, you sort of have to have [monitoring],” he said. “You have more complexity in the system, in the environment, and what remains is the need for performance availability capabilities. In production, this has been a theme for 20 years. This is a need-to-have, not a nice-to-have.”

Not only are there now different data sources, it’s the type of data being collected that has changed how organizations collect, analyze and act on data. “The big change that happened in data for me from 15 years ago, where we were collecting stats every minute or so, to now, we’re collecting synchronous data as well as asynchronous user state data,” Friday said. “Instead of collecting the status of the box, we’re collecting in-state user data. That’s the beginning of the thing.”

Analyzing that data
To make the data streaming into organizations actionable, graphical data virtualization and visualization is key, according to Joe Butson, co-founder of Big Deal Digital, a consulting firm. “Virtualization,” he said, “has done two things: It’s made it more accessible for those people who are not as well-versed in the information they’re looking at. So the virtualization, when it’s graphical, you can see when performance is going down and you have traffic that’s going up because you can see it on the graph instead of cogitating through numbers. The visualization really aids understanding, leading to deeper knowledge and deeper insights, because in moving from a reactive culture in application monitoring or end-to-end life cycle monitoring, you’ll see patterns over time and you’ll be able to act proactively. 

RELATED CONTENT: 
APM: What it means in today’s complex software world
APM, AIOps and Observability

“For instance,” he continued, “if you have a modern e-commerce site, when users are spiking at a certain period that you don’t expect, you’re outside of the holiday season, then you can then look over, ‘Are we spinning up the resources we need to manage that spike?’ It’s easy when you can look at a visual tool and understand that versus going to a command-line environment and query what’s going on and pull back information from a log.”

Another benefit of data virtualization is the ability to view data from multiple sources in the virtualization layer, without having to move the data. This helps everyone who needs to view data stay in sync, as there’s but one version of truth. This also means organizations don’t have to move data into big data lakes. 

When it comes to data, Mist’s Friday said, “A lot of businesses are doing the same thing. They first of all go to Splunk, and they spend a year just trying to get the data into some bucket they can do something with. At ONUG  we’re trying to reverse that. We say, ‘Start with the question,’ figure out what question you’re trying to answer, and then figure out what data you need to answer that question. So, don’t worry about bringing the data into a data lake. Leave the data where it’s at, we will put a virtualized layer across your vendors that have your data, and most of it is in the cloud. So, you virtualize the data and pull out what you need. Don’t waste your time collecting a bunch of data that isn’t going to do you any good.”

Because data is coming from so many different sources and needs to be understood and acted on by many different roles inside a company, some of those organizations are building multiple monitoring teams, designed to take out just the data that’s relevant to their role, and presented in a way they can understand.

Friday said, “If you look at data scientists, they’re the guys who are trying to get the insights. If you have a data science guy trying to get the insight, you need to surround him with about  four other support people. There needs to be a data engineering guy who’s going to build the real-time path. There has to be a team of guys to get the data from a sensor to the cloud. That’s the shift we’re seeing to get insights from real-time monitoring. How you get the data from the sensor to the cloud is changing… Once you have the data to the cloud, there needs to be a team of guys — this is like Spark, Flink, Storm — to set up real-time data pipelines, and that’s relatively new technology. How do we process data in real time once we get it to the cloud?” 

AI and ML for data science 
The use of artificial intelligence and machine learning can help with things like anomaly detection, event correlation and remediation, and APM vendors are starting to build those features into their solutions. 

AI and ML are starting to provide more human-like insights into data, and deep learning networks are playing an important role in reducing false positives to a point where network engineers can use the data.

But Gartner’s Rich pointed out that all of this activity has to be related to the digital impact on the business. Observing performance is one thing, but if something goes wrong, you need to understand what it impacts, and Rich said you need to see the causal chain to understand the event.  “Putting that together, I have a better understanding of observation. Adding in machine learning to that, I can then analyze, ‘will it impact,’ and now we’re in the future of digital business.”

Beyond that, organizations want to be able to find out what the “unknown unknowns” are. Rich said a true observability solution would have all of those capabilities — AI, ML, digital business impact and querying the system for the unknown unknowns. “For the most part, most of the talk about it has been a marketing term used by younger vendors to differentiate themselves and say the older vendors don’t have this and you should buy us. But in truth, nobody fully delivers what I just described, so it’s much more aspirational in terms of reality. Certainly, a worthwhile thing, but all of the APM solutions are all messaging how they’re delivering this, whether they’re a startup from a year ago or one that’s been around for 10 years. They’re all making efforts to do that, to varying degrees.” 

With Jenna Sargent

The post Observability: It’s all about the data appeared first on SD Times.

That, combined with more complex application architectures (APIs, multiple data sources, multicloud distributions and more), and the ephemeral nature of software as temporary and constantly changing, is also changing the world of monitoring and creating a need for observability solutions.

First-generation application monitoring solutions struggle to provide the same level of visibility into today’s more virtual applications – i.e., containerized and/or orchestrated environments running Docker and Kubernetes. Massively distributed microservices-based applications create different visibility issues for legacy tools. Of course, application monitoring is still important, which has driven the need to add observability into the applications running in those environments. 

While legacy application monitoring tools have deep visibility into Java and .NET code, new tools are emerging that are focused on modern application and infrastructure stacks. According to Chris Farrell, technical director and APM strategist at monitoring solution provider Instana, one of the important things about a microservice monitoring tool is that it has to recognize and support all the different microservices. “I think of it like a giant T where the vertical bar represents visibility depth and the horizontal bar represents visibility breadth,” he explained. “Legacy APM tools do great on the vertical line with deep visibility for code they support;  meanwhile, microservices tools do well on the horizontal line, supporting a broad range of different technologies. Here’s the thing – being good on one axis doesn’t necessarily translate to value along the other because their data model is built a certain way. When I hear microservices APM, I think, ‘That’s what we do.’ [Instana has] both the depth of code-level visibility and the breadth of microservices support because that’s what we set out to do, solve the problem of ephemeral, dynamic, complex systems built around microservices.”

When talking about observability and application monitoring, it’s important to think about the different kinds of IT operations individuals and teams you have to deal with. According to Farrell “whether you’re talking about SREs, DevOps engineers or traditional IT operators, each has their own specific goals and data needs. Ultimately, it’s why a monitoring solution has to be flexible in what data it gathers and how it presents that data. 

Even though it’s important for modern monitoring solutions to recognize and understand complexity, it’s not enough. They must also do so programmatically, Farrell said, because today’s systems are simply too complex for a person to understand. “You add in the ephemeral or dynamic aspect, and by the time a person could actually create a map or understand how things are related, something will change, and your knowledge will be obsolete,” he said.

Modern solutions also have to be able to spot problems and deliver data in context. Context is why it’s practically impossible for even a very good and knowledgeable operations team to understand exactly everything that’s going on inside their application themselves. This is where solutions that support both proprietary automatic visibility and manually injected instrumentation can be valuable. Even if you have the ability to instrument an application with an automated solution, there still is room for an observability piece to add some context. “Maybe it’s a parameter that was passed in; maybe it’s something to do with the specific code that the developer needs to understand the performance of their particular piece of code,” Farrell said of the need for contextual understanding.

“That’s why a good modern monitoring tool will have its own metrics and have the ability to bring in metrics from observability solutions like OpenTracing, for example,” Farrell added. “Tracing is where a lot of this nice context comes out.  Like Instana, it’s important to have the ability to do both. That way, you provide the best of both worlds.”

To make the ongoing decisions and take the right actions to deliver proper service performance, modern IT operations teams really require that deep context. It’s valuable for ongoing monitoring, deployment or rollback verification, troubleshooting and reporting. While observability on its own can provide information to an individual or a few individuals. It is the monitoring tool that provides understanding into how things work together; that can shift between a user-centric or an application-centric view, and that can give you a framework to move from monitoring to decision-making to troubleshooting and then, when necessary, moving into reporting or even log analysis.

Farrell pointed out that “the APM piece is the part that ties it all together to provide that full contextual visibility that starts with individual component visibility and ultimately ties it all together for application-level performance and service-level performance.”

The post Monitoring applications in modern software architectures appeared first on SD Times.