There’s considerable confusion about “the best way” to approach CI/CD when no single path exists. There are important considerations organizations should contemplate to avoid wasting time and money that could have been spent making progress, however.

“One of the first things an organization should do is understand what their needs are [in terms of] the business, their application and the application environment,” said Alan Zucker, founding principal of training and consulting firm Project Management Essentials. “The other thing you should do when looking at your CI/CD pipeline needs is figure out what you’re going to get the biggest benefit from, your automation or integration work or where you pain points are. Then start stitching things together.”

RELATED CONTENT:
Add Static Code Analysis to Your CI/CD Pipelines
CI/CD pipelines are expanding
How some organizations are implementing CI/CD

Clearly, progress is always the goal, but sometimes organizations come up short. Following are some of the “gotchas” that can get in the way.

Complexity
CI/CD tool chains tend to be complex because CI/CD involves so many processes and associated types of tools. The resulting complexity can be difficult, time-consuming and expensive to maintain over time.

“Now you’ve got a whole group of people whose only job is to maintain the tool chain, which was supposed to keep you from having all these people in your organization,” said Tom Petrocelli, research fellow for DevOps and Collaboration at Amalgam Insights. “If you’re a midsize company, you’re not going to have the budget to hire all the engineers to do that and maintain those engineers over time. The result is modest tool chains that don’t have the impact the organization expected or the cost prevents them from doing anything.”

The emerging end-to-end solutions alleviate the tool management overhead since all elements are integrated. However, highly complex applications running in highly complex environments may require point solutions that have a greater depth of features.

“I think a successful pipeline really simplifies complex operations. Your pipeline may grow, so it becomes more complex in all of the different aspects of what it’s doing, but it needs to be simple, straightforward and easy to work with,” said Daniel Ritchie, distinguished engineer at Broadridge Financial Solutions.

Security
Security needs to be part of a CI/CD pipeline; however, a lot of times “security” is limited to just a quick CVE scan. Alternatively, a security scan may be left until the end. If a vulnerability is found, developers may be more inclined to file an exception request so they can fix it in the next release rather than spending the time to fix it in the current release.

“Security should be everywhere, not just before or after integration. A lot of organizations think of security as just static analysis or dynamic analysis,” said Hasan Yasar, technical manager of the Secure Lifecycle Solutions group in the CERT Division of the Software Engineering Institute at Carnegie Mellon University.

“Everywhere” means various types of security testing throughout the SDLC, beginning with defining security requirements along with functional requirements. Yasar recommends including tools like Fortify or SonarQube as part of the CI server. Depending on the application, other security considerations may include authentication, network segmentations, regulatory compliance (such as PII usage) and the like, so security testing is happening at the code level and the architectural level.

“Almost every organization is struggling with false positives or how much security testing they need to do because static and dynamic analysis such as pen testing take time,” said Yasar. “If the organization is only doing static and dynamic analysis without tying them into the requirements in the beginning, they don’t know what they’re testing or they don’t understand the value of the testing they’ve done. Security should have threat modeling or understanding of security requirements at the beginning of feature development.”

Don’t think of security as a step, but rather a process that should be integrated throughout the pipeline.

Culture
Some people are really excited about CI/CD because they want to get more done faster, but not everyone. CI/CD automates a lot of manual tasks that have been associated with job descriptions.

“Organization and culture are the biggest barriers. You should have accountability at the lowest responsible level so people are taking ownership for their work and people have that end-to-end responsibility,” said Project Management Essentials’ Zucker. “[Companies are] trying to acquire new methodologies and processes and spend a lot of money training people in the same old tools, but then they still behave the same old way and they don’t get the results they want.”

Having the people affected by CI/CD involved in its planning, problem-solving and implementation can help break down barriers to adoption.

Unrealistic expectations
It’s easy to expect too much too soon. Buying tools alone won’t help, nor will automating processes that weren’t designed well in the first place. The space itself is evolving, and as a process, organizations are wise to approach CI/CD from the perspective of continuous improvement, since they’ll discover many things that could along the way anyway. Trying to do too much too quickly can backfire.

“I would not counsel anybody to go from a well-controlled annual process to push-button full release management. You’re talking about a several-year journey with a lot of fits and starts,” said Zucker.