Share
Organizations obtain net business outcome benefits by focusing on improving their ability to develop and use software and infrastructure effectively. Organizations achieve these benefits through reducing friction between development, operations, and security. Incorporating the responsibilities of each into a single, shared practice is known as DevSecOps. Organizations can improve their DevSecOps practices and achieve even greater business benefits by putting into action the following processes Xebia has implemented or observed while working with our clients: documentation, deployment, reporting, developer relations, observability, and security practices.
Documentation
A cohesive, coherent documentation strategy designed by developers and operations promotes visibility, transparency, collaboration, and compliance across disciplinary and other intra-organizational boundaries. Documentation should be up-to-date. Documentation should provide details on:
- organizational processes
- infrastructure designs and deployments
- applications
- systems design
Good documentation promotes trust and a sense of safety to the organization.
Merely having documentation isn’t a sufficient best practice. Distributed, compartmentalized, conflicting, or non extant documentation leads to process duplication and/or incoherent information and processes that become unhelpful and unproductive time sinks spanning multiple teams. Centralizing distributed documentation into a single repository of clear taxonomies and architecture takes extra effort, but eliminating the time sinks is worth it.
Ease of use, advanced search capabilities, collaborative tools, notifications, and quick review tools are key elements of implementing a great documentation strategy. Centralized wikis provide these elements through software and tooling features. We recommend their use. We also recommend a single documentation maintenance team. This team is responsible for maintaining organizational documentation. They are also responsible for providing tools for collaborating and capturing all the documentation produced by other organizational teams.
Developers and operations work together to ensure individual application READMEs, documentation as code, onboarding, architectural decisions, environment configuration, and deployment follow the organizational procedures and infrastructure documented within the centralized wiki.
Documentation is usually owned by a team responsible for maintaining its high-quality, and, at the same time, facilitating collaboration without bureaucracy. Many have opted to use a centralized wiki for ease of use, advanced search capabilities, collaboration mechanisms, and quick review/notification processes.
A lack of documentation (also documents with no content, or when it is stuck in one person’s head!) is probably the worst scenario, and hopefully, inconceivable to most organizations. However, having multiple sources of documentation with conflicting instructions, no clear owners, or multiple flavors of the same process (everyone has their own favorite cookie recipe) isn’t much better. Combined, these kinds of scenarios are generally unhelpful and unproductive time sinks. Teams have to make an effort to avoid duplication and channel all documentation efforts to a single repository with clear taxonomies and information architecture.
Communication is fluent, focused on collaborative troubleshooting, remediation, and prevention.
Continuous Deployment
Continuous deployment is essential to minimize the consequences of failure and to build resilient, trustworthy software systems.
Formal change requests and other cumbersome approval procedures spark negative business unit and developer creativity. This spawns
- abuse of processes
- circumvention of security or regulatory procedures
- fragile silos
- bottlenecks
- temporary or external solutions
Negative creativity increases overall organizational risk. We should not rely on cumbersome approval processes to foster confidence and trust in our software development and deployment practices.
Successful organizations build trust in these processes and minimize the consequences of failed releases through the following recommended practices:
- comprehensive automated testing
- continuous, minimal, frequent, and revertible releasing
- testing new releases with a limited number of users (Canary release)
- automated rollbacks
- maintaining high deployment environmental parity
- advanced monitoring techniques such as Real User Monitoring (RUM) and synthetic monitoring
Developer Experience and Automated Provisioning
Infrastructure and application provisioning, configuration management, and deployment must be automated and scalable: from local development environments to multi-cloud deployments. This facilitates:
- integration testing
- iterative infrastructure deployment development
- iterative management strategy development
This is a key component in Developer Experience. We want to enable them to do their jobs seamlessly in all environments with as little friction as possible.
These capabilities encourage:
- modern infrastructures
- small sets of cloud-native components
- vendor lock-in resistance
- resilient disaster recovery
Organizations practicing this are more efficient than organizations that forgo this practice for speed of delivery or cost optimizations. We recommend automated, version controlled, infrastructure provisioning, configuration management and deployment.
Environments for QA, Staging, and Production
Organizations that test and develop applications in environments equivalent to their production environments reduce the total number of production incidents. Incident and error mitigation and correction is easier to do when developers can replicate the production environment on their local development machine.
Modern applications deal with massive amounts of data often derived from production users. Testing requires lower environments to have identical data infrastructure and similar data volume and quality.
We recommend automating the environment using Infrastructure as Code (IaC) and data generators and anonymization processes; and applying SecDevOps practices, technologies, and methodologies to enable environment equivalency.
Observability
Organizations that detect issues before users are impacted avoid alert and help-desk service fatigue.
Every element in the environment is:
- instrumented
- traced
- logged
- measured
to produce actionable insights and alerts with attached remediation plans.
Alerts detected from these insights are routed to the appropriate team that uses organization- or application-, documentation to address issues. Teams can develop, test, and deploy a fix release within a few minutes. We recommend limiting alerts to actionable insights, and remediation or fixes that will prevent the issue from reoccurring.
We recommend that developers have access to the monitoring information they need in every environment with the corresponding production data compliance and privacy controls.
We recommend that logging, tracing, monitoring, and alerting is done in every environment.
Security and up-to-date dependencies
Organizations building secure software incorporate automated security scans and tests into their deployment pipelines. The vast majority of their code-base is made of open-source or other third party dependencies; and the more external dependencies a code-base has, the more well-known attack vectors there will be. We therefore recommend automated dependency upgrades and application security scanning.
Self Service
Version controlled Infrastructure As Code allows changes to cloud environments and other infrastructure needs to be quickly reviewed and approved. CI/CD pipelines integrate code and IaC changes into immediate deployment with minimal manual intervention or approval.
Conclusion
Incorporating DevSecOps practices, technologies, and methodologies into the development process enables organizations to innovate faster, reduce costs by automating manual processes and reducing errors, and retain happier engineers!