What is DevOps?
The software development has changed drastically over the last few years. DevOps that largely covers Continuous Integration and Continuous Deployment capability, commonly known as CICD has become a key component of the software projects.
Continuous deployment is what delivers the most value to the business and stakeholders and makes DevOps even more important. Instead of releasing application updates every 1-6 months, DevOps teams now deploy small, incremental changes more frequently, sometimes even daily.
Continuous integration refers to development practice that requires developers working in a team to merge and integrate their code on frequent intervals. This merging is followed by the build process and deployment. This practice has proven to be useful over the period because of it’s an early warning and error detection. It is a perfect solution to the pressures faced by the development teams at the time of their production release.
DevOps is a natural extension of Agile, bringing Dev, Ops, and other IT functions together, enabling the business to create a wider, cross-functional team that can deliver without barriers and achieve greater velocity.
Veracode and the Team
Veracode delivers the application security solutions and services, today’s software-driven world requires. Veracode’s unified platform assesses and improves the security of applications from initiation through production so that businesses can confidently innovate with the web and mobile applications they build or buy. While doing so they generate bulk of historical data that they want to consume, and gain useful insights to be more beneficial to their end clients. Hence there is a need to build a strong serverless Data Lake with strong Infrastructure Automation and Continuous Integration and Deployment capability.
The Northbay team was brought on board at the very start to analyze potential options and to explain how it may work. When working with large clients, you don’t always get the chance to start at the very beginning of a project, with the freedom to do it right the first time around. Getting the chance to do it with this engagement has been one of the most exciting things about this project!
Northbay team crafted the following architecture of the data lake. It heavily relies on AWS platform and services like S3 and Glue consist of the backbone of this solution.
The DevOps component comprised of
- Infrastructure Provisioning
- Continuous Integration/Continuous Deployments
The Veracode Team not only automated the infrastructure provisioning using terraform, but we also automated the deployments of infrastructure using Gitlab.
We built an immutable infrastructure covering
- ECS cluster with scheduled tasks
- Aurora MySQL cluster
- S3 bucket with policies
- SQS with triggers from s3 buckets
- Roles and policies used by Glue and other components
- SSM parameters
- Glue connections
- Glue jobs and triggers
Thus overall with each improvement in infrastructure, a built was ready to be deployed at the cost of a code push. The CICD pipeline not only deployed but further validated the infrastructure as a part of its process.
Continuous Integration/Continuous Deployments
Along with the fast pace of infrastructure deployments, team needed an equally fast deployment pipeline which would cater to the changing needs of the project. As quality is our prime concern, our project evolved over time and with each improvement there needed to be an automated deployment to let the team focus on more important tasks.
The CI/CD pipeline not only handled code deployments but it also included an automated QA framework, that would test the project for vulnerabilities. Eventually, if there were some issues with build or there were some failed check whole pipeline would fail, notifying the concerned team of developers and stakeholders.
Everything is fully managed in Git and made using Terraform; from push-button builds of entire environments to a full pipeline deploying and validating from commit to Production.
Focusing on automating everything is indeed a slower process at the start, and sometimes that can be a struggle to show results when you’re still building the foundations. However, today the client thinks nothing of building and destroying environments multiple times in a day, or teams controlling what they develop and deploy at a pace that’s right for them, and all done without our intervention. This automation allows our team to focus on refining and optimizing the platform and start on work for new features and requirements.
As a result of the deployment of the Veracode Global Data Lake Veracode is now able to provide both internal and external detailed analytics. Internally, the Veracode team leverages Looker to gain insights into a variety of improved analytics KPIs that are used to feed back into the Veracode product development feature loop with the expectation that this will yield new product offerings, increased customer retention increased product usage and increased revenues. Externally, Veracode customers are now able to have more granular insights into their specific product outcomes, risks and remediation strategies. Veracode beleiees that this will enhance product stickiness and increased recurring revenues as a value add.
The team works remotely using Zoom and Slack to pair on work, discuss and design implementations, as well as constantly communicating with the client. Even when one or two of us are in the office, we follow the principle of remote-first to ensure we always include everyone as part of the team.