What is DevOps?

The software development has changed drastically over the last few years. DevOps that largely covers Continuous Integration and Continuous Deployment capability, commonly known as CI/CD has become a key component of the software projects.

Continuous deployment is what delivers the most value to the business and stakeholders and makes DevOps even more important. Instead of releasing application updates every 1-6 months, DevOps teams now deploy small, incremental changes more frequently, sometimes even daily.

Continuous integration refers to development practice that requires developers working in a team to merge and integrate their code on frequent intervals. This merging is followed by the build process and deployment. This practice has proven to be useful over the period because of it’s early warning and error detection. It is a perfect solution to the pressures faced by the development teams at the time of their production release.

DevOps is a natural extension of Agile, bringing Dev, Ops, and other IT functions together, enabling the business to create a wider, cross-functional team that can deliver without barriers and achieve greater velocity.

Veracode and the Team

Veracode delivers the application security solutions and services, today’s software-driven world requires. Veracode’s unified platform assesses and improves the security of applications from initiation through production so that businesses can confidently innovate with the web and mobile applications they build or buy.

While doing so they generate a bulk of historical data that they want to consume, and gain useful insights to be more beneficial to their end clients. Hence there is a need to build a strong serverless Data Lake with strong Infrastructure Automation and Continuous Integration and Deployment capability.

The Northbay team was brought on board at the very start to analyse potential options and to explain how it may work. When working with large clients, you don’t always get the chance to start at the very beginning of a project, with the freedom to do it right the first time around. Getting the chance to do it with this engagement has been one of the most exciting things about this project!

The Project So Far

Northbay team proposed following architecture of the data lake. It heavily relies on AWS platform and services like S3 and Glue consist the backbone of this solution. The DevOps component comprised of:

Infrastructure Provisioning
Continuous Integration/Continuous Deployments

Infrastructure Provisioning

Veracode Team not only automated the infrastructure provisioning using terraform, but we also automated the deployments of infrastructure using Gitlab. We built an immutable infrastructure covering

  • ECS cluster with scheduled tasks
  • Aurora MySQL cluster
  • S3 bucket with policies
  • SQS with triggers from s3 buckets
  • Roles and policies used by Glue and other components
  • SSM parameters
  • Glue connections
  • Glue jobs and triggers

Thus overall with each improvement in infrastructure, a built was ready to be deployed at the cost of a code push. The CICD pipeline not only deployed but further validated the infrastructure as a part of its process.

Veracode – DevOps

Continuous Integration/Continuous Deployments

Along with the fast pace of infrastructure deployments, team needed an equally fast deployment pipeline which would cater to the changing needs of the project. As quality is our prime concern, our project evolved over time and with each improvement there needed to be an automated deployment to let the team focus on more important tasks.

The CI/CD pipeline not only handled code deployments but it also included an automated QA framework, that would test the project for vulnerabilities. Eventually if there were some issues with build or there were some failed check whole pipeline would fail, notifying the concerned team of developers and stakeholders.

To achieve this, a fully managed process was implemented that included push-button builds of entire environments to a full pipeline deployment and validation from commit to Production. Following tools set was used.

  • Gitlab
  • Terraform
  • ECS Containers

Following diagram explains the CI/CD process life cycle:

Veracode – DevOps

Following diagram explains the application code deployment process:

Veracode – DevOps

Focusing on automating everything is indeed a slower process at the start, and sometimes that can be a struggle to show results when you’re still building the foundations. However, today the client thinks nothing of building and destroying environments multiple times in a day, or teams controlling what they develop and deploy at a pace that’s right for them, and all done without our intervention. This automation allows our team to focus on refining and optimising the platform and start on work for new features and requirements.

Working Remotely

The team works remotely using Zoom and Slack to pair on work, discuss and design implementations, as well as constantly communicating with the client. Even when one or two of us are in the office, we follow the principal of remote-first to ensure we always include everyone as part of the team.

About NorthBay

We are a fast-growing, 100% AWS focused onshore/offshore AWS Premier Consulting Partner, supporting our customers to accelerate the reinvention of their applications and data for a Cloud-native world. Our >350 AWS Certified Employees excel in developing and deploying Database & Application Migrations, Data Lakes and Analytics, Machine Learning/AI, DevOps and Application and Data Modernization/Development that drive measurable business impact.