Amway – AWS Database Migration

Amway – AWS Database Migration

  • NorthBay Blogs

Amway – AWS Database Migration

Amway

Amway

Amway is the world’s largest direct selling company ($8.8B reported sales, 2018). They manufacture and distribute nutrition, beauty, personal care and home products—which are exclusively sold in 100 countries through Amway Independent Business Owners (IBOs).

Challenges

  1. Data cannot be recent than a day and is a batch process
  2. Amway Business Owners (ABO’s) rely on Corporate rather than self-service to provide reporting and analytics
  3. Decommision Oracle Exadata

Solution Delivered to Meet Challenges

  1. Consolidate ~20 Terabytes of data for analytics from all regions into one platform for global use
  2. Implement a cloud-based data lake, enterprise data warehouse and data pipeline process
  3. Bulk ingestion of a large ~20 Terabytes of data from different databases and streaming sources to AWS RDS and S3
  4. Real-time change data capture (CDC) for all sources
  5. Ingested data must be validated and reconciled with source
  6. Transform ingested data as per defined business rules
  7. Enable data scientists to perform analytics on transformed data
  8. Secure data at Rest and in Transit
  9. System health monitoring

Architecture Diagram

AWS Services Used

  • AWS Aurora Postgres
  • AWS Redshift
  • AWS Simple Storage Service (S3)
  • AWS EC2
  • AWS Step Functions
  • AWS Lambda
  • AWS Kinesis
  • AWS Kinesis Firehose
  • AWS CloudWatch
  • AWS CodePipeline
  • AWS CodeBuild
  • AWS CloudFormation
  • AWS X-Ray

Database Workload Migration Details

  • Source DB: 2 On-Prem Oracle
  • Target DB: 1 AWS Aurora PostgreSQL
  • Number of Tables Migrated: 300+
  • Size of migrated data: ~10TB
  • Sources and Targets:
    • Custom Business events -> Kafka -> Kinesis -> Aurora
    • Custom Business events -> Kafka -> Kinesis -> S3
    • 1000’s of events per second

Migration Patterns Used

N/A

Migration Tools Used

  • Sqoop on EMR (Data Ingestion)
  • Spark on EMR (Data Validation & Standardization)

Third-Party Tools Used

  • Tibco DV
  • Grafana

About NorthBay – We are a fast-growing, 100% AWS focused onshore/offshore AWS Premier Consulting Partner, supporting our customers to accelerate the reinvention of their applications and data for a Cloud-native world. Our >350 AWS Certified Employees excel in developing and deploying Database & Application Migrations, Data Lakes and Analytics, Machine Learning/AI, DevOps and Application and Data Modernization/Development that drive measurable business impact.