Migration from IBM StreamSets to AWS for ELT

In the next few months, I’ll be shifting my staff over from StreamSets and BMC Control-M for ELT to exclusively AWS. The part where AWS feeds into Snowflake won’t change.

It’ll require some upskilling. I locked down AWS Skill Builder team license for the group to help get us over the hump. I’m going to take the DEA-02 Engineer Associate certification exam myself later this fall (or early winter, at worst).I’ve suggested my engineer and data scientist consider it also.

Overall, the key is going to be getting the hang of Glue and Glue Studio, and then branching into Athena, Quick, and Redshift. Once we’re there, I’d like to optimize the rest of AWS tools, so I’m promoting an exploration of GenAI and Quantum modules.

Why skip the foundations badges, you might ask? I passed the AWS Cloud Practitioner exam earlier in the year. It does help get a feel for the platform, but I don’t think it’s strictly necessary as a precursor to the Engineer Associate exam or for working with ELT tools.

Without further ado, here’s the draft plan I’m pushing out:

Data Analyst

Phase 1

• Data Analyst Learning Path (baseline)
  • EMR Getting Started
  • Glue Getting Started
  • Intro to Athena
  • Analyze Big Data with Hadoop (lab)
  • Google Ngrams with EMR/Hive (lab)
  • Redshift Getting Started + lab
  • Serverless Analytics
  • Building BI Dashboards with QuickSight (lab)
  • OpenSearch Getting Started

Phase 2

• Lab: Query Data with Amazon Athena
• Daily Batch Extraction (SimuLearn)
• Federated Queries (SimuLearn)
• Querying the Data Lake (SimuLearn)
• QuickSight Getting Started (course)
• Business Intelligence Dashboards (SimuLearn)
• Amazon QuickSight Advanced Business Intelligence Authoring (Part 1)

Data Engineer

Phase 1

• Data Analyst Learning Path (baseline)
  • EMR Getting Started
  • Glue Getting Started
  • Intro to Athena
  • Analyze Big Data with Hadoop (lab)
  • Google Ngrams with EMR/Hive (lab)
  • Redshift Getting Started + lab
  • Serverless Analytics
  • Building BI Dashboards with QuickSight (lab)
  • OpenSearch Getting Started
• Data Engineering on AWS
  • Foundations
  • Data Lake Solution
  • Data Warehouse Solution
  • Batch Data Pipeline Solution
  • Streaming Data Pipeline Solution

Phase 2

• Populating the Data Catalog (SimuLearn)
• Event-Driven Serverless ETL
• Event-Driven ETL Automation (SimuLearn)
• Securing the Data Lake (SimuLearn)
• Building Event-Driven Applications With Amazon EventBridge (Includes Labs)
• Exam Prep Plan: AWS Certified Data Engineer – Associate

Data Scientist

Phase 1

• Data Analyst Learning Path (baseline)
  • EMR Getting Started
  • Glue Getting Started
  • Intro to Athena
  • Analyze Big Data with Hadoop (lab)
  • Google Ngrams with EMR/Hive (lab)
  • Redshift Getting Started + lab
  • Serverless Analytics
  • Building BI Dashboards with QuickSight (lab)
  • OpenSearch Getting Started
• Data Engineering on AWS
  • Foundations
  • Data Lake Solution
  • Data Warehouse Solution
  • Batch Data Pipeline Solution
  • Streaming Data Pipeline Solution

Phase 2

• Generative AI Learning Plan for Developers:
  • Intro to GenAI
  • Planning a GenAI Project
  • Bedrock Getting Started
  • Foundations of Prompt Engineering
  • Exploring Nova Models
  • Building GenAI Applications Using Bedrock (labs)
  • Intro to SageMaker Notebooks
• Builder Labs:
  • Generative AI on AWS RAG x2
  • LangChain use cases
  • production-ready apps
  • Bedrock Agents/Guardrails
  • serverless Bedrock integration
  • Python SDK functions
  • Nova Micro/Canvas/Reel/Lite
  • Textract+Bedrock IDP
  • Lab Maker

You (CDO)

Phase 1

• Data Analyst Learning Path (baseline)
  • EMR Getting Started
  • Glue Getting Started
  • Intro to Athena
  • Analyze Big Data with Hadoop (lab)
  • Google Ngrams with EMR/Hive (lab)
  • Redshift Getting Started + lab
  • Serverless Analytics
  • Building BI Dashboards with QuickSight (lab)
  • OpenSearch Getting Started
• Data Engineering on AWS
  • Foundations
  • Data Lake Solution
  • Data Warehouse Solution
  • Batch Data Pipeline Solution
  • Streaming Data Pipeline Solution
• Exam Prep Plan: AWS Certified Data Engineer – Associate

Phase 2

• Quantum Researcher Learning Plan:
  • Intro to Building with AWS Databases
  • Braket Getting Started
  • Braket Quantum Application Development
  • Braket Knowledge Badge Readiness Path
  • Braket Pre-assessment
  • first quantum circuit lab
  • Knowledge Badge Assessment
• AWS Generative AI Developer Advanced Learning Plan
  • Requirements/Design
  • Foundation Model Selection
  • Data Pipelines
  • Vector Stores
  • Retrieval Mechanisms
  • Prompt Engineering Strategies
  • Agentic AI, Model Deployment
  • Enterprise Integration
  • API Integrations
  • App Integration Patterns
  • Safe Interactions
  • Data Security
  • Governance/Compliance
  • Cost Optimization
  • Performance
  • Monitoring
  • Evaluation
  • Troubleshooting
• Builder Labs: Generative AI on AWS

As you can probably tell, my assignments to myself are aggressive. I tend to believe that the builder-leader model is the right modern approach to tech leadership.

It’s going to be a busy fall and winter.