Three production systems in full detail: the problem, the solution shape, the architecture, and why every service earned its place. The plain-language versions live on the main page.
An enterprise data platform on Databricks, delivered through Systems Ltd. I worked within and extended the transformation and orchestration layers: PySpark and Spark SQL across Bronze → Silver → Gold, plus the validation and troubleshooting that keeps a regulated reporting layer trustworthy.
Core transactional extracts arrive as raw files — no schema stability, no deduplication, no conformance across source systems. Analysts can't query raw extracts safely: no ACID guarantees, no reliable "current state" of an entity. Full reprocessing on every run doesn't scale. And in a regulated enterprise, the reporting layer carries audit weight — silent data corruption between layers is the primary risk to design against.
A Delta Lake medallion architecture: Bronze lands raw extracts append-only with ingestion metadata;
Silver cleanses, deduplicates, and conforms entities via incremental MERGE INTO upserts —
no full reloads; Gold builds business-ready aggregates for reporting. Databricks Workflows drives the
job graph as a control plane, and validation checkpoints sit between every layer so corruption is
caught at the boundary, not in a report.
| Service | Role in this system | Why it, and not the obvious alternative |
|---|---|---|
| Delta Lake | Table format for all three layers | ACID transactions and time travel over plain Parquet — an audit trail can't sit on files that can half-write. |
| Medallion layers | Bronze / Silver / Gold separation | Each layer has one job, so failures isolate to a boundary. One monolithic transform makes corruption untraceable. |
| Delta MERGE INTO | Incremental Silver upserts | Maintains a reliable current state without full reloads — append-plus-reprocess doesn't scale on daily extracts. |
| PySpark / Spark SQL | All transformation logic | Extracts are distributed-scale; partitioned Spark jobs with tuned file layout beat any single-node process. |
| Databricks Workflows | Job orchestration | Dependencies, retries, and scheduling live next to the compute — no extra orchestration hop to operate. |
A shared, multi-domain AWS platform delivered through Systems Ltd — research data alongside commercial and operational data, serving several business domains at once. My role was weighted toward reliability: diagnosing production failures, optimizing S3 flows and DAG performance, and promoting workflows dev → prod inside a controlled enterprise access model.
Heterogeneous workloads across multiple business domains can't share one execution shape — some need Python-scale glue, some need distributed Spark. Cluster spin-up latency is unacceptable when a large DAG estate fires continuously. Memory-skewed jobs choke on uniform hardware. And at this DAG count, failures are continuous background noise: the platform's real problem is diagnosability, not happy-path execution.
MWAA (managed Airflow) as the control plane, with both scheduled DAGs and event-based DAGs triggered by S3 file arrivals. Each DAG routes tasks by type: pure Python runs on the Airflow worker layer, PySpark is submitted as steps to a pool of ~six persistent EMR clusters — one provisioned with very large memory for heavy workloads. Curated output lands back in S3 and serves analytics through Redshift, with CloudWatch as the investigation surface and IAM boundaries shaping every change.
| Service | Role in this system | Why it, and not the obvious alternative |
|---|---|---|
| Persistent EMR clusters | Standing Spark compute pool | Transient clusters mean spin-up latency on every run — unacceptable with a DAG estate firing continuously. The trade-off (standing cost, cluster hygiene) was accepted deliberately. |
| Large-memory cluster | Dedicated home for heavy jobs | Memory-skewed workloads shouldn't force every cluster onto expensive uniform hardware — isolate the skew instead. |
| MWAA | Managed Airflow control plane | A big mixed DAG estate needs mature scheduling, sensors, and history — without the team also operating Airflow itself. |
| Python / Spark task split | Routing by workload shape | Glue-code tasks don't deserve a Spark cluster; distributed joins can't run on a worker. Route each to its right engine. |
| Redshift | Analytical serving layer | Domain analysts need warehouse SQL they can validate — and validation queries were part of the reliability loop. |
Listing-data ingestion built fully serverless and event-driven. This is where I owned the most design surface: I designed and deployed these pipelines, onboarded new data feeds, mentored teammates on the AWS patterns, and documented each solution end to end — including rollback.
MLS/RETS feeds are numerous, independently owned, and inconsistent — every new data provider means another feed with its own schema quirks, credentials, and refresh cadence. Volume is bursty and scheduled, so standing compute would idle most of the time. Each feed carries its own credentials — a secrets-handling problem before it's a data problem. And a feed that silently changes schema poisons everything downstream. The real business metric is how cheap it is to onboard the next feed.
No standing compute at all. EventBridge rules — scheduled and event-based — trigger a Step Functions state machine per ingestion workflow, with the error/retry path modeled explicitly in the state machine rather than buried in code. Lambda stages fetch, parse, normalize, and validate each feed into a unified listing schema, landing raw and processed data in S3 and relational serving data in RDS. Downstream apps consume through API Gateway behind Cognito; Secrets Manager feeds per-feed credentials into Lambda. CloudFormation/SAM makes each new feed a repeatable deployment, not a project.
| Service | Role in this system | Why it, and not the obvious alternative |
|---|---|---|
| Serverless (Lambda) | All compute | Feed volume is bursty and scheduled — standing servers would idle most of the day. Serverless is the correct fit here, not a fashionable one. |
| Step Functions | Workflow orchestration | Sequencing, error handling, and retries live in the state machine where they're visible and testable — not buried inside function code. |
| EventBridge | Triggering surface | One place for both cron schedules and event rules — each feed's cadence is configuration, not code. |
| Unified listing schema | Normalization target | Downstream consumers see one listing model no matter which MLS quirk produced it — the alternative is every consumer handling every feed's shape. |
| Secrets Manager | Per-feed credentials | Dozens of independently-owned feed credentials can't live in config files — rotation had to work without touching code. |
| CloudFormation / SAM | Deployment | New-feed onboarding is the recurring workload — templated infrastructure makes the marginal feed cheap to add. |
Where the foundation was built: turning repetitive client work into repeatable systems. Selenium scrapers pulled business data from the web, Python pipelines cleaned it, MongoDB stored it, GPT APIs enriched it, and Flask/Streamlit apps delivered it back as reports and demos — saving clients 30+ hours of manual work per month.
