For fun
A pipeline with no servers to babysit
Files land in a bucket, code wakes up, reshapes them, and puts them somewhere a dashboard can read in near-real time. Nothing runs when nothing happens.
How it works
event-driven, end to end- S3 bucket A raw file arrives. That is the only trigger.
- Lambda Validates, reshapes, enriches. Scales to zero between events.
- DynamoDB Written so the read pattern is cheap, not so the writes are pretty.
- Whoever asks Near-real-time answers instead of a nightly batch.
Why I built it
I wanted to understand event-driven architecture by paying for it, not by reading about it. A nightly batch job is easy. Something that reacts the moment a file lands, costs nothing while idle, and still behaves when two files arrive at once — that is where the real decisions are.
What it taught me
- In DynamoDB you design the table around the question you will ask, not the data you have.
- A retry that is not idempotent is just a slower way to corrupt your data.
- Writing the runbook first made the architecture simpler — anything I could not explain, I removed.
Built with
- AWS Lambda
- S3
- DynamoDB
- Python
- IAM
Documented in myCloudDocs