Projections Explained (DDD EU 2020 talk summary)
Projections Explained
Projections since 2009…
CQRS — split reading and writing (this brought opportunities)
Mostly a logical separation (not a physical one) — that’s not what the pattern is about (but you can do it)
Event Sourcing + DDD — apply it on a bounded context level
You need to know the requirements from the consumers BEFORE building projections
Event Store is a collection of streams (which are in turn collections/sequences of heterogenous events partitioned by name or key)
Aggregate instance has its own stream
Each event appears at a certain position/offset (sequence) in an event stream (a monotonically incremented number)
A subscription is needed in order to be notified when new events arrive
- Starts off at a known position in the ALL stream, or from a VERSION in the aggregate stream (this allows us to resume when we crash — that’s why projection is in control of the cursor)
What is a projection?
It is the act of transforming a stream of events
act of deriving state from a stream of events
…many meanings
More specifically a projection is a transformation function that transforms the stream of events to a certain data structure (derived state)
which could be stored in-memory or to a specific storage engine (database) which are optimized for your reading use case (querying) — don’t ignore transaction support of it
Don’t always use the same stack for everything.
If your read models change because of different reasons — build two projections (it depends)
They often get neglected — they do need some analysis and design!
What does this mean?
- What is the optimal data structure for this read model
- What events do we need
- Where do we store it
Read models (data structures) capture data such as:
- used to identify rows / values / documents
- to return to consumers
- to filter on
- needed to allow updates from future events
Writing Projections
- written with cohesion / as units
- focus on what, not how
- limit amount of boilerplate
- optimized for your platform/language
- don’t be conservative about the number of them (this one is mine)
Forms of projections:
f(state, event) -> state
f(event) -> statement[]
f(connection, event) -> unit — eg. for calling different API's
IHandle<TEvent> — using generic interfaces
handling using pattern matching
can be handled in an actor
Smells
- complexity
- dependencies (e.g. on another projections data structure) — try to avoid unless you can ensure the order of projection building, but it’s still not encouraged, nor fun
- repeated logic in different projections or same logic in the domain model and read model (don’t do this)
- big data structure — depends
Testing
Should test if projection behaves as expected (specifications)
Preferably as integration tests
Nothing special (given these events -> I’m expecting for this state to be in the store)
MISC
Use lookup caches (in-memory or otherwise) if you need historical data while building projections
If you have big read models — think if you really need all of that history — is it temporal — is it still relevant this year / this month … (can help you keep the volume of your data down) — try to partition by time (this is domain-specific)
You can be flexible about a way you deploy and run your projection processes — but keep them lightweight and single-threaded (little use of doing parallel writes)
You want them to be fault-tolerant and able to recover from failure and continue where they left off.
Try to isolate different projections in different processes (so they don’t block each-other)
All in all, an informative talk and I can confirm many of the points he made based on my personal experience in dealing with projections.
