Akka / Pekko Persistence (Scala / Java)

Event-sourced actors for the JVM: a command handler decides which events to append, a pure event handler folds them into state, and an actor is rebuilt after a crash by replaying its own journal from the latest snapshot — the durable-execution shape realised as state machine plus journal rather than as replayed code.

Field	Value
Language	Scala (with a Java DSL); JVM
License	Apache Pekko: Apache-2.0. Akka (since 2.7): Business Source License 1.1 with a change date of 2029-09-09 (`LICENSE`)
Repository	apache/pekko (the Apache fork of Akka 2.6); akka/akka
Documentation	Event Sourcing · Snapshotting · Schema evolution · Testing · Persistence Query · Durable State
Category	`event-sourced actors`
Persistence model	`explicit state machine` — replay of events through a pure fold, from a snapshot; never replay of code
Journal store	Pluggable `AsyncWriteJournal` (in-memory, LevelDB in-tree; JDBC / Cassandra / R2DBC out of tree) plus a pluggable `SnapshotStore`; keyed by `persistenceId` and a per-id `sequenceNr`
Latest release	Pekko `1.7.0`, tagged 2026-08-19 (release notes)
Local clone	`$REPOS/pekko` at `ce44fbe7a1ee2697c293e355064b7dd893914bca` (2026-09-10); `$REPOS/akka` at `f24cb608fa88b650da8859d4604d46279f767cf0` (2026-09-09), consulted only for the license

Last reviewed: September 12, 2026.

Overview

What it solves

Akka Persistence, and its Apache fork Pekko Persistence, make an actor survive a crash, a restart, or a migration to another cluster node without losing state. The trick is that the actor's state is never written; only the events it decided to emit are, and the state is a deterministic fold over them. The typed documentation states the model plainly:

"An event sourced actor (also known as a persistent actor) receives a (non-persistent) command which is first validated if it can be applied to the current state. … If validation succeeds, events are generated from the command, representing the effect of the command. These events are then persisted and, after successful persistence, used to change the actor's state. When the event sourced actor needs to be recovered, only the persisted events are replayed of which we know that they can be successfully applied. In other words, events cannot fail when being replayed to a persistent actor, in contrast to commands." — docs/src/main/paradox/typed/persistence.md

That last sentence is the whole design in miniature. A durable-execution engine such as Temporal replays the program and hopes it takes the same branches; Pekko replays the decisions (events) and folds them with a function that, by contract, cannot fail and cannot observe the world. Command handlers may be arbitrarily effectful; event handlers may not.

The same module also ships a second, non-event-sourced flavour, DurableStateBehavior, which stores only the latest state under an optimistic revision counter; it is covered briefly in Replay or snapshot because it is the "snapshot-only" corner of the design space.

Design philosophy

Three commitments recur through the source and the docs.

Single writer per persistenceId. A journal is a set of independent streams, one per id, each with a dense sequenceNr. Correctness rests on one actor instance appending to a stream at a time; Cluster Sharding exists largely to make that true across a cluster (persistence.md § Cluster Sharding and EventSourcedBehavior).
Side effects only after the write, and only at most once. thenRun callbacks fire when the journal has acknowledged the write and are never re-run on recovery; the documented guarantee is at-most-once, not at-least-once (see Compensation and failure handling).
The event handler is a pure fold. From the same page: "The event handler must only update the state and never perform side effects, as those would also be executed during recovery of the persistent actor." (persistence.md).

Everything that a replay-of-code engine enforces with a sandbox (no clocks, no random, no I/O inside workflow code) Pekko enforces by making the replayed thing a data fold whose inputs are entirely in the journal.

How it works

The user-facing shape

An EventSourcedBehavior is four values: a stable id, an empty state, a command handler and an event handler (EventSourcedBehavior.scala):

scala

// persistence-typed/.../scaladsl/EventSourcedBehavior.scala
def apply[Command, Event, State](
    persistenceId: PersistenceId,
    emptyState: State,
    commandHandler: (State, Command) => Effect[Event, State],
    eventHandler: (State, Event) => State): EventSourcedBehavior[Command, Event, State]

The command handler returns an Effect directive, never performs the write itself. The factory in Effect.scala offers persist(event), persist(events) (one atomic batch), none, unhandled, stash, unstashAll, reply, noReply, and stop; an EffectBuilder chains thenRun(State => Unit), thenReply, thenUnstashAll, thenStop. Internally every one of those is a CompositeEffect(persistingEffect, sideEffects) (EffectImpl.scala), which is what lets the runtime separate what to append from what to do afterwards.

The docs' bank-account example (AccountExampleWithEventHandlersInState.scala) shows the idiom, including withEnforcedReplies, which turns "forgot to reply" into a compile error:

scala

case class Deposited(amount: BigDecimal) extends Event

EventSourcedBehavior.withEnforcedReplies(persistenceId, EmptyAccount, commandHandler(accountNumber), eventHandler)

// inside the command handler
Effect.persist(Deposited(cmd.amount)).thenReply(cmd.replyTo)(_ => StatusReply.Ack)

Behaviour is further configured by builder methods on the returned value: withRetention(RetentionCriteria), snapshotWhen(predicate), eventAdapter(EventAdapter), snapshotAdapter, withTagger, withRecovery(Recovery), onPersistFailure(BackoffSupervisorStrategy), receiveSignal (EventSourcedBehavior.scala).

The four-phase recovery machine

An EventSourcedBehavior is implemented as a chain of four internal behaviours, each documented as "N (of four)" in its header:

Phase	File	What it does
1	`RequestingRecoveryPermit.scala`	Asks the `RecoveryPermitter` for a permit (`max-concurrent-recoveries = 50` by default in `reference.conf`) so a cluster restart does not stampede the journal; stashes every incoming message meanwhile
2	`ReplayingSnapshot.scala`	Loads the latest snapshot matching `SnapshotSelectionCriteria`; on success `state = setup.snapshotAdapter.fromJournal(snapshot)` and the sequence number is taken from the snapshot metadata; on `LoadSnapshotFailed` it either fails recovery or, with `snapshot-is-optional = true`, restarts from `emptyState` at `0L`
3	`ReplayingEvents.scala`	Sends `ReplayMessages(seqNr + 1, toSeqNr)` to the journal and folds every `ReplayedMessage(repr)` through `setup.eventAdapter.fromJournal` and then `setup.eventHandler(state.state, event)`; `RecoverySuccess(highestJournalSeqNr)` moves to phase 4
4	`Running.scala`	Three sub-behaviours: `HandlingCommands`, `PersistingEvents` (commands stashed until the journal acks), `StoringSnapshot` (commands stashed until the snapshot store acks)

The fold in phase 3 is literally one line (ReplayingEvents.scala):

scala

val newState = setup.eventHandler(state.state, event)

Any NonFatal exception thrown there becomes onRecoveryFailure, which cancels the replay, emits the RecoveryFailed signal, returns the permit and throws a JournalFailureException naming the event class and sequence number. When recovery completes, RecoveryCompleted is delivered with the recovered state, the internal stash is drained one message at a time by tryUnstashOne(new running.HandlingCommands(...)), and, if the number of replayed events already exceeds the retention threshold, a snapshot is taken first.

The write path

Running.HandlingCommands.onCommand calls the user's command handler and hands the Effect to applyEffects (Running.scala), which unwraps CompositeEffect, accumulates the side effects, and dispatches on the persisting effect. For Persist(event), handleEventPersist does something worth noticing: it applies the event handler before writing, so that a validation exception in the event handler is raised before an invalid event reaches the journal.

scala

// persistence-typed/.../internal/Running.scala (handleEventPersist, abridged)
_currentSequenceNumber = state.seqNr + 1
val stateAfterApply = state.applyEvent(setup, event)
val eventToPersist = adaptEvent(stateAfterApply.state, event)
val eventAdapterManifest = setup.eventAdapter.manifest(event)
internalPersist(setup.context, cmd, stateAfterApply, eventToPersist, eventAdapterManifest, OptionVal.None)
...
persistingEvents(newState2, state /* visible until ack */, numberOfEvents = 1, shouldSnapshotAfterPersist, shouldPublish = true, sideEffects)

internalPersist (ExternalInteractions.scala) builds the journal record and sends WriteMessages to the journal actor:

scala

val repr = PersistentRepr(
  event,
  persistenceId = setup.persistenceId.id,
  sequenceNr = newRunningState.seqNr,
  manifest = eventAdapterManifest,
  writerUuid = setup.writerIdentity.writerUuid,
  sender = ActorRef.noSender)
val write = AtomicWrite(repr) :: Nil
setup.journal.tell(JournalProtocol.WriteMessages(write, setup.selfClassic, setup.writerIdentity.instanceId), setup.selfClassic)

The actor then sits in PersistingEvents, whose onMessage stashes every IncomingCommand, until WriteMessageSuccess(p, id) arrives for its own writer instanceId. Only then does onWriteResponse set visibleState = state, run applySideEffects(sideEffects, state), and unstash. WriteMessageRejected raises EventRejectedException; WriteMessageFailure emits the JournalPersistFailed signal and throws JournalFailureException, which stops the actor unless onPersistFailure installed a backoff supervisor.

The journal record and the journal contract

The unit of storage is PersistentRepr — the fields are the schema, whatever the backing store:

Field	Meaning
`persistenceId`	The stream key; `PersistenceId(entityTypeHint, entityId)` joins them with a vertical bar (`DefaultSeparator`) by default (`PersistenceId.scala`)
`sequenceNr`	Dense, per-stream, assigned by the writer (`state.seqNr + 1`)
`payload`	The (adapted) event
`manifest`	The type hint the `EventAdapter` returned, replayed back into `fromJournal`
`writerUuid`	Identifies the actor incarnation that wrote it; the replay filter uses it to detect overlapping writers
`timestamp`	Wall-clock at write
`metadata`	Optional; used by Replicated Event Sourcing for version vectors

An AtomicWrite wraps one or more PersistentRepr from a single persist effect. The plugin contract is one method (AsyncWriteJournal.scala):

scala

def asyncWriteMessages(messages: immutable.Seq[AtomicWrite]): Future[immutable.Seq[Try[Unit]]]

whose doc comment fixes the semantics a store must honour: "All PersistentRepr of the AtomicWrite must be written to the data store atomically, i.e. all or none must be stored." … "The Future must only be completed with success when all messages in the batch have been confirmed to be stored successfully, i.e. they will be readable, and visible, in a subsequent replay. If there is uncertainty about if the messages were stored or not the Future must be completed with failure." Rejections (a serialization error, a store that cannot do multi-event atomic writes) are per-AtomicWrite Try failures and mean the event was not stored; connection problems "must not be signaled as rejections." Deletion never lowers the highest sequence number.

Snapshots and retention

RetentionCriteria.snapshotEvery(numberOfEvents, keepNSnapshots) (RetentionCriteria.scala) saves a snapshot every N events and deletes older snapshots outside the retained window; .withDeleteEventsOnSnapshot additionally deletes the events that the retained snapshots make redundant. The default is RetentionCriteria.disabled: no snapshots are ever taken unless asked. snapshotWhen((state, event, seqNr) => Boolean) takes ad-hoc snapshots but never triggers deletion. In Running.StoringSnapshot, SaveSnapshotSuccess drives the delete of old events before old snapshots, so that at no point does a surviving snapshot depend on deleted events (Running.scala; Snapshotting).

Projections

persistence-query exposes read-side streams over the same journal. EventsByPersistenceIdQuery.eventsByPersistenceId(persistenceId, fromSequenceNr, toSequenceNr) returns a live Source[EventEnvelope, NotUsed] ordered by sequence number that "is not completed when it reaches the end of the currently stored events, but it continues to push new events when new events are persisted" (EventsByPersistenceIdQuery.scala). The docs describe it as "a query equivalent to replaying an event sourced actor" (Persistence Query) and point resumable projections (store the processed offset, resume from it) to the separate Pekko Projections module.

Analysis

The eight questions below were written for replay-of-code engines. Where a question is about "replaying code", this page translates it to the event-sourcing analogue and says so.

1. Step identity and replay matching

Analogue: how is a persisted event identified, and how is it matched on recovery?

By position, not by name. Every event carries (persistenceId, sequenceNr); the writer assigns sequenceNr = state.seqNr + 1 in internalPersist (ExternalInteractions.scala) and recovery asks the journal for the contiguous range seqNr + 1 .. toSeqNr (ReplayingEvents.scala). There is no step name, no args hash and no attempt counter, because there is nothing to match against: the replayed thing is the event stream itself, and the consumer of it, the event handler, has no other input. The manifest string is the only "identity" a payload carries, and it identifies the event's schema for the adapter, not its position in a program.

The one matching problem that does exist is writer identity: a writerUuid per incarnation lets the ReplayFilter "detect corrupt event stream during replay … to find events emitted by overlapping writers". That is the event-sourcing form of the "two runs journaled into one file" hazard.

2. Journal versus world

Analogue: when the journal and the world disagree, which wins?

The journal wins, unconditionally, and the design makes disagreement structurally impossible to detect during recovery: the event handler may not observe the world (no I/O, no clock), so there is no code path in phases 2 to 3 in which world and journal could be compared. Recovery is a fold over stored facts, and the docs guarantee that "events cannot fail when being replayed" (persistence.md).

Where the world does re-enter is after recovery, in the RecoveryCompleted signal handler, and the docs are explicit that this is where one reconciles unacknowledged side effects: "You may inspect the state when receiving the RecoveryCompleted signal and execute side effects that have not been acknowledged at that point. That may possibly result in executing side effects more than once." (persistence.md § Side effects ordering and guarantees). The runtime offers no rule table and no re-observation: the application encodes "what was started but not confirmed" in the state and decides.

The journal-vs-journal case is handled: replay-filter.mode (repair-by-discard-old / fail / warn / off) says what to do when two writers produced the same sequence number (ReplayFilter.scala; persistence.md § Replay filter). And snapshot-vs-journal has a documented policy: with snapshot-is-optional = true a snapshot that fails to load is discarded and the full event stream is authoritative (Snapshotting § Optional snapshots), with the caveat that this is unsafe if events were deleted.

3. Determinism enforcement

Analogue: command handler determinism, event handler purity.

By discipline, documented as a hard rule, with the type signature doing the only mechanical work. The event handler's type is (State, Event) => State (EventSourcedBehavior.scala); nothing stops a Scala function of that type from reading a clock, and the runtime does not sandbox it. The docs state the rule and the reason:

"The same event handler is also used when the entity is started up to recover its state from the stored events. The event handler must only update the state and never perform side effects, as those would also be executed during recovery of the persistent actor. Side effects should be performed in thenRun from the command handler after persisting the event or from the RecoveryCompleted after Recovery." — persistence.md

The command handler, by contrast, is not required to be deterministic. It runs once, on the live path, and is explicitly allowed to do "a conversation with several external services" (persistence.md). Its decision is captured as an event; nothing about the decision is ever recomputed. This is the cleanest split in the survey: the nondeterministic part is never replayed, and the replayed part is a fold with no ambient access.

Two mechanical aids exist. withEnforcedReplies makes the compiler reject a command handler that forgets to reply. And handleEventPersist applies the event handler before the write (Running.scala), so an event that the handler cannot fold never enters the journal — a cheap invariant check that a replay-of-code engine cannot offer, because there the "handler" is the rest of the program.

4. Compensation and failure handling

There is no compensation primitive. The docs for persistence-typed never mention sagas or compensation; the word does not occur in docs/src/main/paradox/typed/persistence*.md. A saga is written as an ordinary event-sourced entity (or a process manager) whose state records which steps are done and whose command handler emits StepCompensated events. Ordering and triggering are therefore whatever the application's state machine says.

What the runtime does define is the side-effect guarantee, and it is weaker than the brief expected:

"Any side effects are executed on an at-most-once basis and will not be executed if the persist fails. Side effects are not run when the actor is restarted or started again after being stopped." — persistence.md

The mechanism is in Running.PersistingEvents.onWriteResponse (Running.scala): the accumulated sideEffects list is applied only after the journal's WriteMessageSuccess for the last event of the batch, and the list lives in actor memory, so a crash between ack and callback loses it. The at-least-once story is opt-in by the application: persist a "started" fact, run the effect in thenRun, persist a "confirmed" fact on the reply, and re-drive the unconfirmed ones from RecoveryCompleted. That is exactly the started/completed pair, but Pekko leaves it to the user to write.

Failure of the write has a sharp policy: WriteMessageFailure throws JournalFailureException and the default is to stop the actor; onPersistFailure accepts only a BackoffSupervisorStrategy because, as the API doc says, "Resume is not allowed as it will be unknown if the event has been persisted" (EventSourcedBehavior.scala). Rejections (EventRejectedException) are distinct: the event is known not to be stored, so the actor may continue.

Atomicity is the other lever: Effect.persist(events) becomes a single AtomicWrite, so "the recovery of a persistent actor will therefore never be done partially with only a subset of events persisted by a single persist effect" (persistence.md § Atomic writes).

5. Versioning against old histories

Analogue: schema evolution of events.

This is where Pekko's design is richest. The premise is stated up front:

"Since events are never deleted, we need to have a way to be able to replay (read) old events, in such way that does not force the PersistentActor to be aware of all possible versions of an event that it may have persisted in the past. Instead, we want the Actors to work on some form of 'latest' version of the event and provide some means of either converting old 'versions' of stored events into this 'latest' event type, or constantly evolve the event definition - in a backwards compatible way - such that the new deserialization code can still read old events." — persistence-schema-evolution.md

The page then catalogues the four common changes — add a field, rename or remove a field, remove an event type, split one event into several — with a pattern for each, and it is careful to say that the serializer (protobuf, Jackson with its own schema-evolution support, Avro) carries most of the load, with Java serialization explicitly disrecommended for anything that must persist.

The runtime hook is EventAdapter[E, P]: toJournal(e): P (declared "must be an 1-to-1 transformation. It is not allowed to drop incoming events"), manifest(event): String, and fromJournal(p, manifest): EventSeq[E], which "may be adapted into multiple (or none) events". The adapter's own docs list its uses: envelope extraction, domain-to-data-model mapping, "migration by splitting up events into sequences of other events" and "migration filtering out unused events, or replacing an event with another". Because the manifest is stored beside the payload, an adapter can upcast by version tag. The schema-evolution page adds the "tombstone" refinement: a SerializerWithStringManifest can recognise a retired manifest and return a placeholder without deserializing, so the old class can be deleted from the codebase.

Two limits are stated in the source. fromJournal "is not called in any read side so will need to be applied manually when using Query" (EventAdapter.scala), so projections must repeat the adaptation. And snapshots have their own, coarser story: a SnapshotAdapter, or simply SnapshotSelectionCriteria.None to "replay all journaled events … if snapshot serialization format has changed in an incompatible way" (Snapshotting).

What is absent is any versioning of the handlers: there is no getVersion/patched marker as in Temporal or Azure Durable Functions, because nothing about the handlers is journaled. Changing the event handler silently changes what every old stream folds to; the only guard is the fact that the fold is pure and therefore testable against a recorded stream.

6. Concurrency under replay

Analogue: one actor is one writer; what happens to concurrent inputs during recovery and during a write?

Concurrency is excluded rather than replayed. Within one persistenceId:

During all three pre-Running phases, every IncomingCommand goes to the internal stash (stashInternal) (ReplayingSnapshot.scala, ReplayingEvents.scala); the docs: "New messages sent to the actor during recovery do not interfere with replayed events. They are stashed and received by the EventSourcedBehavior after the recovery phase completes." (persistence.md).
During a write, PersistingEvents.onCommand stashes; during a snapshot, StoringSnapshot.onCommand stashes (Running.scala). Exactly one WriteMessages is outstanding per actor at any time; the journal doc relies on it: "A PersistentActor will not send a new WriteMessages request before the previous one has been completed." (AsyncWriteJournal.scala).
The stash is bounded (stash-capacity = 4096, reference.conf) with a Drop or Fail overflow strategy (StashManagement.scala), and the user-level Effect.stash buffer is "kept in an in-memory buffer, so in case of a crash they will not be processed" (Effect.scala).

Across nodes, the single-writer rule is delegated to Cluster Sharding: "Pekko Persistence is based on the single-writer principle, for a particular PersistenceId only one persistent actor instance should be active at one time. … Cluster Sharding ensures that there is only one active entity for each id." (persistence.md). When that invariant is nonetheless broken, the ReplayFilter is the after-the-fact repair.

Recovery itself is throttled cluster-wide by the permit (max-concurrent-recoveries), and the journal paces one actor's replay with replay-batch-size and ReplayBatchAck (ReplayingEvents.scala). Replicated Event Sourcing (multi-writer with version vectors, the metadata field) is the deliberate exception and is out of scope here.

7. Replay or snapshot

Both, layered, and the choice is a retention policy rather than an architecture. Recovery always starts from the latest snapshot (if any) and replays only the tail; with RetentionCriteria.disabled (the default) that means full replay forever. snapshotEvery(N, keepN) bounds replay to at most about N events plus the retained window; adding withDeleteEventsOnSnapshot turns the journal into a rolling window and gives up the audit history and the ability to rebuild after a bad snapshot (RetentionCriteria.scala; Snapshotting). Snapshot cost is paid on the write path: the actor sits in StoringSnapshot and stashes commands until the store acks, which is why "the state can safely be mutable although the serialization and storage of the state is performed asynchronously" (Snapshotting).

DurableStateBehavior is the pure-snapshot pole: "only the latest state is stored, we don't have access to any of the history of changes"; its persist "will be updated only if the revision number of the incoming record is 1 more than the already existing record. Otherwise persist will fail" (Durable State). That is the compare-and-set version of single-writer, and it rules out projections over history and any after-the-fact reinterpretation.

8. Testing

Two kits, both in persistence-testkit, both against an in-memory journal (PersistenceTestKitPlugin.scala).

EventSourcedBehaviorTestKit drives a real actor one command at a time and returns a synchronous CommandResult with events, event, state, stateOfType, and, for runCommand[R](replyTo => cmd), the reply (AccountExampleDocSpec.scala):

scala

val result = eventSourcedTestKit.runCommand[StatusReply[Done]](AccountEntity.CreateAccount(_))
result.reply shouldBe StatusReply.Ack
result.event shouldBe AccountEntity.AccountCreated
result.stateOfType[AccountEntity.OpenedAccount].balance shouldBe 0

It also verifies, by default, that every command, event and state survives a serialization round trip (SerializationSettings), and restart() re-runs the whole four-phase recovery against what the test persisted: "It will restart the behavior, which will then recover from stored snapshot and events from previous commands." (Testing). initialize(state, events*) seeds the storage so a test can start from a hand-written history.

PersistenceTestKit is the fault injector: expectNextPersisted, expectNothingPersisted, persistedInStorage(persistenceId), persistForRecovery, and the failure knobs failNextPersisted, failNextNPersisted, rejectNextPersisted, failNextRead, plus a user-supplied ProcessingPolicy for arbitrary per-operation decisions (ProcessingPolicy.scala). Together they cover "crash the write at event k" and "the store rejects this event" without touching a real database.

What is absent: there is no built-in "crash at every event index and resume" driver, though it is a short loop over failNextNPersisted plus restart; and there is no notion of mutating the outside world between crash and resume, because the runtime has no world-observation step to mutate around.

9. Journal integrity and the single writer

This is the dimension where Pekko is richest, and the only system in the survey that ships a policy for what to do when two writers are detected after the fact.

Overlapping writers are detected on replay, by design. ReplayFilter exists to "Detect corrupt event stream during replay. It uses the writerUuid and the sequenceNr in the replayed events to find events emitted by overlapping writers" (ReplayFilter.scala). Every persisted event carries the writerUuid of the actor incarnation that wrote it, so a stream containing events from two incarnations at overlapping sequence numbers is recognisable without any cooperation from the storage plugin.

Four policies, including a repair. The filter's modes are Fail, Warn, RepairByDiscardOld and Disabled, with a sliding windowSize and a maxOldWriters bound (ReplayFilter.scala). RepairByDiscardOld is notable: the library will discard the losing writer's events and carry on. No other system surveyed offers an automatic repair for a split-brain record, and none offers the choice as configuration.

Appends are atomic in batches, and the batch is constrained. AtomicWrite requires a non-empty payload and requires every event in it to share one persistenceId (Persistent.scala), so a batch is one aggregate's events and commits together or not at all. That is the primitive several replay engines lack: a way to write a group of records as a unit.

Sequence numbers are dense per persistence id, so a gap is itself a detectable fault rather than an ordinary state. Positional identity (§1) depends on that density.

The write contract makes the ambiguous case explicit. The journal plugin's asyncWriteMessages documentation requires that "If there is uncertainty about if the messages were stored or not the Future must be completed with failure" — so a plugin must not report success it is unsure of, and the actor treats a failed write as fatal by default. This is the same distinction Temporal draws with OperationPossiblySucceeded, pushed down into the storage contract instead of up into recovery logic.

Single-writer at runtime is Cluster Sharding's job. One entity id has one live activation in a shard region, and the documentation directs users to sharding precisely to obtain that guarantee. The ReplayFilter is the backstop for when it has not held.

Torn writes are the plugin's problem, and because the plugin interface is the contract rather than a format, "what a partial record means" varies by backend — the cost of pluggability.

10. Operator recovery and intervention

Pekko is a library, not a platform, and this dimension is where that shows most starkly: there is essentially no operator surface, and the absence is structural rather than an oversight.

No fork, no rewind, no reset. Nothing resumes an entity from a chosen sequence number with a fresh identity. The recovery model is "replay everything from the last snapshot", and the only lever over that is which snapshot exists.

Deletion is the one destructive lever, and it is a programmatic one.deleteMessages(toSequenceNr) deletes events up to an inclusive upper bound (Eventsourced.scala), and snapshots have a matching deletion plus RetentionCriteria for automatic pruning. These are calls the entity makes about itself, not commands an operator issues from outside.

Inspection is a query-side concern. persistence-query's eventsByPersistenceId and currentEventsByPersistenceId let an application read a stream back, so "what did this entity do" is answerable — but by writing a program, not by running a tool. Notably the EventAdapter is not applied on the query side, so a reader sees stored shapes rather than upcast ones.

Cancellation and pausing do not apply. An entity is not a run with a lifecycle to interrupt; it is an actor that exists and processes messages. Stopping it is passivation, which is invisible to correctness, and there is no state meaning "deliberately stopped for an operator".

Dead-lettering is the actor system's generic mechanism, not a persistence feature: a message to a stopped actor goes to dead letters, with no relationship to a record that cannot be applied. A failed recovery stops the actor.

The finding here is a real one about library scope. Everything an operator might want — list runs, inspect one, resume from a point, park a broken one — is available in the platform systems and absent here, because those affordances require a component that outlives any one process. A library can supply the record and the replay; the surface over them is a separate product decision.

11. Suspension and external input

The concept does not transfer cleanly, and saying why is the finding.

An event-sourced actor never suspends in the durable sense. It is alive, holding its folded state in memory, and it waits by simply not having received a message yet. There is no journal entry for "waiting", no resumption by replay, and no process that ends. Replay happens on activation, and activation is caused by a message arriving — so the mechanism a replay engine uses for waiting is, here, the mechanism for starting.

Waiting inside a command is stash. Effect.stash() and Effect.unstashAll() (Effect.scala) let a handler defer commands it is not ready for, and the runtime also stashes automatically during recovery, during a persist, and while a snapshot is in flight. The stash is bounded and in-memory, so it is a concurrency tool rather than a durability one — a crash loses whatever is stashed.

External input is an ordinary message, addressed to the entity id through sharding. Delivery is at-most-once by default, which inverts the replay engines' assumption: there, a duplicate signal is the hazard; here, a lost message is. At-least-once has to be built, with the sender re-driving from its own persisted state.

Side effects after a write are documented as at-most-once. "Any side effects are executed on an at-most-once basis and will not be executed if the persist fails. Side effects are not run when the actor is restarted or started again after being stopped." So a thenRun is not a durable step: it is a best-effort action attached to a durable fact, and re-driving it is the author's job, keyed off state observed in RecoveryCompleted.

Timers are not durable. Scheduled messages come from the actor system's timer facility and do not survive a restart, so a durable delay must be modelled as persisted state plus a re-scheduling decision taken on recovery. Several replay engines give durable timers away for free; here they are a design task.

Human-in-the-loop is not a pattern the documentation addresses, which follows from the model: an approval is a message, and remembering that one is outstanding is state the author persists like any other.

Strengths

The cleanest determinism boundary in the survey. Nondeterminism lives in the command handler, which runs once; the replayed fold has no inputs but state and event. No sandbox, no forbidden-API list, no "deterministic time" shim.
Journal semantics are a written contract for plugin authors — atomic batches, "uncertain means failed", rejection vs failure, never-decreasing highest sequence number (AsyncWriteJournal.scala).
Schema evolution is a first-class topic with a stored manifest, an adapter that can split, drop or upcast, and a long documented playbook (persistence-schema-evolution.md).
Fold-before-write catches un-foldable events before they are stored (Running.scala).
Projections are the same stream the actor recovers from, ordered by the same sequence number (EventsByPersistenceIdQuery.scala).
Testing is honest: the test kit runs the real recovery machine, and the storage plugin is a fault injector.

Weaknesses

At-most-once side effects, and the at-least-once discipline is homework. The started/completed pattern is neither a primitive nor a documented recipe beyond one paragraph.
No compensation model at all. Sagas are entirely application code.
Handler evolution is invisible. Only events have versions; a changed event handler re-derives every old state with no marker and no check.
Purity is by contract. The (State, Event) => State type is the only enforcement.
Command-during-recovery is stash-or-drop, and both stashes are memory-only; a bounded buffer with Drop silently loses commands under load.
The single-writer guarantee is outsourced to Cluster Sharding; without it, the ReplayFilter can only repair after the fact.
fromJournal does not run on the read side, so projections re-implement adaptation.

Key design decisions and trade-offs

Decision	Rationale	Trade-off
Journal events, not state or code	Events cannot fail on replay; nondeterminism is confined to the once-run command handler	The application must design an event vocabulary; every state change is a schema decision
Pure fold `(State, Event) => State`	Recovery is a deterministic function of the stream; no sandbox needed	Purity is unenforced; a stray side effect fires on every recovery
Side effects after ack, at-most-once, in memory	Never act on an unpersisted decision; keep the runtime simple	At-least-once must be encoded in state by the user; crash between ack and callback loses the callback
One outstanding `WriteMessages`, stash everything else	Dense sequence numbers, ordered stream, journal writes serialized per id	Throughput per entity bounded by journal latency; stash is bounded and memory-only
Dense per-id `sequenceNr` + `writerUuid`	Cheap contiguity checks; overlapping writers are detectable	Detection is post hoc; the real guard (sharding) is a separate module
`AtomicWrite` per `persist` effect	Multi-event decisions are all-or-nothing	Journals that cannot batch reject them; the actor must handle `EventRejectedException`
Snapshot = optimisation, retention = policy	Full-replay semantics by default; bound recovery only when asked	Snapshot save stalls the actor; event deletion trades history for space and forbids `snapshot-is-optional`
`EventAdapter` + stored manifest	Old events are upcast at the boundary; the fold sees only the latest shape	Not applied on the query side; snapshots need their own adapter
`onPersistFailure` allows backoff but never resume	After a failed write it is unknown whether the event was stored	The actor restarts and replays; a slow journal turns into restart storms without careful backoff tuning
Sagas out of scope	Keep the entity model minimal; process managers are entities too	Every compensation order and trigger is application-specific code

Implications for a durable-execution library

Stamp every record with the identity of the incarnation that wrote it, and check it on read (§9). Pekko's writerUuid plus dense sequence numbers detects two overlapping writers without help from storage, and it does so even when the runtime guarantee that should have prevented it has already failed.
Offer a policy for what to do about a detected split-brain, not just an error.Fail, Warn, RepairByDiscardOld and Disabled is a richer answer than any other system gives, and the repair mode is evidence that "discard the losing writer" is a defensible automatic choice in some deployments.
A batch append primitive is worth having. AtomicWrite commits several events for one aggregate as a unit (§9), which is what an intent-and-result pair or a step-plus-compensation registration wants and what most replay engines cannot express.
Push the ambiguity into the storage contract. Requiring a plugin to fail when it does not know whether a write landed puts the uncertainty where it originates, rather than making every recovery path guess.
Event adapters at the read boundary are the mature answer to schema evolution (§5), and the detail worth copying is that an adapter may split, drop or replace one stored record. A read-time function is strictly more powerful than a version branch inside the program.
Fold before writing. Applying an event to state before it is persisted means a record the projection cannot consume never reaches the log.
Post-write side effects are at-most-once, and a library should say so. Pekko documents that a thenRun does not re-run after a restart, which makes the contrast explicit: durability of a fact is not durability of the action attached to it (§11). The stronger guarantee needs an intent record, not a callback.
Non-durable timers are a trap the model does not warn about. Scheduled messages are lost on restart, so a delay must be persisted state plus a re-scheduling decision on recovery — something replay engines with durable timers hand over for free.
A library without an out-of-process component cannot have an operator surface (§10), and pretending otherwise is worse than the honest absence. Everything an operator wants — list, inspect, resume-from, park — needs something that outlives a single process.

Sources

Pekko source at ce44fbe7a1ee2697c293e355064b7dd893914bca: EventSourcedBehavior.scala, Effect.scala, EffectImpl.scala, RequestingRecoveryPermit.scala, ReplayingSnapshot.scala, ReplayingEvents.scala, Running.scala, ExternalInteractions.scala, StashManagement.scala, RetentionCriteria.scala, EventAdapter.scala, PersistenceId.scala, AsyncWriteJournal.scala, Persistent.scala, ReplayFilter.scala, EventsByPersistenceIdQuery.scala, EventSourcedBehaviorTestKit.scala, PersistenceTestKit.scala, PersistenceTestKitPlugin.scala, ProcessingPolicy.scala, reference.conf (persistence), reference.conf (persistence-typed), AccountExampleWithEventHandlersInState.scala, AccountExampleDocSpec.scala
Pekko documentation sources in the same tree: typed/persistence.md, persistence-schema-evolution.md
Official docs (Pekko 1.7.0): Event Sourcing, Snapshotting, Schema Evolution, Testing, Persistence Query, Durable State, Release notes 1.7
Akka license at f24cb608fa88b650da8859d4604d46279f767cf0: LICENSE
Sibling pages: Temporal, this catalog's index

Foundations (Theory)

Syntax Highlighting

Durable Execution

Engines & SDKs

Event Sourcing

Theory