Sparkles

Appearance

Temporal (Go / TypeScript) ​

A durable-execution platform in which ordinary workflow code is made crash-resumable by recording every command it issues as an append-only event history on a server and re-executing the code against that history after a failure; the server owns the journal, the SDKs own replay.

FieldValue
LanguageServer: Go. SDKs surveyed: Go (go.temporal.io/sdk) and TypeScript (@temporalio/*, whose worker embeds the Rust sdk-core as a git submodule)
LicenseMIT (server, Go SDK, TypeScript SDK)
Repositorytemporalio/temporal · temporalio/sdk-go · temporalio/sdk-typescript · temporalio/documentation
Documentationdocs.temporal.io
Categorydurable-execution engine (server) + durable-execution SDK (Go, TypeScript)
Persistence modelreplay
Journal storePer-execution event history persisted by the server in a history_node table (Cassandra, MySQL, PostgreSQL, SQLite), keyed by shard, tree, branch, node and transaction id; the SDK never writes it directly
Latest releaseThe checkouts declare server ServerVersion = "1.33.0" (common/headers/version_checker.go), Go SDK SDKVersion = "1.48.0" (internal/version.go) and TypeScript worker package 1.23.0 (packages/worker/package.json)
Local clone$REPOS/temporal at 2f7ba7ce3b4afb0db9ccf7aebbf7189e2d38bbef · $REPOS/temporal-sdk-go at fee1a45426e6dd90389ac5af4a45d98a357566b4 · $REPOS/temporal-sdk-typescript at 72615f23ef735ab27ef0cc18cbf3de68f823fdc1 · $REPOS/temporal-documentation at 01c99ad49289ccd45519f113dcd05b5f6503a331

Last reviewed: September 12, 2026.

Overview ​

What it solves ​

A long multi-step program that talks to the outside world (a release pipeline, an order saga, a provisioning run) dies partway through and must resume without redoing what it already did. Temporal's answer is to split the program into a workflow (orchestration logic that must be deterministic) and activities (arbitrary side-effecting calls). The workflow never performs an action itself; it emits commands to the server, and the server turns each command into events in a durable per-execution history. When a worker crashes, any worker can fetch the history and re-run the workflow function from the top; every command the code re-issues is matched against the recorded events, every already-completed activity result is fed back from history instead of being executed again, and execution continues from the first unrecorded command. The documentation's summary (docs/encyclopedia/event-history/event-history.mdx):

"Each time your Workflow Definition makes an API call to execute an Activity or start a Timer for instance, it doesn't perform the action directly. Instead, it sends a Command to the Temporal Service. … These Commands are then mapped to Events which are persisted in case of failure. For example, if the Worker crashes, the Worker uses the Event History to replay the code and recreate the state of the Workflow Execution to what it was immediately before the crash."

Design philosophy ​

Three commitments shape everything below.

  1. Replay, not snapshot. Workflow state is never serialized. The program is its own checkpoint: history plus deterministic code reconstructs any intermediate state. The price is a hard determinism contract on workflow code, stated bluntly in the Go SDK's internal workflow interface doc (internal/internal_workflow.go): "Code of a workflow must be deterministic. It must use workflow.Channel, workflow.Selector, and workflow.Go instead of native channels, select and go. It also must not use range operation over map as it is randomized by go runtime. All time manipulation should use current time returned by GetTime(ctx) method."
  2. The server is the only writer of the journal. A command becomes an event only when the server's workflowTaskCompletedHandler accepts it (workflow_task_completed_handler.go); the SDK cannot append, edit or compact history. The SDK's escape hatch for recording arbitrary data is one event type, MarkerRecorded, which the server "will only store … and will not try to understand" (docs/references/events.mdx).
  3. Activities are at-least-once; idempotency is the user's job. The documentation recommends but cannot enforce idempotency (docs/encyclopedia/activities/activity-definition.mdx): "By design, completed Activities will not re-execute as part of a Workflow Replay. However, Activities won't record to the Event History until they return or produce an error. If an Activity fails to report to the server at all, it will be retried."

Compared with the effect-handler systems elsewhere in this topic, Temporal is closest to Effect's Effect values in spirit (a description of a computation, interpreted by a runtime) but with the interpretation split across a network boundary and with the journal, not the handler stack, as the source of truth.

How it works ​

The history model ​

An execution's journal is a list of HistoryEvent records, each with a monotonically increasing event_id, an event_time, an event_type and a type-specific attributes message. The Go SDK's replayer dispatches on every type in one switch (internal/internal_event_handlers.go, ProcessEvent); the server's mutable state exposes one Add…Event method per type (service/history/workflow/mutable_state_impl.go). The families that matter for this survey:

FamilyEventsWritten by
Workflow taskWorkflowTaskScheduled, WorkflowTaskStarted, WorkflowTaskCompleted, WorkflowTaskTimedOut, WorkflowTaskFailedServer, around each invocation of the workflow function
ActivityActivityTaskScheduled, ActivityTaskStarted, ActivityTaskCompleted, ActivityTaskFailed, ActivityTaskTimedOut, ActivityTaskCancelRequested, ActivityTaskCanceledScheduled from a command; the rest from activity worker RPCs
TimerTimerStarted, TimerFired, TimerCanceledStarted/Canceled from commands; Fired by the server
MarkerMarkerRecordedFrom a RecordMarker command; opaque to the server
Child workflowStartChildWorkflowExecutionInitiated, ChildWorkflowExecutionStarted, …Completed, …Failed, …Canceled, …TimedOut, …TerminatedInitiated from a command; the rest by the server
Execution lifecycleWorkflowExecutionStarted, …Completed, …Failed, …TimedOut, …Canceled, …Terminated, …ContinuedAsNew, …Signaled, WorkflowExecutionUpdateAccepted/Completed, UpsertWorkflowSearchAttributesMixed

Two structural facts about this journal are load-bearing. First, ActivityTaskScheduled is the command (intent), and ActivityTaskCompleted (with result, scheduled_event_id, started_event_id) is the observation (outcome); the ActivityTaskStarted event between them "is not written to History until the terminal Event … occurs" (docs/references/events.mdx). Second, each command-derived event carries workflow_task_completed_event_id, which pins it to the workflow task that produced it, so history is a sequence of [WorkflowTaskScheduled, WorkflowTaskStarted, WorkflowTaskCompleted, <command events>…, <server events>…] groups. Replay walks those groups: the SDK runs the workflow function up to the WorkflowTaskStarted boundary, collects the commands it produced, and checks them against the events that follow the matching WorkflowTaskCompleted.

On the server, events are stored as opaque encoded blobs in a history_node table; the PostgreSQL schema (schema/postgresql/v12/temporal/schema.sql):

sql
CREATE TABLE history_node (
  shard_id       INTEGER NOT NULL,
  tree_id        BYTEA NOT NULL,
  branch_id      BYTEA NOT NULL,
  node_id        BIGINT NOT NULL,
  txn_id         BIGINT NOT NULL,
  --
  prev_txn_id    BIGINT NOT NULL DEFAULT 0,
  data           BYTEA NOT NULL,
  data_encoding  VARCHAR(16) NOT NULL,
  PRIMARY KEY (shard_id, tree_id, branch_id, node_id, txn_id)
);

A tree_id/branch_id pair identifies one history branch; node_id is the first event id of a batch; txn_id orders concurrent writers. Branching exists for cross-cluster replication and reset, which is why history is a tree rather than a flat list.

The user-facing shape ​

The Go SDK (internal/workflow.go) exposes the determinism-safe vocabulary as functions over a workflow.Context:

go
func Now(ctx Context) time.Time                       // "the time when the workflow task is started or replayed"
func Sleep(ctx Context, d time.Duration) error        // a server-side timer, not time.Sleep
func Go(ctx Context, f func(ctx Context))             // a coroutine, not a goroutine
func NewSelector(ctx Context) Selector                 // instead of select
func SideEffect(ctx Context, f func(ctx Context) any) converter.EncodedValue
func MutableSideEffect(ctx Context, id string, f func(ctx Context) any, equals func(a, b any) bool) converter.EncodedValue
func GetVersion(ctx Context, changeID string, minSupported, maxSupported Version) Version
func ExecuteActivity(ctx Context, activity any, args ...any) Future
func ExecuteChildWorkflow(ctx Context, childWorkflow any, args ...any) ChildWorkflowFuture

Now is literally the replay clock: it returns wc.currentReplayTime, which is set from the event_time of each WorkflowTaskStarted event as it is processed (internal/internal_event_handlers.go). The TypeScript SDK (packages/workflow/src/workflow.ts) exposes the same surface as module functions inside a sandbox: sleep, condition, proxyActivities, executeChild, patched, deprecatePatch, uuid4, continueAsNew.

From command to event and back ​

Each SDK keeps a per-execution command buffer. In Go, every command-producing call draws a fresh id from one monotonic counter, commandsHelper.getNextID(); ExecuteActivity sets ScheduleID from it and derives ActivityID = getStringID(ScheduleID) unless the caller supplied one (internal/internal_event_handlers.go). Timers, signals, cancellations and version markers all draw from the same counter, so the n-th command-producing call in program order always gets the same id on replay. In TypeScript the same role is played by per-kind sequence numbers, activator.nextSeqs.timer++ and friends, attached to each command pushed with activator.pushCommand (packages/workflow/src/internals.ts).

When a workflow task completes, the SDK sends the buffered commands; the server's handleCommand switch converts each into exactly one event via mutable state (workflow_task_completed_handler.go):

go
switch command.GetCommandType() {
case enumspb.COMMAND_TYPE_SCHEDULE_ACTIVITY_TASK:
    historyEvent, response, err = handler.handleCommandScheduleActivity(ctx, command.GetScheduleActivityTaskCommandAttributes())
case enumspb.COMMAND_TYPE_START_TIMER:
    historyEvent, err = handler.handleCommandStartTimer(ctx, command.GetStartTimerCommandAttributes())
case enumspb.COMMAND_TYPE_RECORD_MARKER:
    historyEvent, err = handler.handleCommandRecordMarker(ctx, command.GetRecordMarkerCommandAttributes())
// … CompleteWorkflow, FailWorkflow, CancelTimer, RequestCancelActivity, StartChildWorkflow,
//   SignalExternalWorkflow, ContinueAsNew, UpsertSearchAttributes, ModifyWorkflowProperties, ProtocolMessage
}

The nondeterminism check (Go) ​

Replay produces a list of commands; the history segment that followed the corresponding WorkflowTaskCompleted is a list of events. matchReplayWithHistory zips them (internal/internal_task_handlers.go):

go
func matchReplayWithHistory(replayCommands []*commandpb.Command, historyEvents []*historypb.HistoryEvent, msgs []outboxEntry, sdkFlags *sdkFlags) error {
    di, hi := 0, 0
    for hi < len(historyEvents) || di < len(replayCommands) {
        // … skip events/commands the check ignores (server-originated events, version-marker + search-attribute pairs)
        if d == nil {
            return historyMismatchErrorf("[TMPRL1100] nondeterministic workflow: missing replay command for %s", util.HistoryEventToString(e))
        }
        if e == nil {
            return historyMismatchErrorf("[TMPRL1100] nondeterministic workflow: extra replay command for %s", util.CommandToString(d))
        }
        if !isCommandMatchEvent(d, e, msgs) {
            return historyMismatchErrorf("[TMPRL1100] nondeterministic workflow: history event is %s, replay command is %s", …)
        }
        di++; hi++
    }
    return nil
}

isCommandMatchEvent is deliberately shallow. A ScheduleActivityTask command matches an ActivityTaskScheduled event when the activity_ids agree and the last dotted segment of the activity type name agrees (lastPartOfName); a StartTimer matches a TimerStarted on timer_id alone; CompleteWorkflowExecution matches any WorkflowExecutionCompleted. Arguments, timeouts and retry policies are not compared, which is exactly the freedom the documentation grants: "you can change: The input parameters, return values, and execution timeouts of Child Workflows and Activities" (docs/encyclopedia/workflow/workflow-definition.mdx).

There is a second, earlier detector. Events are applied to per-command state machines (internal/internal_command_state_machine.go); when history delivers, say, an ActivityTaskCompleted for a command the replayed code never issued, getCommand panics with "[TMPRL1100] During replay, a matching %v command was expected in history event position %s. However, the replayed code did not produce that. Possible causes are nondeterministic workflow definition code, or an incompatible change in the workflow definition.", and an event arriving in an impossible state trips failStateTransition (internal/internal_command_state_machine.go). The comment above the check in ProcessWorkflowTask names both routes (internal/internal_task_handlers.go): "Non-deterministic error could happen in 2 different places: 1) the replay commands does not match to history events. … 2) the command state machine is trying to make illegal state transition while replay a history event (like activity task completed), but the corresponding workflow code that start the event has been removed."

What happens next is a worker policy, not a workflow outcome (internal/worker.go): BlockWorkflow (the default) "causes workflow to get stuck in the workflow task retry loop. It is expected that after the problem is discovered and fixed the workflows are going to continue without any additional manual intervention"; FailWorkflow "immediately fails workflow execution … WARNING: enabling this in production can cause all open workflows to fail on a single bug or bad deployment."

The nondeterminism check (TypeScript) ​

The TypeScript worker does not implement the matcher. Its .gitmodules pins packages/core-bridge/sdk-core to the Rust core (.gitmodules); the core owns history pagination, the command/event state machines and the check. The TypeScript side sees the verdict as an eviction: evictionReasonToReplayError maps EvictionReason.NONDETERMINISM to a DeterminismViolationError whose message reads "Replay failed with a nondeterminism error. This means that the workflow code as written is not compatible with the history that was fed in" (packages/worker/src/replay.ts).

The TypeScript determinism sandbox ​

Each workflow runs in a Node vm context (vm.ts describes the creator as making "VMWorkflows in the current isolate"; reuseV8Context, default true, shares one context across workflows for a measured "2/3 reduction in memory usage" (packages/worker/src/worker-options.ts)). Calls into the context run under isolateExecutionTimeoutMs (packages/worker/src/workflow/vm.ts). Inside, overrideGlobals rewrites the ambient nondeterminism (packages/workflow/src/global-overrides.ts):

ts
global.WeakRef = function () {
  throw new DeterminismViolationError('WeakRef cannot be used in Workflows because v8 GC is non-deterministic');
};
global.Date = function (...args: unknown[]) {
  if (args.length > 0) return new (OriginalDate as any)(...args);
  return new OriginalDate(getActivator().now);   // the activation's timestamp, not the wall clock
};
global.Date.now = function () { return getActivator().now; };
global.setTimeout = function (cb, ms, ...args) {   // becomes a StartTimer command
  const seq = activator.nextSeqs.timer++;
  activator.pushCommand({ startTimer: { seq, startToFireTimeout: msToTs(ms) } });
  …
};
Math.random = currentRandom;                        // alea PRNG seeded from the activation's randomnessSeed

injectGlobals then installs a fixed allowlist (URL, TextEncoder, assert, AbortController, a console that is silent while isReplayingHistoryEvents) (packages/worker/src/workflow/vm-shared.ts). The sandbox blocks the accidental leaks (time, randomness, GC observation, timers) but not deliberate ones: a workflow module can still import fs unless the bundler's module overrides refuse it, so the enforcement is partial.

Side effects, markers and versioning ​

SideEffect runs f once when live and records the result in a MarkerRecorded event; on replay it looks the value up by its side-effect id and panics if absent (internal/internal_event_handlers.go):

go
if wc.isReplay {
    result, ok = wc.sideEffectResult[sideEffectID]
    if !ok {
        panicIllegalState(fmt.Sprintf("[TMPRL1100] No cached result found for side effectID=%v. KnownSideEffects=%v", sideEffectID, keys))
    }
} else {
    result, err = f()
}
wc.commandsHelper.recordSideEffectMarker(sideEffectID, result, wc.dataConverter, userMetadata)

MutableSideEffect is the one place Temporal re-observes the world: when live it runs f, compares with the recorded value through the caller's equals, and records a new marker only if the value changed; when replaying it returns the recorded value without calling f, and a missing marker is a nondeterminism panic (internal/internal_event_handlers.go). GetVersion is a marker too: the first live call records versionMarkerName with the change id and chosen version and upserts a TemporalChangeVersion search attribute; on replay a missing marker yields DefaultVersion ("GetVersion for changeID is called first time in replay mode, use DefaultVersion"), and the recorded version is validated against the [minSupported, maxSupported] window the current code declares.

TypeScript's patched(id) is the same mechanism with a boolean: on replay, notifyHasPatch jobs from the core populate knownPresentPatches; patchInternal returns false if replaying without the marker, otherwise records a setPatchMarker command and returns true (packages/workflow/src/internals.ts). deprecatePatch(id) records the marker but never branches; its doc warns that mixing workers with and without the patch is "undefined" (packages/workflow/src/workflow.ts).

Beneath user versioning sits a second, SDK-internal layer: WorkflowTaskCompleted carries sdk_metadata.lang_used_flags, and both SDKs gate their own replay-affecting behaviour changes on flags read back from history (internal/internal_flags.go; packages/workflow/src/flags.ts). The TypeScript flag table is candid about why: one entry notes that "SDKs v1.11.0 and v1.11.1 were not properly writing back the flags to history, possibly resulting in NDE on replay" and works around it by inferring the flag from the recorded build id.

Activities: retries, heartbeats, idempotency ​

Activities are retried by the server under a declarative per-activity RetryPolicy; "Temporal's default behavior is to automatically retry an Activity that fails", bounded by the activity's schedule_to_close_timeout, whereas "a Workflow Execution itself is not associated with a Retry Policy by default" (docs/encyclopedia/retry-policies.mdx). Long activities must call RecordHeartbeat within HeartbeatTimeout, both to prove liveness and because "Activities must heartbeat to receive cancellations from a Temporal Service" (docs/encyclopedia/activities/activity-execution.mdx). Heartbeat details are the only activity-side progress that survives a retry; everything else is the activity's own idempotency.

Continue-As-New and history size ​

History is bounded by dynamic config (common/dynamicconfig/constants.go): limit.historySize.error at 50 MiB, limit.historyCount.error at 51,200 events, warnings at a fifth of each, and limit.historySize.suggestContinueAsNew at 4 MiB, which surfaces as suggest_continue_as_new on WorkflowTaskStarted and as continueAsNewSuggested in the Go SDK's WorkflowInfo. The escape is NewContinueAsNewError: "the current execution is ended and the new execution with same workflow ID is started automatically" (internal/error.go) with a fresh history and whatever state the workflow passes as arguments. The documentation adds a second motive (docs/encyclopedia/workflow/workflow-execution/continue-as-new.mdx): "To prevent long-running Workflows from running on stale versions of code, you may also want to Continue-as-New periodically."

Analysis ​

1. Step identity and replay matching ​

A step is identified positionally, by the order in which command-producing calls execute, and the position is materialized as an id drawn from a per-execution counter (Go: commandsHelper.getNextID() → ScheduleID/ActivityID/TimerID; TypeScript: nextSeqs.<kind>++). Matching is a two-pointer zip of the replayed command list against the recorded event list per workflow task (matchReplayWithHistory), comparing command type, the counter-derived id, and, for activities, only the last segment of the type name. Arguments are never hashed or compared. The result is that inserting or removing any command-producing call before a step shifts every later id and fails replay, while changing what a step does (its inputs, timeouts, retry policy) is invisible to the check. The documentation lists exactly which calls are command-producing and therefore "must not be reordered, added, or removed without proper Versioning techniques" (docs/encyclopedia/workflow/workflow-definition.mdx). Explicitly named steps exist only as opt-ins: a caller-supplied ActivityID, a MutableSideEffect id, a GetVersion change id, a patched id.

2. Journal versus world ​

The journal wins, unconditionally, and the world is not consulted. Replay feeds recorded ActivityTaskCompleted results back to the code; a completed activity is never re-run, and nothing re-checks whether its effect still holds. Disagreement is detected in only one direction: the code disagreeing with the journal (nondeterminism). The world disagreeing with the journal is by design invisible to the workflow and is handled at the activity boundary by retries plus idempotency: "If an Activity fails to report to the server at all, it will be retried" (docs/encyclopedia/activities/activity-definition.mdx), so the same external action can execute twice with one recorded completion. MutableSideEffect is the single reconciling primitive, and it reconciles only during live execution, never during replay. There is no rule table for observations; Temporal's position is that a workflow should not observe the world except through activities, and that activity results, once recorded, are facts.

3. Determinism enforcement ​

Discipline, backed by runtime substitution and runtime detection; never by the language. Go offers no isolation at all: the SDK substitutes workflow.Now, Sleep, Go, Channel, Selector for their stdlib counterparts and documents that the stdlib forms "must not be used in workflow code" (internal/workflow.go), but a stray time.Now() or go func() compiles and runs. TypeScript goes further with a vm context whose globals are rewritten (Date, Date.now, setTimeout, Math.random) or made to throw (WeakRef, FinalizationRegistry) (packages/workflow/src/global-overrides.ts). Both SDKs then rely on detection after the fact: the command/event zip and the state-machine transitions raise [TMPRL1100] errors, and the TypeScript replayer surfaces them as DeterminismViolationError. Detection is bounded by the check's shallowness (§1): a nondeterministic branch that happens to issue the same command sequence passes.

4. Compensation and failure handling ​

No built-in compensation in the Go or TypeScript SDKs; it is a documented pattern. The saga page (docs/design-patterns/saga-pattern.mdx) shows a []func() slice appended to before each step and drained in reverse ("register a compensation before or after each step, and run all compensations in reverse order on failure"), with two rules that matter: register the compensation before the step it undoes ("disconnectBankAccounts is registered before addBankAccount runs, so it executes even if addBankAccount failed mid-flight — its implementation must be idempotent"), and "Use a disconnected context for cancellation compensation. In Go, use NewDisconnectedContext to run compensation Activities after Workflow cancellation, since the original context is already cancelled." Compensations are themselves activities, so they are journaled and retried like any other step; the LIFO order is reconstructed by replay because the slice is rebuilt deterministically. Only the Java SDK ships a Saga class. Failure handling proper is layered: activity retries (server, declarative), workflow task retries (unbounded, exponential, for panics and nondeterminism under BlockWorkflow), and workflow-level RetryPolicy (opt-in, uncommon).

5. Versioning against old histories ​

Marker-based branching, plus deployment-level pinning. GetVersion/patched record a marker at the first live call and read it back on replay, so old histories take the old branch and new executions take the new one; the recorded version is validated against the window the current code supports, which is how a too-old history fails loudly rather than silently (internal/internal_event_handlers.go). The documentation's patched() semantics page spells out the sharp edges (docs/encyclopedia/workflow/patching.mdx): a marker that appears later in history than the current call is a nondeterminism error, and "if there is no marker for a given patch ID, the execution will return false and will not add a marker to the event history. In addition, all future calls to patched() with that ID will return false -- even after it is done replaying and is running new code." The lifecycle is three deploys: add patched, replace with deprecatePatch, remove. Orthogonally, Worker Versioning assigns each worker a deployment name plus build id and each workflow type a behaviour: "A Pinned Workflow is guaranteed to complete on a single Worker Deployment Version", while auto-upgrade workflows "need to be kept replay-safe manually, that is with patching" (docs/encyclopedia/workers/worker-versioning.mdx; Go WorkerDeploymentOptions{UseVersioning, Version, DefaultVersioningBehavior} in internal/worker.go). Continue-As-New is the third tool: a fresh history has no old code to be compatible with.

6. Concurrency under replay ​

Cooperative, single-threaded, deterministically scheduled. The Go SDK's dispatcher "executes coroutines one by one in deterministic order until all of them are completed or blocked on Channel or Selector" (internal/internal_workflow.go); workflow.Go creates a coroutine on that dispatcher, and ExecuteUntilAllBlocked loops over the coroutine slice in order, inserting eagerly-created children right after their parent. Selector.Select iterates its cases in registration order (for _, pair := range s.cases) even though the interface comment says an eligible branch is "picked randomly", a discrepancy worth knowing when reading that doc. Because the interleaving is a pure function of the event order, concurrent steps replay by re-running the same schedule. TypeScript gets the same property from JavaScript's single thread and microtask queue, at the cost that the order in which the SDK dispatches activation jobs becomes part of the replay contract: ProcessWorkflowActivationJobsAsSingleBatch exists because reordering signal dispatch relative to other jobs changed promise interleavings and hence command order (packages/workflow/src/flags.ts). Unblocked concurrent activities are matched by id, so their completion events may arrive in any order.

7. Replay or snapshot ​

Replay, with a cache to avoid paying for it. A worker keeps the live workflow in a sticky cache and the server routes later tasks to it; only on eviction or crash is the full history fetched and replayed (docs/encyclopedia/workflow/workflow-execution/workflow-execution.mdx). Replay cost is linear in history, so history is capped (51,200 events, 50 MiB) and the server nudges the workflow to Continue-As-New at 4 MiB. Replay rules out: any state not reconstructible from history (hence the sandbox), any code change that alters command order (hence markers), and any history larger than the cap (hence Continue-As-New, which is a manual snapshot expressed as workflow arguments). What it buys: the journal is small and typed (intents and outcomes, not state), any worker can resume any execution, and the same history is a regression test.

8. Testing ​

Three tiers. Replay tests: worker.NewWorkflowReplayer().ReplayWorkflowHistory(...) in Go (worker/worker.go) and Worker.runReplayHistory(options, history) in TypeScript, which "Will resolve as soon as the history has finished being replayed, or if the workflow produces a nondeterminism error" (packages/worker/src/worker.ts); the documentation recommends downloading "a representative set of recent open and closed Workflows" and failing CI on any replay error (docs/develop/go/best-practices/testing-suite.mdx). Unit tests with a mock clock: Go's TestWorkflowEnvironment runs the workflow against a clock.Mock that "automatically move[s] forward to fire next timer when workflow is blocked" (internal/workflow_testsuite.go), with RegisterDelayedCallback to inject signals at workflow-clock times. Integration tests with a real server: TypeScript's TestWorkflowEnvironment.createTimeSkipping runs the Java time-skipping test server, where "Time skipping, which is automatically done when awaiting a workflow result and manually done on sleep, is global to the environment", and createLocal runs the real dev server without time skipping (packages/testing/src/testing-workflow-environment.ts). There is no crash-at-every-event test primitive; the replayer is the closest thing, and it tests compatibility of code with a finished history rather than resumption from each prefix.

9. Journal integrity and the single writer ​

The history is server-owned, and its integrity rests on two fencing tokens at different scopes plus one predicate that admits what a failed write does not tell you.

A shard lease fences every write. Each workflow execution belongs to a shard, and a shard is owned by exactly one history host holding a monotonically increasing RangeID. Every write carries that token: AddHistoryTasksRequest and AssertShardOwnershipRequest both take a RangeID, and UpdateShardRequest carries a PreviousRangeID so acquiring a shard is itself a conditional update (data_interfaces.go). The failure mode is named after the mechanism: "ShardOwnershipLostError is returned when conditional update fails due to RangeID for the shard". A host that has lost its lease cannot write, whether or not it has noticed.

A per-execution version fences the mutable state. GetWorkflowExecution returns a DBRecordVersion alongside the state, and an update supplies it back; a mismatch is a WorkflowConditionFailedError. The older mechanism it replaces is visible in the same file, where NextEventID is marked "TODO deprecate NextEventID in favor of DBRecordVersion" — that is, the conditional append moved from "the history is exactly this long" to an opaque record version (data_interfaces.go).

Appends are atomic across history and tasks. One transaction writes the new history events, the mutable-state update and the transfer or timer tasks that follow from them, which is why a completed workflow task never leaves a scheduled activity unrecorded. The transaction is size-bounded, and exceeding the bound is its own error (TransactionSizeLimitError).

Whether a failed write committed is explicitly modelled.OperationPossiblySucceeded enumerates the errors that mean "Persistence failure that means that write was definitely not committed" — CurrentWorkflowConditionFailedError, WorkflowConditionFailedError, ConditionFailedError, ShardOwnershipLostError, AppendHistoryTimeoutError and a few more — and returns true for everything else (error_type.go). That distinction is exactly the started-without-completed ambiguity, made a first-class predicate rather than left to a caller's judgement, and it is the piece most systems in this survey leave implicit.

Duplicate work is prevented by the task token, not by the journal. A worker completes a workflow task by presenting the token it was handed; a token from a superseded attempt is rejected, so two workers racing on the same task cannot both append. Identity of the writing attempt is therefore checked, but it is checked at the task boundary rather than stamped on each record.

10. Operator recovery and intervention ​

Temporal has the richest operator surface in the survey, and the reason is structural: because the history is a first-class server-side object with an API, every intervention is an API call rather than a database edit.

Reset is fork-from-a-chosen-event, and it is first-class.ResetWorkflowExecution takes an event id to reset to and starts a new run that inherits the history up to that point (resetworkflow/api.go). What makes it more than a truncation is the reapply policy: ResetReapplyType and ResetReapplyExcludeTypes decide which post-reset-point external inputs — signals, updates — are replayed onto the new run rather than lost. A library that offers only "truncate here" leaves the caller to rediscover that signals received after the reset point are still owed to the program.

Cancel and terminate are deliberately different operations. CancelWorkflow delivers a cancellation the program can observe and handle, so cleanup runs; TerminateWorkflow takes a reason and stops the execution without giving it a turn (client.go). The distinction matters precisely because compensation is user code: only the first gives it a chance to run.

Inspection is a first-class API, not a database query. History can be fetched or long-polled event by event, an execution described, and both are public client operations. This is the affordance that library designs backed by opaque storage give up.

Signals and updates are also intervention tools. Because any external process can send a signal or an update to a running execution, an operator can supply a value the program is waiting for without touching storage — the same mechanism the program's own peers use.

Undeliverable work goes to a dead-letter queue. The history service exposes getdlqtasks and deletedlqtasks APIs, so a task that cannot be processed is parked and inspectable rather than retried forever or dropped.

What is not offered: editing a recorded activity result in place. The supported route to "that step recorded the wrong thing" is reset to before it and re-run, which keeps the record append-only at the cost of a new run id.

11. Suspension and external input ​

Waiting ends the worker's turn, not the execution. A workflow task completes with commands and the worker stops running that execution; nothing is held except an optional sticky cache entry, which is a performance optimisation and not a correctness requirement. There is no threshold below which a wait is held in memory — the shortest timer and the longest both become history events.

The primitives are timers, signals, updates, queries and child executions. Durable timers are TimerStarted/TimerFired pairs in the history (timers and delays); signals and updates arrive as history events and are surfaced to the program through channels and handlers (message passing). Await and AwaitWithTimeout block a workflow coroutine until a predicate over already-received input holds (workflow.go), which is how a program waits on a condition rather than on a specific event.

There is no distinct persisted "suspended" state. An execution waiting on a signal is Running, exactly like one whose activity is in flight. That is a deliberate simplification — the history says what is outstanding — but it means a caller cannot ask "is this blocked on me?" without reading the history and interpreting it.

External input is addressed by workflow id and signal name, and delivery is at-least-once, so a duplicate signal is a second history event rather than a deduplicated no-op. Deduplicating is the program's job, and signal-with-start exists precisely because "deliver this, creating the execution if needed" is otherwise a race. An input that arrives before the program reaches its handler is buffered in the history; one that never arrives leaves the execution running until its own timeout fires.

Human-in-the-loop is a documented pattern built from these parts — a timer raced against a signal, with the timer as the escalation path — rather than a dedicated construct.

History size bounds how long a wait can last. An execution accumulates events while it waits, and the documented limits (limits) are what eventually force Continue-As-New. A program that waits on a human for a month is therefore a design problem, not just a long timer.

Strengths ​

  • The journal is intents and outcomes, not state. Every event is a small typed record with an id; the schema doubles as an audit log and a debugger input, and the replayer is the same code path as production.
  • Two independent detectors for drift (the command/event zip and the state-machine transitions), both with the searchable [TMPRL1100] tag.
  • Markers make versioning a data problem. GetVersion/patched write the decision into the journal at the first live call, so the old branch is provable, not inferred.
  • Deterministic scheduling is owned by the SDK, so workflow.Go/Selector replay by construction and concurrent activities are matched by id, not by completion order.
  • Retries, heartbeats and timeouts are declarative and server-enforced, which keeps the activity-side contract to one word: idempotent.
  • History caps plus suggest_continue_as_new turn an unbounded-replay failure mode into a signal the workflow can act on.

Weaknesses ​

  • Positional identity is brittle. Inserting one sleep before a step renumbers everything after it; the fix is a marker at every insertion point, forever, until every old history is gone.
  • The match is shallow. Arguments are not compared, so a nondeterministic branch that yields the same command shape is undetected; conversely the activity-type match uses only the last dotted name segment.
  • The world is never re-observed. A recorded activity completion is a fact even if the effect was later undone; there is no reconciliation rule beyond "make the activity idempotent".
  • No compensation primitive in Go/TypeScript. The LIFO slice is a convention, and its correctness under cancellation depends on remembering NewDisconnectedContext.
  • Enforcement is partial. Go has no sandbox; TypeScript's sandbox patches globals but relies on bundler discipline to keep fs and process out.
  • BlockWorkflow means silent stalls. The default response to a nondeterminism error is to retry the workflow task forever until a deploy fixes it.
  • The SDK's own behaviour changes are versioned by flags in history, and the TypeScript flag table records at least one release that failed to write them, which then had to be inferred from build ids.

Key design decisions and trade-offs ​

DecisionRationaleTrade-off
Server owns the journal; SDK only proposes commandsAny worker can resume any execution; the server can validate, cap and replicate historyEvery step is a network round-trip; MarkerRecorded is the only SDK-controlled record
Positional step ids from one counterZero annotation burden; ids are stable for unchanged codeAny inserted command-producing call invalidates all later ids; versioning markers are the only remedy
Shallow command/event match (type + id + last name segment)Lets inputs, timeouts and retry policies change without a version bumpCannot detect a branch that happens to issue the same command shape with different meaning
Activities at-least-once, results journaled on completionSimple activity contract; retries are server-side and declarativeDuplicate side effects are the user's problem; no reconciliation of journal against world
SideEffect/MutableSideEffect markers for in-workflow nondeterminismKeeps short observations out of the activity machineryValue is captured once; MutableSideEffect re-observes only when live
TypeScript vm sandbox with rewritten globalsCatches accidental time/random/GC leaks at runtime with a clear errorNot a security boundary; reuseV8Context shares a context for performance
Go: cooperative coroutine dispatcher instead of goroutinesDeterministic interleaving without OS schedulingA whole parallel vocabulary (workflow.Go, Channel, Selector) that the compiler does not enforce
Nondeterminism default = BlockWorkflowA bad deploy can be rolled back without losing executionsWorkflows stall silently until someone notices the failing-task metric
History caps + Continue-As-NewBounds replay time and storageThe user must design a checkpoint-as-arguments shape and pick when to cut
Worker Versioning with pinned/auto-upgrade behavioursOld code keeps running old executions; no patching needed for pinned typesOperational surface (deployments, ramps, drain states) and a still-shifting API (BuildID already deprecated)

Implications for a durable-execution library ​

  • Two fencing tokens at two scopes is the strongest integrity design in the survey (§9): a monotonic shard lease on every write, and a per-execution record version on the state. A library with one journal still needs both ideas — something that stops a stale process writing at all, and something that makes an individual append conditional.
  • Model "did that failed write commit?" explicitly. OperationPossiblySucceeded enumerates the errors that mean the write definitely did not land and treats everything else as possibly landed. Most systems leave that judgement to a caller; making it a predicate is what lets recovery be correct rather than hopeful.
  • Reset-to-an-event with a reapply policy is the operator affordance to copy (§10). Truncating the record is the easy half; deciding which external inputs received after the reset point are still owed to the new run is the half that makes it usable, and Temporal is the only system here that models it.
  • Cancel and terminate must be different operations. One gives the program a turn so its cleanup runs, the other does not. A library that offers only "terminate" has silently decided that compensation is optional.
  • Positional identity works, and its cost is visible in the mechanisms built to survive it: patch markers, worker pinning by build id, and a deliberate refusal to compare arguments so that inputs may change without a version bump (§1, §5). That last choice is defensible and should be made consciously.
  • Substituting the nondeterministic primitives is necessary but not sufficient. Temporal replaces the clock, the scheduler and the random source, and still needs an after-the-fact mismatch error, because a host language has more nondeterminism than a runtime can intercept (§3).
  • A history that can be fetched and long-polled is a product feature, and it is what makes replay-against-stored-histories a practical CI gate (§8). A library whose record is opaque gives up both.
  • There is no distinct persisted "waiting" state (§11), which is a real cost: a caller cannot distinguish "blocked on me" from "working" without reading and interpreting the record.

Sources ​