Hatchet v0.114.2 adds rate limit pausing, RetryAfter errors, and FIPS images. It also includes a set of security fixes and a large batch of DAG, scheduler, and database fixes.
Highlights
Rate limit pause: a rate limit key can be paused. Tasks that use the key wait, and do not consume the limit, until the pause ends (#5209).
RetryAftererrors: a task can return aRetryAfterErrorto choose when its next retry runs. See Retry Policies (#5162).FIPS images: every server image is also published with a
-fipssuffix, built against the validated Go Cryptographic Module. See FIPS 140-3 Images (#4997).Workflow pause in the Go SDK: workflows can now be paused and unpaused from the Go SDK (#5025).
Dashboard: the trigger dialog has a Form tab built from the workflow's input schema, the runs table has a "Replay as new" action that re-triggers a run with its original input, and the workflows page shows triggers, description, and last deploy (#5183, #5140, #5242).
Engine: the lookup, DAG data, and DAG-to-task tables in the core and OLAP databases are now partitioned by time, like the task tables already were (#5038, #5050).
Engine: payloads are stored once and shared between events, DAGs, and tasks instead of being written several times (#5196, #5198, #5224).
Self-hosting: default entitlements for new tenants can be set with env vars (#5266).
Security
Build: Go 1.26.10, newer base images, and fixed Node releases block the compromised eMudhra CAs (CVE-2026-111943) (#5256, #5258, #5275).
Auth: an API token's JWT subject must match the tenant stored for that token (#5215).
Engine: SNS message verification is stricter. Signed fields other than
Messagemay not contain newlines, the signing certificate URL must be a regional SNS host with a.pempath, outbound requests have a timeout and a 1 MiB response cap, and messages signed more than an hour ago are rejected (#4926, #5279).Engine: worker slot units must be zero or more and are summed without overflow, so a worker cannot skip the tenant slot limit (#5274).
API: listing workflow runs with payloads checks payload permissions before querying (#5205).
Fixed
Engine: DAG replays no longer hang or retry forever when a running task is replayed, a child is spawned twice, or tasks finish out of order (#5186, #5189, #5207, #5265).
Engine: the DAG operator no longer claims idempotency keys that its children need, no longer reports a cycle when a task shares its workflow's name, and no longer checks idempotency keys for DAG steps (#5136, #5137, #5211).
Engine: task errors are passed to the on-failure task, returned verbatim without a
dag failed:prefix, and internal errors no longer leak into task run errors (#5178, #5302, #5212).Engine:
skip_ifandcancel_ifconditions are evaluated when a parent returns empty output, empty parent outputs still appear in theparentsmap, and an invalid CEL condition is ignored instead of failing the DAG (#5295, #5296).Engine: a failing CEL expression no longer causes a flood of retries (#5151).
Engine: the DAG operator is registered for workflows put by very old SDKs, skips deleted tenants, and is metered once per run instead of once per step (#5203, #5177, #5158).
Engine:
CANCEL_*concurrency strategies no longer cancel a task's own retry (#5229).Engine: when a later concurrency batch fails, results from batches that already committed are still published, so cancelled tasks get their cancellation and filled tasks are not delayed (#5155).
Engine: the in-memory concurrency index uses much less memory per key, and its build timeout is 30s (#5156, #5214, #5228).
Engine: idle queues go inactive again instead of being polled forever, the rate limit requeue runs without a transaction and only when there is work, and
REQUEUED_RATE_LIMITevents are reported at most once a minute per task (#5144, #5138, #5245, #5170).Engine: retry decisions for a batch of failed tasks match each task to its own row. Large batches previously got wrong results or timed out and never retried (#5243).
Engine: the trigger path caches event-to-workflow lookups, workflow versions, step expressions, and match conditions, and runs its preflight checks before opening a transaction (#5251, #5259, #5165).
Engine: cron and scheduled pollers no longer rewrite rows they already own on every poll (#5301).
Engine: new indexes for DAG orchestration action lookups, stuck evicted durable orchestrators, and cron name search, and a redundant OLAP runs index is dropped (#5288, #5160, #5267, #5090).
Engine: the durable task connection runs one worker-status handler per worker, fixing an out-of-memory crash when a worker stops reading, and a
wait_forcondition without a base returns an error instead of crashing the handler (#5201, #5260).Engine: statement timeouts now apply behind a transaction-mode PgBouncer, with a startup warning when they are missing. See Using PgBouncer (#5065, #5218).
Engine: the Go memory limit is set to 90% of the container limit to reduce OOM kills (#5180).
Engine: orchestrator terminal events are written to OLAP once, retried tasks are counted by their final status in the workflow duration metric, and
is_dag_orchestratoris recorded on task events (#5161, #5176, #5199).Engine: task durations were sometimes matched to the wrong task (#5227).
Engine: empty and
nullpayloads are no longer written to the payloads table (#5252, #5253).Engine: authentication returns a 5xx error instead of "invalid token" when the database is unreachable (#5173).
Engine: analytics events from in-process callers carry the tenant id (#5120).
API: listing workflow runs with payloads returns 400 above 1,000 runs or 1,000 DAG child tasks, and DAG outputs are no longer duplicated per child task (#5205).
Dashboard: run counts load separately from the runs list, which makes the runs page much faster for tenants with many runs (#5164).
Dashboard: workflow search UX fixes, the Manage Billing button is hidden on legacy plans, and the support chat widget follows the app theme (#5036, #5150, #5219).
Go SDK: waiting for a workflow result ends when the caller's context ends, and
WithTokenoverrides the API token for the OTel collector (#5234, #5217).Go SDK: durable status polling fixes, and shutdown and evict acks are serialized (#5128).
What's Changed
fix(engine): dag operator cycles from task and workflow sharing names by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5137
fix(dashboard): workflow search ux tweaks by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5036
fix(ts, ruby, go): durable status polling fixes, serialize shutdown and evict acks by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5128
fix(engine): dag operator causes idempotency key collisions by default by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5136
feat(go,ts,rb): add workflow pause to the rest of the sdks by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5025
fix(operatorsvc): set the analytics tenant id on in-process contexts by @grutt in https://github.com/hatchet-dev/hatchet/pull/5120
perf(engine): requeue rate limited items without a transaction by @grutt in https://github.com/hatchet-dev/hatchet/pull/5138
fix(ts): shutdown causes heartbeat error flood in logs by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5143
fix(scheduler): let idle queues go inactive again by @igor-kupczynski in https://github.com/hatchet-dev/hatchet/pull/5144
chore: dependabot fixes by @mnafees in https://github.com/hatchet-dev/hatchet/pull/5146
fix: do not sign tags when originated from Taskfile by @mnafees in https://github.com/hatchet-dev/hatchet/pull/5148
feat(engine): skip stale NATS pub/sub messages instead of working through a backlog by @igor-kupczynski in https://github.com/hatchet-dev/hatchet/pull/5103
feat(engine): run NATS pub/sub handlers on a bounded per-subscription pool by @igor-kupczynski in https://github.com/hatchet-dev/hatchet/pull/5104
fix: Consolidate entitlement logic by @juliusgeo in https://github.com/hatchet-dev/hatchet/pull/5149
fix(frontend): hide Manage Billing button on legacy plans by @jbsouvestre in https://github.com/hatchet-dev/hatchet/pull/5150
fix(engine): retry flood on CEL evaluation failures in DAG operator by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5151
feat(durable-streams): large messages, metadata by @juliusgeo in https://github.com/hatchet-dev/hatchet/pull/5139
perf(olap): drop ix_v1_runs_olap_tenant_ins_at_status by @igor-kupczynski in https://github.com/hatchet-dev/hatchet/pull/5090
feat(engine): partition core tables by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5038
fix(engine): charge DAG-operator runs once for TASK_RUN by @igor-kupczynski in https://github.com/hatchet-dev/hatchet/pull/5158
fix(engine): run analyze in migration by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5159
fix(engine): avoid seq scan on ListStuckEvictedDurableOrchestrators by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5160
test(python): couple small fixes to python tests by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5163
feat: partition more olap tables by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5050
fix: throttle
REQUEUED_RATE_LIMITmonitoring events to once per task per minute by @mnafees in https://github.com/hatchet-dev/hatchet/pull/5170fix(engine): stop double-writing orchestrator terminal events to OLAP by @igor-kupczynski in https://github.com/hatchet-dev/hatchet/pull/5161
fix(dashboard): split count queries to improve page performance by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5164
feat: add RetryAfter(Exception|Error) to SDKs and engine by @juliusgeo in https://github.com/hatchet-dev/hatchet/pull/5162
fix(engine): return 5xx if we fail to reach the db on auth by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5173
fix(python): re-add log queue max size by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5175
fix(engine): keep committed concurrency results when a later batch fails by @igor-kupczynski in https://github.com/hatchet-dev/hatchet/pull/5155
fix: ignore deleted tenants in DAG operator claim by @mnafees in https://github.com/hatchet-dev/hatchet/pull/5177
fix(engine): dag operator not propagating task run errors back to on failure task by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5178
chore: Bump github.com/rabbitmq/amqp091-go from 1.13.0 to 1.14.0 in /examples/go/embedded by @dependabot[bot] in https://github.com/hatchet-dev/hatchet/pull/5174
feat(dashboard): add Replay as new action to the runs table by @jbsouvestre in https://github.com/hatchet-dev/hatchet/pull/5140
fix: SNS ingestor outbound requests hardening by @mnafees in https://github.com/hatchet-dev/hatchet/pull/4926
feat: publish
-fipsimages built against the validated Go crypto module by @mnafees in https://github.com/hatchet-dev/hatchet/pull/4997fix: automemlimit by @jbsouvestre in https://github.com/hatchet-dev/hatchet/pull/5180
fix(engine): dag replays causing infinite retry loop by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5186
fix(engine): dag replays, part ii by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5189
test(loadtest): add a concurrency strategy workload by @igor-kupczynski in https://github.com/hatchet-dev/hatchet/pull/5192
chore(frontend): add knip and remove unused dependencies by @jbsouvestre in https://github.com/hatchet-dev/hatchet/pull/5191
perf(engine): shrink the in-memory concurrency index by @igor-kupczynski in https://github.com/hatchet-dev/hatchet/pull/5156
feat(durable-streams): add Reset method for durable streams by @juliusgeo in https://github.com/hatchet-dev/hatchet/pull/5194
chore: reporters need to mention if they used AI to find security findings by @mnafees in https://github.com/hatchet-dev/hatchet/pull/5202
fix(dispatcher): run one durable worker-status handler per stream by @igor-kupczynski in https://github.com/hatchet-dev/hatchet/pull/5201
fix(engine): dag operator not registered on very old SDKs by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5203
fix(engine): statement timeouts behind transaction poolers by @igor-kupczynski in https://github.com/hatchet-dev/hatchet/pull/5065
fix(engine): wire up
is_dag_orchestratorcol on the task events by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5199chore(engine): duped migration version by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5206
chore(engine): improve in-memory concurrency index visibility by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5197
fix(engine): durable dags hang on replays of running tasks by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5207
fix(engine): don't check idempotency keys for dag steps by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5211
fix(engine): stop leaking errors from the operator by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5212
fix(api): cap tasks per workflow-run list read that includes payloads by @igor-kupczynski in https://github.com/hatchet-dev/hatchet/pull/5205
fix(auth): verify JWT subject matches stored API token tenant by @igor-kupczynski in https://github.com/hatchet-dev/hatchet/pull/5215
chore: Bump the go-modules group across 1 directory with 20 updates by @dependabot[bot] in https://github.com/hatchet-dev/hatchet/pull/5182
chore: Bump the npm-dev-deps group across 2 directories with 4 updates by @dependabot[bot] in https://github.com/hatchet-dev/hatchet/pull/5193
feat(go-sdk): option to override API token for Hatchet OTel collector by @mnafees in https://github.com/hatchet-dev/hatchet/pull/5217
fix: bump index build timeout for concurrency Strats by @juliusgeo in https://github.com/hatchet-dev/hatchet/pull/5214
feat(frontend): sync Pylon chat widget theme with app theme by @jbsouvestre in https://github.com/hatchet-dev/hatchet/pull/5219
fix: cross-batch terminal events to OLAP with retried tasks would cause incorrect Prom metrics by @juliusgeo in https://github.com/hatchet-dev/hatchet/pull/5176
chore(engine): add exhaustiveness and sum type checking linters by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5221
feat(dashboard): JSON Schema form for workflow trigger input by @jbsouvestre in https://github.com/hatchet-dev/hatchet/pull/5183
chore: Bump github.com/getkin/kin-openapi from 0.144.0 to 0.149.0 in /examples/go/embedded by @dependabot[bot] in https://github.com/hatchet-dev/hatchet/pull/5223
fix:
GetTaskDurationsByTaskIdscould get wrong row for taskId by @juliusgeo in https://github.com/hatchet-dev/hatchet/pull/5227fix: set concurrency index creation timeout to 30s by @juliusgeo in https://github.com/hatchet-dev/hatchet/pull/5228
chore: dependabot fixes by @mnafees in https://github.com/hatchet-dev/hatchet/pull/5230
fix(test): reduce flake on cancellation during engine startup by @abelanger5 in https://github.com/hatchet-dev/hatchet/pull/5231
fix(test): flake on sending action deltas in operator client by @abelanger5 in https://github.com/hatchet-dev/hatchet/pull/5232
fix: match each failed task to its own locked row when deciding retries by @igor-kupczynski in https://github.com/hatchet-dev/hatchet/pull/5243
fix: cancel_* concurrency strategies could cancel a task's own retry by @juliusgeo in https://github.com/hatchet-dev/hatchet/pull/5229
chore(engine): bump pgoutbox by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5244
test: use
timeout_func_only = truefor pytest by @abelanger5 in https://github.com/hatchet-dev/hatchet/pull/5247fix(go-sdk): end the workflow result wait when the caller's context ends by @abelanger5 in https://github.com/hatchet-dev/hatchet/pull/5234
perf(scheduler): don't run the rate limit requeue on every queue iteration by @abelanger5 in https://github.com/hatchet-dev/hatchet/pull/5245
fix(docs): return 405 for SSE GET requests on stateless MCP endpoint by @grutt in https://github.com/hatchet-dev/hatchet/pull/5249
feat(frontend): add Storybook to the dashboard app by @jbsouvestre in https://github.com/hatchet-dev/hatchet/pull/5250
feat(dashboard): show triggers, description and last deploy on the workflows page by @jbsouvestre in https://github.com/hatchet-dev/hatchet/pull/5242
fix(dev): set statement timeouts through local PgBouncer by @igor-kupczynski in https://github.com/hatchet-dev/hatchet/pull/5218
test(typescript): deflake durable eviction test by @abelanger5 in https://github.com/hatchet-dev/hatchet/pull/5246
refactor(engine): simplify task data shape by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5198
fix(engine): don't write null-string payloads for task inputs by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5252
fix(engine): remove one more source of
null::jsonbpayloads by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5253perf: use caches properly in the trigger path by @abelanger5 in https://github.com/hatchet-dev/hatchet/pull/5251
chore: bump Go to 1.26.10 for eMudhra CA distrust by @igor-kupczynski in https://github.com/hatchet-dev/hatchet/pull/5256
refactor(engine): typed payload shapes by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5196
chore: upgrade ca-certificates on publishing CI jobs (eMudhra distrust) by @igor-kupczynski in https://github.com/hatchet-dev/hatchet/pull/5258
feat: serverless operator by @abelanger5 in https://github.com/hatchet-dev/hatchet/pull/4912
refactor(engine): shared payloads by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5224
fix(engine): handle durable wait_for conditions without base by @igor-kupczynski in https://github.com/hatchet-dev/hatchet/pull/5260
perf: add some more caching to the trigger path by @abelanger5 in https://github.com/hatchet-dev/hatchet/pull/5259
fix: exhaustive checks from serverless operator by @juliusgeo in https://github.com/hatchet-dev/hatchet/pull/5261
fix: missing eventKeyWorkflowsCache in task_test.go by @juliusgeo in https://github.com/hatchet-dev/hatchet/pull/5263
fix(ts-core): properly export
HatchetErrorand redirect settings by @abelanger5 in https://github.com/hatchet-dev/hatchet/pull/5264feat(engine): add env vars for default tenant entitlements by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5266
ci: use fixed Node in publish jobs for eMudhra CA distrust by @igor-kupczynski in https://github.com/hatchet-dev/hatchet/pull/5275
fix(engine): tighten SNS signature verification by @igor-kupczynski in https://github.com/hatchet-dev/hatchet/pull/5279
fix(engine): reject negative worker slot units by @igor-kupczynski in https://github.com/hatchet-dev/hatchet/pull/5274
fix(engine): replays of dag tasks can cause hang by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5265
feat(engine): index cron names for search by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5267
docs: list Sydney as a live data plane region by @grutt in https://github.com/hatchet-dev/hatchet/pull/5284
feat: add durable streams to the core SDK by @abelanger5 in https://github.com/hatchet-dev/hatchet/pull/5282
refactor: add
read_payloadsas sql method which can be shared across queries by @abelanger5 in https://github.com/hatchet-dev/hatchet/pull/5281feat(engine): rate limit pause by @juliusgeo in https://github.com/hatchet-dev/hatchet/pull/5209
fix(db): index DAG orchestration action lookup by @grutt in https://github.com/hatchet-dev/hatchet/pull/5288
fix: all open Dependabot alerts by @grutt in https://github.com/hatchet-dev/hatchet/pull/5287
fix(engine): [dag operator] return outputs even when empty, and ignore invalid CELs on conditions by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5295
fix(engine): evaluate dag conditions against empty parent output by @igor-kupczynski in https://github.com/hatchet-dev/hatchet/pull/5296
fix(engine): return errors out of the dag operator verbatim by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5302
chore(engine): fix migration numbering by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5303
perf(ticker): stop rewriting already-owned cron and scheduled rows on every poll by @grutt in https://github.com/hatchet-dev/hatchet/pull/5301
feat: REST API,
refreshTimeout, andreleaseSlotin the/corepackage by @abelanger5 in https://github.com/hatchet-dev/hatchet/pull/5292feat: add durable streams to the serverless operator by @abelanger5 in https://github.com/hatchet-dev/hatchet/pull/5293
fix(ts): durable stream shutdown, worker label propagation by @mrkaye97 in https://github.com/hatchet-dev/hatchet/pull/5304
fix: allow batching messages on publish for durable streams by @juliusgeo in https://github.com/hatchet-dev/hatchet/pull/5308
fix: accidentally broke backwards compatibility with engine by @juliusgeo in https://github.com/hatchet-dev/hatchet/pull/5311
fix: bump typescript SDK version by @juliusgeo in https://github.com/hatchet-dev/hatchet/pull/5312
chore: Bump the npm-dev-deps group across 1 directory with 9 updates by @dependabot[bot] in https://github.com/hatchet-dev/hatchet/pull/5324
chore: Bump the github-actions group across 1 directory with 7 updates by @dependabot[bot] in https://github.com/hatchet-dev/hatchet/pull/5323
chore: Bump zod from 4.3.6 to 4.6.6 in /frontend/docs by @dependabot[bot] in https://github.com/hatchet-dev/hatchet/pull/5306
chore: Bump the go-modules group across 1 directory with 18 updates by @dependabot[bot] in https://github.com/hatchet-dev/hatchet/pull/5322
chore: Bump github.com/labstack/echo/v4 from 4.15.4 to 4.16.1 in /sdks/go/examples/embedded by @dependabot[bot] in https://github.com/hatchet-dev/hatchet/pull/5310
refactor(engine): run trigger preflight before the owned transaction by @igor-kupczynski in https://github.com/hatchet-dev/hatchet/pull/5165
fix: dependabot alerts by @mnafees in https://github.com/hatchet-dev/hatchet/pull/5325
feat(durable-streams): add a frontend UI by @juliusgeo in https://github.com/hatchet-dev/hatchet/pull/5285
feat(engine): add fairpool connection gate by @igor-kupczynski in https://github.com/hatchet-dev/hatchet/pull/5166
chore: changelog entries for next OSS release by @mnafees in https://github.com/hatchet-dev/hatchet/pull/5329
Full Changelog: https://github.com/hatchet-dev/hatchet/compare/v0.110.5...v0.114.2