5.4 Release Notes
Patch Releases
All patch release notes for 5.4.x are available on the releases page.
Upgrade Notes
Full-text indexes using fulltext 0.5.0 with current field-based declarations, or prerelease 0.6, rebuild locally after this upgrade. Plan for search unavailability and review the Tantivy upgrade requirements before deploying; direct downgrade to fulltext 0.5.x is unsupported.
An OIDC exchange now refuses a token that two enabled trust policies give a different user or different operations, where before the first policy by id answered (see Conflicting Trust Policies Are Refused). Before upgrading, run list_oidc_trust and look for two enabled policies, with the same issuer and audience, whose claims can match the same runs but which name different users or operations, such as the policies of two harper deploy setup=true provider=github-actions runs that share a workflow, branch and environment. Drop, disable or narrow one of each pair, or the runs both match stop authenticating. A 5.4 node's list_oidc_trust names each pair in invalid_reason. Upgrade every node: a node still running 5.3 keeps answering with the first policy by id.
Harper 5.4 marks failed transactions as aborted in the RocksDB transaction log by default (storage.rocks.transactionLogAbortMarkers; see Failed Transactions Are Marked Aborted). Harper 5.3.4 and earlier cannot read those markers and would treat the failed transactions as committed. Do not downgrade a node to them, or open its database files or backups with them, after it has run 5.4 with the setting on. Harper 5.3.5 and later read the markers. Setting it to false later stops new markers but does not remove ones already written.
- Upgraded instances keep their
applications.lockdownmode. Only new installs get the newseal-after-loaddefault; an instance installed earlier keeps the mode it was installed with (freeze-after-loadsince 5.0.2,freezein 5.0.0 and 5.0.1) until you change it. A configuration carried over from v4 without alockdownvalue still has no lockdown. To switch, setapplications.lockdown: seal-after-load(in a cluster, only after every node runs v5.4.0 or later). See Intrinsics Are Sealed Instead of Frozen by Default. - Before downgrading an instance that uses a seal mode to 5.0.1 through 5.3, set
applications.lockdown: freeze-after-load. Those releases treat the seal modes asfreezeand lock the intrinsics before any application code loads, so a dependency that extends one while it loads, such asreflect-metadata, fails. Code that assigns to an existing intrinsic property after load, which sealing allows, also throws on the older release. 5.0.0 has no after-load mode.
MCP clients using expiring credentials must renew their subscriptions before expiry. Harper preserves the original subscribing POST expiry across workers and reconnects; an open GET does not guarantee that updates are still being delivered. Open a replacement GET and re-subscribe each URI with the new credential. See Subscription lifetime and credential rotation.
System-table permission declarations
Role declarations cannot grant direct system-table access. Superusers retain Harper-managed read permissions.
- The
add_roleandalter_roleOperations API requests, andcreate_authentication_tokensrequests minting a scoped token, rejectpermission.systemwith a 400 response. - Legacy
systemdeclarations in roles files are ignored with a warning naming the role. Other permissions and roles still apply. Remove these declarations from configuration. - Legacy scoped tokens lose declared system grants when validated, while their other permissions remain effective. This does not invalidate the token itself.
See roles-file configuration and fix: resolve system table permissions against the live registry #2122 and fix: discard legacy system grants without blocking role-file upgrades #4038.
Nested Relationship Constraints
Honor nested relationship filters and pagination and Fix wrong rows, unfiltered selects and ignored sorts in relationship queries correct REST and programmatic relationship queries that accepted nested constraints, non-leading relationship conditions, or top-level relationship sorts without applying them completely. Filters, ordering, offset, and limit inside select apply per parent in collection and single-record reads, and parents remain present when the nested selection has no matches. Outer relationship filters joined with AND — whether they lead the query or follow another condition — constrain the selected children as well as the parents, so a single filter does not need to be repeated inside the nested selection; see Conditions on To-Many Relationships for several filters on one relationship and for filters inside an OR. When a top-level sort across a to-many relationship (such as sort(products.name)) runs in memory — when the target attribute is not @indexed, a more selective condition on another attribute leads the query, or enforceExecutionOrder: true is set — Harper now orders matched parent records by the list of each parent's related values in relationship order, compared element by element, instead of ignoring the sort; when an @indexed target attribute leads the query, Harper continues to order via the related table's index join.
Upgrade impact: requests using nested constraints can return fewer children, and requests with top-level sorts across to-many relationships now return parents ordered by their related values. Unsupported filters on plain JSON projections or custom non-table resources, and nested full-text predicates, now return HTTP 400 instead of being ignored. Sorting and pagination of plain JSON arrays remain supported. Nested selections on source-backed caches use the same local-entry eligibility and ordering rules as a top-level query; the correction now applies those rules to constrained nested results. Refreshed nonmatches are excluded from the returned child page, so cache-backed child arrays can also shrink. See Filtering and Paging Each Relationship and sort.
These corrections are not released yet.
Null and Omitted Values in Query Conditions
Match null and omitted values the same way whether a condition leads the scan or filters makes REST, Resource API, and search_by_conditions conditions return the same records for null and omitted values whichever condition leads the query, and whether or not the attribute is indexed:
newith a non-null value (?discount=ne=5) matches records whose value is null or omitted. Before, an indexednethat led the query dropped those records.- No range comparator matches a null or omitted value, and a
nulllower bound means no lower bound:?discount=between=(null,10)returns the same records as?discount=le=10, and?discount=ge=nullreturns every record with a non-nulldiscount. Before,ge=nullandbetween=(null,x)also returned storednullvalues. - On an array attribute,
ne=vmatches a record when at least one element is notv, and a null or omitted array matchesne=vfor a non-nullv. Before, an unindexednecompared the whole array, so["x", "x"]and[]matchedne=x, and[]matchedne=null; now none of them do. - A sort on the same attribute as a condition that cannot read that attribute's index, such as a condition on an unindexed attribute or a non-null
ne, now returns records in sort order instead of primary-key order. - In SQL, a range compared with a
NULLliteral (discount >= NULL,discount BETWEEN NULL AND 10) now matches no rows. Before, it could fail with anInvalid value for attributeerror, orBETWEEN NULL AND xcould return rows whose value isNULL.<>,NOT IN, andIS NULLare unchanged.
Upgrade impact: ne queries can return more records, and ge=null or between=(null,x) queries fewer. A sorted ne query such as ?discount=ne=5&sort(discount)&limit(10) now reads every record and sorts in memory, where it used to read its page from the index and skip null and omitted values, so it takes longer on a large table. See Null and Omitted Values, NULL and omitted values in SQL, and Sorting.
This change is not released yet.
Conditions on To-Many Relationships
Make each relationship condition existential, and add relationship[...] for conditions one related record must meet makes each AND condition on a to-many relationship be met by some related record, possibly a different one for each condition, as on an array attribute, regardless of which condition Harper evaluates first: GET /Brand/?products.name=Keyboard&products.name=Cable returns a brand that has both. To require one related record to meet several conditions, put them in brackets after the relationship name: GET /Brand/?products[name=Keyboard&price=ge=30] returns a brand with a Keyboard that costs at least 30. Programmatically, the group is { attribute: 'products', conditions: [...] }. Groups can use |, nest another relationship (departments[employees[tag=oncall&level=ge=3]]), and follow a path (departments.employees[tag=oncall&level=ge=3]). select() of the relationship returns the related records that meet any one of the AND conditions, a group counting as one condition, and a relationship condition inside an OR decides which parents match without narrowing the selection.
Upgrade impact: before 5.4, whether separate conditions on one to-many relationship had to be met by one related record depended on which condition Harper evaluated first. A query or collection DELETE with such conditions can now match more parents: use the bracket form where one related record must meet every condition, and check a destructive filter with a GET first. select() of the relationship can return more related records. A request with a relationship condition inside an OR can return more parents and more related records. Brackets after a relationship name among the conditions, such as products[name=Cable], used to apply their conditions to the queried table; they now apply to the related records, and brackets after a name that is not a relationship return HTTP 400. See Conditions on a To-Many Relationship and relationship groups.
This change is not released yet.
SQL
LIKE Honors ESCAPE and No Longer Backtracks
Match SQL LIKE in linear time and honor LIKE … ESCAPE, in both SQL engines fixes two LIKE problems on both SQL engines. LIKE pattern ESCAPE 'c' now makes the character after c literal in SELECT, UPDATE and DELETE. Before, ESCAPE was ignored, so ip LIKE 't0!%' ESCAPE '!' selected, updated or deleted every value starting with t0! instead of only t0%. Matching also no longer backtracks: testing a value takes time at most proportional to its length times the pattern's length, where a pattern such as '%_%_%_%_%_%_%_%_%_%_%_%_%_%_Z' against a long value could block a thread before.
Upgrade impact:
- On both engines,
_now matches line breaks (\n,\r, U+2028, U+2029); previously it matched none on either engine, so a pattern such as'a_b'inSELECT,UPDATEorDELETEnow also matchesa\nb. - On the new engine,
*is now a literal (it acted as a regular-expression quantifier), and%now matches line breaks, as it already did on the legacy engine. - On the legacy engine,
\,^and$are now literals. They acted as regular-expression syntax, so'a\d'matcheda5. - On both engines, these are rejected with a 400 error: a pattern segment between
%wildcards longer than 10,000 characters, a pattern ending in the escape character, and anESCAPEvalue that is not a single-character string. On the legacy engine, so is a character-class range written backwards (such as[z-a]), or a pattern with more than 64_wildcards across segments that also contain literal characters or[...]classes when it also contains a character from every consecutive pair of reserved Unicode noncharacters (U+FDD0toU+FDEF,U+FFFEtoU+FFFF) and Private Use Area characters (U+E000toU+F8FF). The legacy engine checks a pattern when it tests a row against it.
LIKE is still case-sensitive on the new engine and case-insensitive on the legacy engine. Under the default sql.engine: auto, the case behavior depends on which engine runs the statement, so adding or removing an index can change the rows a LIKE in an UPDATE or DELETE matches. See LIKE Pattern Matching.
LIKE Keeps Numeric Values and Patterns Distinct
SQL LIKE on a non-String column now matches the stringified value in SELECT, UPDATE, and DELETE. Numeric 105 matches %05% and 1%, but not 0105 or 05%; an empty pattern matches no numeric, null, or omitted value. Previously, converting a pattern to the column type could select or modify the wrong rows.
Only declared scalar String columns use LIKE index rewrites. ID columns (which also permit tuple values), untyped columns, Any, and lists now use row filtering too. Lists match their comma-joined text rather than individual elements: ['news', 'sports'] matches news,sports and %s,sp%, but not sports%. Without another index driver, standalone LIKE queries fall back to legacy under auto with the default sql.allowFullScan: false; sql.allowFullScan: true keeps them on the new engine unless they ORDER BY an indexed column other than the primary key. Under new, they need another index driver or sql.allowFullScan: true. Upgrade impact: a standalone prefix or exact LIKE on an ID, Any, list, or untyped column can now run case-insensitively under auto. For example, mixed LIKE 'Pen%' can match both Penny and penelope, including in UPDATE and DELETE. Add an indexed condition to keep residual matching on the new engine, or declare a scalar String column to retain prefix index service. See LIKE Pattern Matching.
REST and Resource conditions also preserve string operands for contains, ends_with, and starts_with, including aliases and negation. Leading zeros stay significant: contains '05' no longer matches numeric 5. On numeric fields, empty contains and ends_with operands now match every non-null value. Text operands are no longer parsed as numeric, Boolean, or Date values; text that previously caused a conversion error now runs the text comparison. On Date fields the text is compared with the stored epoch milliseconds, so contains and ends_with with a date string no longer match that instant; use equality or a range for dates. Resource starts_with retains its string-only matching behavior; SQL uses its own matcher for other types.
Literal-on-the-Left Range Comparisons
Under sql.engine: auto and new, a range comparison with the literal on the left of an indexed attribute (WHERE 5 < id, WHERE 7 >= id, WHERE NOT (5 >= id), WHERE 5 <= id AND id <= 7) now matches the same rows as the column-on-the-left form in SELECT, UPDATE and DELETE. Before, it matched the mirror-image range: DELETE FROM t WHERE 5 < id deleted the records with id below 5 instead of above it. Comparisons on unindexed attributes, and statements that ran on the legacy engine, were not affected.
Upgrade impact: if you ran UPDATE or DELETE statements with a literal before <, <=, > or >= on an indexed attribute, check the affected records.
Falsy, Blank and NaN Values in SQL Conditions
On the new SQL engine, a bare value used as a condition (WHERE active, a join ON, HAVING) no longer selects rows whose value is 0, '' or NaN, in SELECT, UPDATE or DELETE. It now selects exactly the truthy values, as the legacy engine does. Before, DELETE FROM t WHERE region = 'west' AND active also deleted the rows where active was 0 or ''.
Everything below applies to a statement the new engine runs. Under the default sql.engine: auto, a statement with no index-driving condition (for example, its only condition is on an unindexed column) and no ORDER BY on the primary key falls back to the legacy engine, UPDATE and DELETE included, unless sql.allowFullScan is true, and keeps the legacy results, which this change does not touch. An indexed condition joined with AND (WHERE grp = 'a' AND qty <> 0, with grp indexed) keeps a statement on the new engine. sql.engine: new never falls back: it runs such a statement when sql.allowFullScan is true and otherwise rejects it with a 400 error. See The fallback contract.
An empty or whitespace-only string is no longer read as 0: qty = 0 and qty IN (0) no longer match it. A join no longer pairs two NaN keys.
Upgrade impact:
qty <> 0andqty NOT IN (0)now match blank strings on the new engine, as they already did on an indexed column, so anUPDATEorDELETEthat it runs with them on an unindexed column now also changes those rows. The legacy engine already matched blank strings withqty NOT IN (0)and still excludes them fromqty <> 0, so a statement that falls back keeps both.- Against a number, a string that is not a number (blank or not) sorts after every number on an unindexed column too, as indexes and REST conditions already order it. On the new engine,
qty > 0andqty >= 5now match blank strings, andqty <= 0andqty BETWEEN 0 AND 1no longer do. - Arithmetic on a blank string is
NaN, soUPDATE t SET total = qty * 2storesNaNwhereqtyis'', as doesSET qty = qty + 1. Before, they stored0and1. - A comparison with
NaNon a column without an index is nowNULL. Before, it was false against a number and compared as text against a quoted number, sov > '0'matched aNaNvalue. Neither that norv NOT BETWEEN 0 AND 1matches it now, andNOT (v < 1)in a select list returnsnullfor one. - A bare
WHERE vno longer selects a0nBigInt. - Equal infinities are now equal in ordering comparisons: on a column without an index,
Infinity <= Infinityandv BETWEEN 0 AND vare true for anInfinityvalue, where they were false. v NOT BETWEEN NULL AND 1now matches values above1, asNOT (v >= NULL AND v <= 1)does.BETWEENwith aNULLbound still matches no rows.
See Falsy, Blank and NaN Values.
Relationship Columns Follow the Related Table's Permissions
SQL now applies a role's permissions on the related table when a statement reads a @relationship column, on both SQL engines, as REST, GraphQL and the NoSQL operations do (harper#3509). Before, SELECT dept FROM data.Emp returned the whole related record to any role that could read Emp. A statement that reads a relationship whose related table the role cannot read now returns HTTP 403, wherever the column appears, including WHERE, aggregates, legacy SQL's -> and the rows an UPDATE or DELETE reads, and SELECT * leaves that relationship out. A role with attribute_permissions on the related table gets only the related attributes it may read. Super users see no change; roles that can read the related table see none apart from the member-access rule below. See Relationship Columns.
Upgrade impact:
- A statement from a role that cannot read a relationship's related table now fails with HTTP 403 instead of returning related data. Grant
readon the related table, or leave the relationship column out of the statement. - For roles other than super user, legacy SQL member access through a relationship no longer reaches the related record's own relationships:
team->vaultis left out of the row even when the role can read the related table (an@enumerablerelationship the role may read is still part of the record), and a related attribute the role cannot read is absent (dept->budget). In anUPDATEorDELETE, member access through a relationship returns HTTP 403 when its path names a table or attribute the role cannot read, or when it is applied to another expression, such asCOALESCE(dept, x)->name.
SQL UPDATE Takes Every SET Assignment
On the new SQL engine, an UPDATE that assigns a column more than once now applies every assignment in order: SET qty = qty + 2, qty = qty + 3 adds 5 and SET qty = 20, qty = qty + 5 sets 25. Before, only the last assignment counted (qty + 3, and 15). Every other SET expression still reads the row as it was before the UPDATE.
SET qty = T.qty + 1, naming the target table, is now an atomic increment like SET qty = qty + 1; before, concurrent statements lost increments.
Upgrade impact:
SET col = col + Non a value that is not a number writes the expression's result instead ofN:NULL + 5isNULL(it was5), and'abc' + 5isNaN(it was5). A missing value also counts asNULL.SET name = name + 5on aStringattribute no longer fails withOperation add ... requires a numeric property. ANULLvalue staysNULL; any other value is rejected as a number written to aStringattribute, asSET name = 5is.SET at = at + 5000on aDateattribute, which failed the same way, now moves the date 5 seconds later.- The legacy engine still keeps only the last assignment, so under the default
sql.engine: auto, anUPDATEthat falls back to it gives the old result. See Update.
Replication
Sharding
@sharded hashes primary keys to fixed slots on every thread, including dedicated replication workers. Per-database slot maps remain authoritative after initialization; replication.shards supplies the deployment shard set for a first default map. Conversion checks support equivalent legacy by-id policies without rewriting records. Attribute-derived policies and custom expressions are refused pending concurrent-move design (#1178). See Schema-defined sharding.
Sharding configuration failures are isolated to the affected database. A live disagreement holds that database until the configuration is repaired on all peers; healthy databases in the same application keep serving, including during worker replacement. Invalid configuration at process startup still requires a repaired full restart because its audit replay was deferred. First-time sharding initialization waits for the configured members; owner recovery requires a certified same-shard source or a complete backup. See Configuration failures and recovery sources.
Co-owner snapshot recovery does not restore the replacement's former local audit lineage. Cluster record-lock freshness treats the recovered database as recloned, including its ordinary tables; use a complete backup to preserve that history. See Recovery sources.
Snapshot-only recovery of a sharded node requires a fresh root and a new node name. Reusing the original name requires a complete stopped-node backup preserving audit through the last accepted write. A reused name missing its origin history refuses sharded writes and recovery-source eligibility, with both remedies in cluster status. See sharding recovery.
Replica Nodes in a Spare Shard
replication.spareShards lists shard IDs that a default map gives no slots, so a cluster can keep every slot on one shard and hold the other IDs for later, and the first database deployed after setup no longer spreads slots across them. Every node must share the list. A clone can now join a sharded cluster as a replica node with --shard replica (or CLONE_SHARD=replica). It takes a shard from replication.spareShards that owns no slots, preferring the one other replica nodes already hold, copies no full records, and reads them from the owners. --shard auto now skips such spare shards, so it no longer places a new data-holding node in a shard that owns nothing. See Spare shard IDs, Replica nodes in a spare shard and harper-pro#1344.
Base Copies That Could Not Finish
Stop forcing base copies that never finish fixes three ways a base copy could start without need or restart forever. The same fixes are also in 5.3.6 (harper-pro#1262), except refusal as a sharded owner: stuckCopy there has only retrying and held.
- Idle databases no longer force copies. A peer used to force a base copy whenever a returning node's position was older than
logging.auditRetentionor its own oldest retained entry, even for a database it had not written to since. It now forces one only when a prune may have removed history the node would replay. See When a Peer Forces a Base Copy. - Rows with no origin name now copy. Rows whose origin has no node name on the sending node, such as data migrated from v4, made the receiver fail and restart the copy. The sender now names that origin
unnamed-origin-<id>@<node name>, and those names can appear in logs. See Rows From Unnamed Origins. - Sharded owners resume without a copy (harper-pro#1278). A peer no longer forces a base copy on a node that owns sharding slots it cannot supply, which that node had to refuse, so replication between owners idle past
logging.auditRetentionnever resumed. It replays what it still holds and logs what may be missing. A write through a non-owner for a key whose owners all refuse that node returns503instead of success. See When a Peer Forces a Base Copy. - Stuck copies are held and reported. When a copy starts five times from the same position, or the receiving node refuses it as a sharded owner, that node now holds its reconnects to the peer for that database: 60 seconds at first, then longer each time, up to 15 minutes. Before, it retried every few seconds.
cluster_statusreports the state asstuckCopyon the database socket, and each change of state logs one line. See Stuck Copies.
Upgrade impact: after upgrading from a version before 5.3.6, a transaction log that was already purged empty can force one more base copy of its database for each node returning with a position older than that purge.
LMDB Replication Confirmation
LMDB writes that request replicatedConfirmation now wait for peer acknowledgments. Applications that previously received immediate success can now observe replication latency or an uncertain committed outcome if confirmation fails. See Confirmation Failure After a Local Commit before retrying a failed write.
Dedicated Replication Threads
replication.threads (default 0) starts worker threads that run replication only. They own the replication port and every subscription, so a peer catching up no longer competes with application requests. These threads load no application code, which limits applications that use setResidencyById or setComputedAttribute on indexed attributes. See Dedicated Replication Threads.
Local Replicas of Records Fetched From Another Node
On RocksDB, a table sharded with setResidency lets a node that reads a record it does not hold keep the fetched record as a local replica instead of moving it. The replica is served locally until an owner writes the record or its lifetime passes, set by the new @table arguments replicaExpiration and replicaRefreshFraction (default one hour, renewed by reads after half of it). Local replicas are RocksDB-only; LMDB continues to proxy remote reads without storing replicas or running a replica sweep.
A Hot Record Lock Hands Over at Release
Keep a hot cluster record lock from stalling waiters to their deadline fixes a stall on a record many nodes lock at once. A node that had been granted the lock gave it up if another node asked for it before the node finished checking it had the latest data. With enough network delay every grant went that way, so waiters on every node failed with 423 or 503 at the end of their lock wait, though the holder had released. No lock was ever held by two nodes.
A node now finishes that check and serves the waiter it asked for before handing the lock on. It gives up a check that has not finished 6 seconds after being asked, so a node that cannot complete one does not hold up the others. The fix is complete once every node is upgraded.
Cloning From a Sharded Leader
A node cloned from a leader with static sharding now joins. It used to copy the leader's replication.shard without saying so, and every copy of an @sharded database then failed, so it never finished and could hold the leader's database as well.
A clone now chooses its own shard with --shard or CLONE_SHARD: a shard number, or auto for the listed shard with the fewest live members. Cloning from a sharded leader with neither, and no shard already configured on the node, stops with an error naming the option. See Cloning a node into a sharded cluster.
A clone reports Available only when its sharded databases are ready, including the copy of its owned slots from a same-shard peer. A database whose copy keeps failing is named in the log with its last error, and fails the clone at its sync timeout. A leader that accepts a clone's request and never answers now counts as a failed attempt instead of holding setup forever.
Shard Owners in cluster_status
cluster_status now shows who owns what in a sharded database. Each entry of sharding that has an installed map lists shards, one item per shard with its slot count and the names of its owners, and nodesOwningNoSlots, the cluster members that own no slot. One response from any node shows the layout that node sees. Nothing is reported without an installed map: the responding node does not fall back to its own replication.shard setting. See Sharding Status.
The top-level shard field also reports shard 0. It used to be left out for a node in that shard.
Increment Re-delivery on audit:false Tables
On tables with audit: false, re-delivering a transaction no longer applies an increment twice after the record was invalidated, or after the same transaction replaced it with a put. A re-delivered increment can still apply again if a later transaction has since replaced the record with a put.
Restarts Behind a Backward Clock Step Stay Above the Origin Floor
On RocksDB, opening a database now raises the transaction clock above the origin-closed floor and issuance ceiling the database last persisted. After a restart behind a backward wall-clock step, every transaction therefore starts above the floor this node advertised to peers and above every key it issued ahead of the clock. Before, only the transaction-log keys Harper assigned to its own writes were moved up, and once a record lock was admitted ahead of the clock, every write paid an extra atomic step for the rest of the process (harper#2629).
- If a database's persisted floor or ceiling is more than ten years ahead of the host clock, Harper now refuses to start, and the error names the database and both values. This usually means the host clock is wrong (for example, a reset hardware clock): correct it and restart. If the clock is already correct, the bound was persisted while the clock ran more than ten years ahead: Harper starts only once wall time is within ten years of it, and the node's transaction clock then runs that far ahead.
storage.maxLockClockSkewis now measured from wall time only, so a lock admitted ahead of the clock can no longer stretch the limit for the next one.
update_node Changes and a Node's Own Shard
update_node that changes nothing but revoked_certificates and/or shard now writes the change to this node's record of the target without contacting it, as documented. It used to run the full add_node exchange, which reopened replication connections and replaced a requested shard with the target's own setting. On a node's own hostname, a shard other than its replication.shard is refused with 400.
cluster_status now reports a sharded database as held, with the error local shard membership disagrees with replication.shard, when the responding node's own record and its setting disagree about a shard in the installed map. Write admission already refused those slots; the status said available. See Update Node and Sharding Status.
Subscriptions
Resuming a Subscription From a Checked Position
A subscribe request can now carry the databaseGeneration its position came from. Harper then checks the position before and during the replay, and refuses one whose history was replaced (DatabaseGenerationChangedError, status code 409) or pruned (ResumeHistoryUnavailableError, status code 410) instead of replaying what is left. The subscription's resumeVerified promise resolves true once the replay is complete and checked. Requests without the field behave as before, and the check applies to RocksDB databases only. See Resuming from a position.
Durable MQTT Sessions No Longer Replay Short
A durable MQTT session now resumes through the same checked replay. When a reconnect's saved position names history that audit retention has removed, Harper deletes the session and reports sessionPresent: false (or, if the problem appears after CONNACK, sends an MQTT v5 DISCONNECT with reason code 0x83 and closes the connection) instead of delivering a partial catch-up. A position saved before a restore or copy, or on another cluster node, resumes best-effort, as before. A quiet topic's position is kept current, so retention on other tables does not reset it, and positions no longer skip unacknowledged messages that share a transaction. Resubscribing to a topic the session already holds continues from its saved position, and QoS 0 subscriptions are kept with the session and resume live. Each connection's Last Will is kept separately, so a connection that is taken over publishes its own will, never the newer connection's. See Durable Sessions.
MCP Clients Can Subscribe to Custom Resources
A static mcpResources entry can now name a subscribe method, so MCP clients can watch a custom URI such as memory:///{id} with resources/subscribe. The method is called like the entry's read method, as the session's user, and returns an async iterable. Each value it yields sends notifications/resources/updated with the URI. Harper releases the stream on unsubscribe, when the GET stream closes, and on DELETE /mcp. For subscriptions tied to a stored user, delivery also stops if that user is dropped or no longer has a role. Updates also stop at the earlier POST/GET bearer expiry. A subscribe that fails now answers a JSON-RPC error instead of an HTTP 500, for row-backed resources too. See Subscribing to custom resources.
A startTime Subscription No Longer Drops a Late Commit
On RocksDB, a table subscription with startTime used to switch from catch-up to live delivery at the newest localTime it had replayed. A transaction that committed during catch-up with a lower localTime was then delivered by neither, and some replayed events arrived twice. The switch now happens at live delivery's position in each transaction log, so each event arrives once. A write merged into a newer version, which live delivery already reports as the merged record, is now reported the same way during catch-up. If a transaction log cannot be read up to that position for about five seconds, the subscription ends with a 503 error; subscribe again from the same startTime to retry. See subscribe().
Subscriptions of Users a server.getUser Override Builds Stay Open
A component can override server.getUser to build users from a password or token, such as a JWT sent as the MQTT password, with no hdb_user record behind them. Their MQTT, WebSocket and SSE subscriptions stopped delivering at the first periodic re-authorization, about 30 seconds after subscribing, with the connection left open, and their MCP sessions lost their tools at the next user change. Harper now keeps such a user as it was admitted when it logs in through MQTT, HTTP Basic or a Bearer validator: re-authorization runs allowRead against it, and its subscriptions end when the connection closes, or at its authExpiresAt if the override sets one. Users that log in with an hdb_user record behind them are still revoked when they are dropped, deactivated or moved to another role, and so is a user whose name lost its hdb_user record within logging.auditRetention (3 days by default). An HTTP Basic or Bearer-validator login through server.getUser that returns a memoized user whose hdb_user record was dropped since its first lookup is also refused (401 on Harper's own routes, or deferred to the route owner on application routes); plain mTLS and cookie-session HTTP requests still keep the principal the override returns, while live subscriptions recheck by name. If the hdb_user table isn't audited (see logging.auditLog), a delete leaves no trace, so a user returned under a name other than the one it logged in with is checked by name at each re-authorization, and revoked unless an hdb_user record has its name. See Row-Level Access Control with Explicit Filters.
Resource API
Deferred Write Retries and Repeated Updates
A deferred save() that commits outside an open transaction() scope now stays pending if it is refused with 409 because an earlier write under the same record lock is still committing. Await the earlier write, then call save() again: the retry persists the update without closing its instance prematurely or moving it behind later staged writes.
Repeated update() calls on one instance now merge into one pending write at its original transaction position, like set(). Later writes from other instances retain their order and override overlapping fields. For example, A.update({ n: 1 }); B.update({ n: 5 }); A.update({ n: 2 }); in one transaction commits n: 5. A full replacement followed by a patch keeps the replacement's fields and semantics. Patches over a pending replacement still reject invalid operations with 400. Saving the merged update leaves no orphaned operation blocking a later lock().
Await an asynchronous update() before committing or checkpointing its transaction. If the transaction seals that update while its model hooks are still running, the update rejects with 409; call update() again to start a fresh generation.
If initial write preparation fails, save() repeats that error until an explicit update() succeeds. This also applies to a held lock after correcting its fields with set(): call update() to prepare the corrected draft before saving. Retrying a prepared write after the lock-overlap 409 above still requires only save().
After the first preparation failure, that update's changes remain in the draft and merge into the next update(), including the keyed form update(id, changes). A later correction that also fails restores the preceding draft; fields supplied only by that rejected correction are discarded. Use a full replacement or a fresh instance to discard the retained draft.
Array PUT elements remain eager writes: a refused element is discarded because no instance owns its retry. Queued conditional updates may be discarded safely when save() reports the precondition failure.
Repeated numeric additions now fold by operand type instead of the loaded record's type, preserving large BigInt deltas when an earlier write changes the base type. Mixed integer operands and overflowing integer sums fold as BigInt. Untyped fields with no numeric base can take the sum's type; rounding and extreme-magnitude behavior can differ from separate writes. See addTo().
In-Place Date and Bytes Changes No Longer Save
On a record returned by update(), a Date or Bytes property must now be reassigned to be saved. Changing it in place (record.when.setTime(...), record.bytes[0] = 9) used to save; it no longer does, and no error is raised. Assign a new value instead, such as record.when = new Date(...). Date and Bytes elements of an array property still save when changed in place. See What a tracked record writes.
Tracked Records Write Only What You Change
A record returned by update() no longer writes properties you only read: reading a sub-object or an array element saves nothing, sends nothing to a source's patch and adds nothing to the audit history. On a table that is not @sealed, assigning an undeclared property saves it even when nothing else changed.
addTo and subtractFrom Calls Combine
Repeated addTo / subtractFrom calls on one property in a transaction now add up instead of the last one winning, and a delta after an assignment adds to the assigned value (the write is still an assignment, so it does not merge with concurrent writers). subtractFrom accepts a BigInt, and on an Int, Long, Float or Number property a BigInt delta is converted to a number (one beyond 2^53 in magnitude is rejected). See addTo.
Built-in Agent
Turning Off or Host-Limiting http_fetch
agent.httpFetch sets whether the built-in agent has its http_fetch tool: false removes it, and { allow: [...] } refuses any request whose host is not on the list. It is read at startup only, and set_agent_config rejects it. http_fetch now also checks every redirect hop before following it, in every mode. See Restricting http_fetch.
Turning Off the Agent's Own Tool Groups
The built-in agent's filesystem tools, schedule_followup, the V8 inspector tools and the best-practices lookup can each be removed from its toolset at startup with agent.tools.filesystem, agent.tools.schedule, agent.tools.inspector and agent.tools.bestPractices (default true). A group that is off is not described to the model, and a call to one of its tools runs nothing; its tool names stay reserved, so no operations or injected tool can take them. With every group and httpFetch off, the agent has only the operations tools agent.user may call, so an application can send it a prepared brief and take a one-turn answer. The switches are read at startup only, and set_agent_config rejects them. The new get_agent_config operation returns the configuration in effect, the startup-only keys included. See Turning off tool groups and get_agent_config (harper#3993).
Cut-Off Agent Replies End the Run error
A model reply that did not finish now ends the agent session error, with the reason in lastError, instead of completed. That covers a reply cut off at the output-token cap or at the model's context window, one stopped by a content filter or a refusal, one that asks for tool calls of which none could be parsed, and an empty reply. Nothing from that reply reaches the transcript, and its tool calls are neither run nor queued for approval. Its usage is still kept on the session, and counts toward the agent's spend caps. The new agent.maxTokens (default 16384) is sent with every model request the agent makes; lower it for a model whose own output limit is smaller, or the provider rejects each request. See agent and Agent.
The config Scope Is the Config File
The built-in agent's read-only config filesystem scope used to be the directory holding harper-config.yaml. On a default install that is rootPath, so the agent could read keys/, ssh/, and the raw database files. It is now the config file alone. The new agent.configScope replaces it at startup with another file or directory, which keeps the config file readable only if it contains it. In every scope, the agent's filesystem tools now refuse to read Harper's key directories (<rootPath>/keys and <rootPath>/ssh), files named *.pem, *.key, or .jwtPass, and text holding a PEM private key, and they refuse to write into the key directories. set_agent_config now rejects configScope and componentsScope with a 400, as it already did httpFetch. An agent that read other files through config needs configScope set to a directory that holds them. See Filesystem scopes (harper#3041).
Credential Values Are Masked in the Agent's Config and Log Reads
The config file can hold plaintext credentials, such as a model apiKey passed through HARPER_SET_CONFIG, and the agent's config scope returned them, as did its logs scope and read_log for anything a component logged. The agent's reads of the logs and config scopes, and its read_log results, now show the value of every credential-named key (apiKey, password, token, secret, and similar) as *s, including a value that continues on more-indented YAML lines and a value a positional read starts in the middle of. grep_files searches the masked text in those scopes, and the agent's read_log refuses filter, which matches the unmasked log. The components scope, and read_log called through the operations API or an MCP client, are unchanged. Sessions recorded before this release keep whatever they already read, so rotate any credential an agent session may have read. See Masked credential values (harper#3097).
Capped and Paged Agent Tool Results
The built-in agent added every tool result to its conversation in full, so one large result, such as a log read whole, could fill the model's context window and leave the session unusable. A tool result is now cut to agent.maxToolResultBytes (default 65536) before it is stored, and read_file pages through a file by line, with no size limit, returning a cursor (nextLine, nextOffset) to continue from. When a provider still rejects a request as too long for the model, the agent cuts the most recent tool results to 2 KiB and retries once, and prompting a session left unusable by an earlier version usually recovers it the same way. The agent also stops sending its system prompt twice per request. See When a conversation outgrows the model's context window.
Spend Caps for the Built-in Agent
The agent records the usage of each model call on its session, and get_agent_session and list_agent_sessions report a session's totals as usage. Two caps bound a whole session, across every run in it: the new agent.maxSessionTokens, and agent.maxCostUsd, which is now enforced from model pricing and no longer defaults to 5.00. A reply the agent rejects, such as one cut off at agent.maxTokens, still counts toward a cap, though it stays out of the transcript. A session at a cap makes no further model call: it ends error, or completed with lastError naming the cap when the call that reached it was the final answer. A cap that cannot be measured stops the session rather than letting it run uncapped. Caps count calls that return: a call that fails, even one the provider billed, is not counted. If your configuration already sets agent.maxCostUsd, declare pricing on the generative entry the agent uses, and start new sessions: a session created before 5.4.0 recorded no usage, so it cannot be continued under a cap. See Agent spend caps.
Agent Cancellation and Limits
cancel_agent_run now answers every pending approval with a cancelled_by_operator result in the same write that marks the session aborted, so an approved action that has not started never runs, then or on the session's next prompt, and the session stays aborted even when a run was mid-step. The 409s agent_prompt returns for a busy session or a cancel now carry a code and retryable, so a client can tell a prompt it may resend from one already saved, and get_agent_session and list_agent_sessions report runInFlight. agent.maxTurns is now validated as a positive integer at startup and by set_agent_config, where a value such as null used to make every later run end without a model call; a bare maxTurns: in the configuration file now fails the agent's start. See cancel_agent_run.
Calling the Agent's Operations From Application Code
server.operation() called from application code now reaches operations that only the main thread registers, such as agent_prompt, get_agent_session and get_agent_config. Applications load on worker threads, where these calls used to fail with Operation '<name>' not found although the same operation answered over the Operations API. The call is sent to the main thread, which applies the operation's permission check and returns the result. Such a call does not join the caller's transaction, and a streamed result is refused. See server.operation.
Models
Model Pricing and Per-Call Cost
A models.generative entry can now declare pricing: { inputPerMtok, outputPerMtok }, what you pay in USD per million prompt and completion tokens. Harper ships no prices of its own. Each generate() call is priced at the rates of the entry that served it, a fallback included, and the result carries costUsd when that entry is priced and the provider reported both token counts. With toolMode: 'auto' it covers the final round, and streaming calls are not priced. This makes maxCostUsd work on toolMode: 'auto' calls; before, it had no prices behind it and never tripped. A toolMode: 'auto' round that leaves maxCostUsd or maxToolTokens unmeasurable, because the entry is unpriced or the provider reported no token counts, now stops the call with the budget error (429) instead of warning once and continuing, so set pricing on entries you cap by cost. See Pricing.
Truncating Long Embedding Inputs
models.embed(input, { truncate: true }) cuts each input to what the embedding model accepts, keeping the start of the text and any task prefix, instead of failing on an input longer than the model's context window. Only a backend that can truncate serves such a call: Ollama, Bedrock Cohere embed v3 and v4, and OpenAI-compatible embedding entries that set the new truncation: vllm for a vLLM 0.18 or later endpoint. If no candidate can, the call fails before any request is sent. Calls without the option send the same requests as before. Custom backends declare the capability with defineBackend({ truncate: true }). The @embed directive and the /v1/embeddings endpoint do not expose the option yet. See Truncating long inputs.
Configurable Embedding Task Prefixes
OpenAI embedding entries can opt into inputPrefixes.document and .query to prepend exact task instructions when a call supplies inputType. This supports Nomic v1/v1.5 retrieval prefixes through OpenAI-compatible endpoints. Existing entries keep their input unchanged until configured; changing document input text requires re-embedding stored documents. See Embedding task prefixes.
Memory
Smaller Per-Thread Baseline
An idle Harper thread now uses roughly 40% less JavaScript heap (about 58 MB down to 35 MB in a default install, before any application code loads). Harper's compiled code is smaller in memory, and several dependencies used only by specific features (LMDB storage, certificate revocation checks, certificate generation, system_information) now load when that feature is first used.
Source maps are no longer enabled by default, so stack traces from Harper's core point at its compiled JavaScript. If you control the process environment (not on Fabric clusters that Harper hosts), set NODE_OPTIONS=--enable-source-maps to map them back to the TypeScript sources. See Source Maps in Stack Traces.
Component Deploys
Certifying a Release in a Canary Worker
A deploy_component with restart: true or restart: "rolling" now checks its release before rolling it out. The first worker started on the release, the canary, is held out of traffic until it reports whether the deployed component loaded, and the rollout goes on only if it did. Each later replacement is checked the same way. When the node that received the deploy rejects the release, the deploy fails with 400. The release it replaced is made live again, or, with nothing to go back to, the component fails closed on that node. A new certification field in the response reports the outcome, and says when a restarting deploy went out unchecked (uncertified or unavailable, as for a package: file:<dir> deploy). With "rolling", each other node then certifies the release with a canary of its own, and a node that rejects it does not fail the deploy: the deploy's record ends partial, as Following a Rolling Deploy to Every Node describes. A node running a version before 5.4.0 activates without a canary. A deploy that restarts nothing is not test-loaded. If its release throws when it loads, get_status reports the failure. See Certifying a release in a canary worker.
Following a Rolling Deploy to Every Node
A certified "rolling" deploy now responds with status: "rolling_out", and its deployment record follows the job that takes the release to the other nodes. The record names the job in restart_job_id, adds each node's outcome to its peer_results entry as activation, and ends success once every node has taken the release, or partial with error.nodes naming each node that did not. A job that fails before it reaches every node ends the record partial; one cut short by its node restarting, or unable to write the record's end, is settled at that node's next start from the job's own result. The record replicates, so any node answers get_deployment for it, where before only the node that received the deploy knew the job's outcome. harper deploy restart=rolling now waits on the record, prints each node's outcome, and exits non-zero when a node did not take the release or the wait runs out; wait=false skips the wait, and wait_timeout=<ms> bounds it. That wait needs get_deployment, which the role harper deploy setup=true creates now grants. A CI role written by hand with an operations list needs it added with alter_role, as does a trust policy scoped with operations; until then pass wait=false, or the wait is refused after the node you called has taken the release. See Waiting for a rolling deploy.
Which Release Is Live on Each Node
The new list_releases operation lists, for each component, the release live on a node and the releases it keeps on disk to make live again, each with its state (staged or kept), at and size; replicated: true asks every node. get_components entries now carry the live release's deployment_id, deployed_at, when the deploy that made it live finished, and restarted_at, when that node last loaded it. See list_releases.
Where a Deploy Came From
Deployment records now carry source. A deploy made with a token from exchange_oidc_token records the claims the exchange verified, such as the repository, commit and workflow run, with via: "oidc". Otherwise via: "declared" records what the client said, such as the commit of the checkout the harper CLI deployed. list_deployments takes sha to find the deploys of one commit. See get_deployment.
What Activating a Release Puts Back
Activating a staged or earlier release puts back the host, urlPath, isolated and branchedDatabases it was deployed with, so it can move an application onto or off its private database copy. Deployment records now say which, as routing, so a client can show it before it activates a release. A routing of null means the release leaves them as they are, and a null key means activating it removes that key. A record without routing is older, or its deploy has not determined it yet. See get_deployment (#4077).
Failed Deploy Payloads Expire
A failed deploy keeps its payload for retrying or debugging, but no longer forever: deployment.payloadRetention.failedMaxAge (default 7d) and failedMaxCount (default 2) bound how long, and how many per project, are kept. Each project's newest failed payload is always kept, and the record stays. See deployment.payloadRetention.
Failed Component Loads Close Their Scope
When a component load fails on a worker thread (for example, when handleApplication() or an initial entry load rejects, or when the load or its lock wait times out), Harper now closes the component's Scope instead of leaving its file watchers, options watcher, deploy-lifecycle listeners, and secret-subscription holds open until the worker exits (when an initial entry load fails, one recovery watcher per failed entry handler stays open until the next file edit, reload, or shutdown so correcting the file still requests a restart). scope.close() also now removes remaining listeners and releases the component's secret subscriptions even when a 'close' listener throws or rejects. See scope.close().
Warning When a Declared Entry Point Matches Nothing
A deploy now warns when a jsResource.files or graphqlSchema.files pattern in the component's own config matches no file in the release, as a truncated payload would. The deploy still goes ahead. See Declared entry points that matched nothing.
Starting and Stopping One Isolated Application
The first restart: true deploy of a new isolated application now starts only its own worker, which is that release's canary, instead of also restarting the shared workers. Dropping an isolated application stops its worker with or without restart, and drop_component removes an application's forks by default; keepBranches: true keeps them. A removal the drop cannot finish, because a worker still has the fork loaded or the removal failed, is recorded and retried until it succeeds (harper#3600). Dropping an application a node does not have restarts nothing there. The response reports node, componentRemoved, workerStopped, and branchesRemoved for the node that answered, and the same for each peer in replicated, so a preview's cleanup can be one replicated drop. See Restarting and dropping and Removing a fork.
Isolated Applications Are Admitted in Startup Order
The threads.maxIsolated cap now counts only the isolated applications that get a dedicated worker: those already running, then the rest in root-config order, the order startup admits them. An application whose live release failed closed, such as a first release its canary rejected, no longer holds a place, so it does not block other isolated deploys. A deploy_component that would leave its application without a worker in that order is refused with 409, including a redeploy of an application that is on disk but has no running worker; before, such a deploy could succeed and never start a worker. The same check runs when a refused release is activated again by its deployment_id. The deploy also answers 503 when the node cannot read which releases are refused. See Requirements and refusals.
Setting Up GitHub Actions Deploys
harper deploy setup=true provider=github-actions sets a cluster up so a GitHub Actions workflow can deploy with its OIDC identity token instead of a stored credential. It creates a deploy-only role and user, a trust policy matching the workflow's repository, branch and environment, and the repository's HARPER_CLI_TARGET variable. It checks the policy against the workflow file in the repository's checkout before it writes, where it can, and reads it back after. It only creates: running it again leaves the cluster and the variable as they are, a record that differs, or that someone deactivated or disabled, stops it before it writes, and a HARPER_CLI_TARGET that names another cluster is left in place. A policy with the same id that another super user creates in the moment before setup writes it is still overwritten. See GitHub Actions deploys (OIDC).
A Rolling Restart Fails When a Node Did Not Restart
The job behind a replicated restart_service (and behind a restart: "rolling" deploy whose certification is unavailable) now ends ERROR when any other node did not restart, instead of COMPLETE. Its message names each node that failed or was skipped, and result.replicated holds one entry per other node in both outcomes, with an error key, not message, for a node that did not restart. A node whose restart failed or may still be running stops the roll, and the nodes after it are reported as skipped; a node whose write queue was full is retried after the others. See replicated (harper#3498).
Pull Request Previews
A Preview of Each Pull Request
An application can now have a preview of each pull request, deployed from GitHub Actions to the application's own cluster, with no secret stored anywhere (harper#3042). harper preview setup app=<app>, run once from a checkout of the repository, calls the new enable_previews operation, which creates a user <app>-ci-preview whose role can deploy and drop only that application's previews, and two OIDC trust policies for the repository's preview workflow. Setup checks that workflow, or writes a starter one, and sets the HARPER_PREVIEW_TARGET repository variable. For each PR, the workflow runs harper preview deploy, which deploys the PR's head commit as the isolated application <app>-pr-<n> at <branch slug>.app.<cluster host>, on its own branch of the databases. Fabric's Colocated and Dedicated clusters serve those hosts with no setup, though a cluster created before preview hosts gains them only as each instance's certificate renews, within about seven weeks. Closing the PR runs harper preview drop. The HarperFast/deploy-action GitHub Action runs the same lifecycle as a workflow step, and comments each preview's URL on its PR. harper preview teardown, or disable_previews, revokes the identity and drops every preview of the application on every node, and get_preview_policy reports how previews are set up. See Per-PR previews and harper preview (harper#3505, harper#3617, harper#3809, harper#3807).
disable_previews is a barrier on every node: a preview deploy already going live finishes and is dropped, and one that would go live later is refused with 403. A node that does not acknowledge it stops disable_previews with the identity revoked, and running it again resumes. See disable_previews (harper#3812).
Previews Expire, and Are Capped
A deploy_component can carry preview metadata naming its application, PR, branch, commit and expiresAt, deployed as <app>-pr-<n>. The cluster drops a preview once it expires, checking every deployment.previews.sweepInterval (10 minutes by default); deployment.previews.maxPerApplication caps how many previews of one application a node keeps live; and a preview's component keeps deployment.previews.stagingRetention.maxCount (default 1) replaced releases. get_components and the deployment record show the metadata. See Preview deployments.
Roles That Deploy Only Some Components
A role's new component_deploy grant names the components it may deploy, drop and read, such as my-app-pr-*, and can require each deploy to be isolated, under a host suffix, on branched databases, or a preview, with its own maxPreviews and maxTtlDays. It answers for deploy_component, drop_component, get_components, list_deployments and get_deployment, ahead of the role's operations list. component_deploy is now a reserved name, which no database can use. See Component Deploy Grants.
A secret can be granted to a component-name pattern, such as my-app-pr-*, so one private-source deploy credential serves every preview; a component's own code still reads only secrets granted to its exact name. harper deploy setup=true accepts such a project for the github and npm providers. See grant_secret.
Payload Deploys Can Set Routing, Isolation and Branching
A payload deploy, such as harper deploy from a directory, can now set host, urlPath, isolated and branchedDatabases, which it used to refuse or ignore. One that sets any of them sets all four, keeping the current value of each it leaves out, and going back to its release restores them. See How a deploy updates the root config.
Upgrade impact: a project with a character other than letters, numbers, - and _, after an npm scope and archive extension are removed, now fails with 400. my-app.pr-12 used to deploy over my-app.
Isolated Applications
- The shared workers answer a request for an isolated application's
hostwith503andRetry-After: 5, and close a WebSocket for it with1013, instead of serving another application's routes. See Reaching an isolated application. - In an isolated application, and under the
vmandcompartmentmodule loaders, the baredatabasesandtablesglobals now reach the application's branched databases, as importing them fromharperalready did. See Reaching the fork from code. - An isolated application with a
hostcan loadfastifyRoutes, which a host-mounted application in the shared workers still cannot. See What a mount does not do. http.corsAccessListandoperationsApi.network.corsAccessListaccept an origin such ashttps://*.app.example.com, where*is any one label. Seehttp.corsAccessList.
Conflicting Trust Policies Are Refused
When two enabled OIDC trust policies match one identity token and give it a different user or different operations, exchange_oidc_token now refuses the token, without spending it, where before the first policy by id answered. Operations compare as what they allow, so a permission group such as read_only equals the list it expands to. list_oidc_trust names each such pair in invalid_reason, and its entries carry a new exchangeable field. harper deploy setup=true provider=github-actions and enable_previews refuse to write a policy that would make such a pair. See the Upgrade Notes and exchange_oidc_token (harper#3811).
MCP
Session Creation Is Rate-Limited by Default
MCP initialize is now limited per client identity, on both profiles, by the new mcp.<profile>.rateLimit.initializePerSecond (default 10) and initializeBurst (default 50). Each session is a stored row kept for the idle timeout, and the application profile accepts anonymous initialize, so this limit is on by default; set initializePerSecond: 0 to turn it off. A denied initialize gets HTTP 429 with a Retry-After header and a JSON-RPC error, and no session is created. Clients that open sessions in bursts, such as a test suite or a pool of agents behind one IP address, may now see that 429 after upgrading. initialize also stores only the names of the client capabilities Harper checks (sampling, elicitation, roots), not the capabilities object the client sent. See initializePerSecond (harper#3537).
Operations-Profile MCP Clients Are Identified by Address on Bun
Under Bun, the operations-profile MCP endpoint identified every client as 127.0.0.1, so all clients shared one initialize limit and one perClientPerSecond bucket. Each client is now identified by its own address, as on Node.
Resource Subscriptions Preserve Credential Expiry Across Workers
MCP resource subscriptions now work when the subscribe POST and GET-SSE stream reach different HTTP workers. Reconnecting GETs continue to wait for restoration before HTTP 200; a transient restoration failure now returns HTTP 503 with Retry-After: 1. The original subscribing POST expiry is preserved across workers and reconnects. Delivery is also bounded by the current GET credential's expiry. Re-reading a user's permissions after a role change also retains the GET credential's expiry. A network reconnect with matching credentials restores unexpired subscriptions without new subscribe calls. Stored scoped subscriptions require a scoped GET with the same username and role permissions; an ordinary credential with that username cannot inherit them. To renew after rotating credentials, close the old GET, open a replacement GET with the new credential, and send each resources/subscribe POST with that credential too. Subscriptions can end at expiry without closing the GET stream. See Subscription lifetime and credential rotation (Make MCP resource subscriptions reliable across HTTP workers; Fix scoped MCP reconnects and concurrent subscribe failures).
Clients matching the formerly documented -32603 failed to subscribe: <uri> or -32602 resources/subscribe for <uri> was cancelled: <reason> responses should handle the current -32603 resource subscription failed; retry the request response and, when the GET closes during setup without a successful reconnect, -32603 resource subscription timed out; retry the request. Let overlapping requests settle before retrying a transient failure; do not retry solely because the message says to.
Data Loader
Records Matched by the Table's Primary Key
The Data Loader now matches and tracks records by the table's primary key. On a table with a custom @primaryKey, user edits to loaded records are kept and data file changes apply; versions before v5.4.0 looked up records by their id field, rewriting any record whose id field did not match its primary key on each load. A data file that names a database now loads only into that database, creating the table there if needed, instead of into a same-named table in the default data database. Reordering keys in a nested object no longer counts as a content change. After upgrading, some records loaded by an earlier version stop receiving data file updates; see Upgrading from versions before 5.4.0 (harper#3582).
REST
A JSON Response That Fails Mid-Stream Is Cut Off
Cut off a JSON response that fails mid-stream changes how a streamed JSON array reports a failure after its 200 status is sent. Harper still writes an {"error": ...} element, but now leaves out the closing ] and aborts the connection, instead of closing the array and ending the response normally. A partial result no longer parses as a complete one. Clients should treat a cut-off body, such as response.json() rejecting, as a failed read. NDJSON and Server-Sent Events responses are unchanged. A json export_local whose query fails now ends its job with ERROR and removes the partial file.
A common trigger is a client that stops reading a large collection query for longer than storage.maxTransactionOpenTime (default 30 seconds), which on RocksDB fails the read with ReadSnapshotExpiredError. See When a Streamed Response Fails and storage.maxTransactionOpenTime (harper#2392).
CSV select(*) Keeps Declared Columns
A text/csv response to select(*) from a custom get now starts its header with the table's declared attributes, then adds any other key on the first row. An optional attribute missing from the first row keeps its column and the values of later rows. A custom get that returns a response envelope ({ headers, data }) or throws a Response also gets its scalar and tuple select() shaped as a plain return is, instead of an empty CSV body. See CSV.
Conditional Writes Without Cluster Record Locks
A conditional PUT, PATCH or DELETE (If-Match, If-None-Match, If-Unmodified-Since) takes the record's lock by default in 5.4, so requests that saw the same ETag are checked one at a time. On a replicated table that lock is a cluster record lock. When a node runs without cluster record locks (Harper Pro's replication.recordLocks is not enabled, the self-hosted default), its conditional writes to replicated tables are checked without a lock, as in 5.3: they no longer fail with 503. Concurrent requests that saw the same ETag can then all succeed, and the last write wins.
With cluster record locks enabled, conditional writes keep the lock. Until a home map is in place they fail with 503, and the message says how to establish one. rest.preconditionLocks: false still checks every conditional write without a lock. See Conditional Writes.
Configuration
Graceful Shutdown on SIGTERM, SIGINT and SIGQUIT
On SIGTERM, SIGINT or SIGQUIT, Harper no longer exits at once. It stops accepting connections, gives requests in flight a few seconds to finish, closes WebSocket connections with code 1001 once no request is in flight (not under Bun or the uWS HTTP server), closes component scopes (their 'close' listeners run once requests in flight have had a few seconds to finish), and exits with code 0. The new threads.shutdownTimeout (default 8000 ms, 30000 under harper dev) bounds that shutdown; 0 restores the immediate exit, and a second signal exits at once. Set it a few seconds below your orchestrator's stop grace period (Docker and Compose default to 10 seconds, Kubernetes to 30, minus any preStop hook time), leaving time for the database close at exit, which the setting does not bound. harper stop now waits for Harper to exit, and exits 1 if it is still running threads.shutdownTimeout + 60 seconds after the signal. See Shutting down on a termination signal and harper stop (harper#3943).
Applying V8 Flags From the Configuration
The new threads.v8Flags setting takes a V8 flag, or a list of them, and applies them to the Harper process before its first worker thread starts, so every worker thread runs with them. It is for flags such as --optimize-for-size, which Node refuses in NODE_OPTIONS and which cannot be passed to a worker thread. It can also be set through HARPER_SET_CONFIG or HARPER_CONFIG. An entry must start with --; set_configuration refuses one that does not, and Harper stops at startup with an error naming the setting. Unlike the command line, this does not apply a flag's implied flags, so list them yourself (for example --max-semi-space-size=1 alongside --optimize-for-size). A changed value takes effect when the process restarts (the restart operation, on each node); restart_service restarts only worker threads and does not apply it. Not applied under Bun. See Applying V8 flags.
Transactions
Failed Transactions Are Marked Aborted in the Transaction Log
The new storage.rocks.transactionLogAbortMarkers setting is on by default (RocksDB only). When a transaction fails after its entry was written to the transaction log, the entry is now marked aborted, and replication, subscriptions, read_audit_log and replay after a crash all skip it. A write that keeps conflicting with other writes after its log entry was written no longer retries without limit: like other conflicting writes, it fails with HTTP 500 after 40 commit conflicts, or with HTTP 503 and code TRANSACTION_COMMIT_CONFLICT_TIMEOUT once it has been in conflict for longer than storage.maxTransactionQueueTime, whichever comes first. Writes applied from replication or a caching source, and a transaction part of which has already committed, still retry until they commit. Set the setting to false to keep the 5.3 behavior. Harper reads it once per process, so a change needs a restart of each node; restart_service does not apply it. Harper 5.3.4 and earlier cannot read the markers. See the Upgrade Notes and storage.rocks.transactionLogAbortMarkers (harper#3928).
The new storage.rocks.transactionLogWriteIntents setting, off by default, makes a commit that would conflict with an older, still-open transaction wait for that transaction to finish, usually before writing its transaction-log entry. This shortens the longest waits under heavy write contention, but adds retries and lowers throughput when transactions are large or contention is low. It needs abort markers on and a full restart to change.
Record Lock Fixes for Lapsed Leases, Deletes and Mid-Scope Commits
Six fixes to table.lock() and immediate writes outside transaction():
- A hold lock whose lease ran out keeps refusing writes to its record from every instance on the same context, until that lock's
unlock()or a newlock()of the record. Outsidetransaction(), the next write to any record in the database used to clear it, so a late write landed after another caller could have locked the record (harper#4053).unlock()of a lapsed lock now also lets later writes to the record through insidetransaction(); before, they kept failing with409. - A
delete()orinvalidate()made through a transaction-scoped lock now commits when the lock is released early withunlock()or upgraded to{ hold: true }. It was discarded (harper#3902). - On RocksDB, outside
transaction(),publish(),invalidate(), anddelete()now wait for their native commit to settle before resolving, and reject if that commit fails. An unlocked write a table source's hook makes during that commit, on the same context and database, resolves once it is part of the commit instead, since the commit is waiting for the hook. Before, they resolved while the commit was still in flight, so a followingsave()orput()under a{ hold: true }lock on anaudit: falsetable could be silently dropped. An instancedelete(query)outsidetransaction()resolves once every row's commit has settled. It commits up to 16 rows at a time; after a row fails it issues no more rows, but the rows already issued still commit before it rejects. - A record lock now has at most one commit in flight. Outside
transaction(), a write through a lock issued while an earlier write through the same lock is still committing is refused with409; await each write before the next. The lock and the record's unsaved changes stay usable, so the next awaitedsave()commits them. Before, overlapping writes could share a version or land out of order, so a later write could be silently dropped or an earlierpublish()could roll the record back (harper#4098). A write to the locked record from a table source's hook during that record's own commit is refused with409the same way; before, it was silently overwritten or refused in a way that blocked later writes through the lock (harper#4109). A locked write made while another commit on the same context is still in flight, such as one waiting for a streamed blob, now commits on its own instead of joining that commit, which could store it under a log key below its version, so replicas could resolve it differently (harper#3238). - A second
lock()of a record from the same context now joins the first one's wait when a commit happened while the first one waited. Before, it waited behind the first one's lock and failed with423(harper#3913). - A
lock()whose wait ends while its owntransaction()is running a mid-scopecommit()now waits for that commit and then takes the lock. It used to fail with500 Transaction was closed while waiting for a record lock, or return a lock that the commit then released (harper#4080). Such alock()gives the record back while it waits, so another caller can take it first. Alock()from a table source's method that the commit runs, or from the source method of a write that one makes, does not wait, since the commit is waiting for it.
Refusals for Source Hook Writes and Record Locks in a Closing Transaction
On RocksDB, writes a table source's write-through method makes to the record being written, and table.lock() calls in a transaction() that is closing for its own commit, now fail with 409 and a message saying what to do, instead of losing a write or failing with 500:
- A source's
put,patch,deleteorpublishthat writes the record being written, before its firstawait, is refused with409. Before, the write resolved but the cached write overwrote it (harper#4112). After anawaitthe write follows the cached write, as before. - When the cached write goes through a
lock()holder, that source method can no longer change the holder before its firstawait(409), whether the holder patched the record or replaced it withput(). Before, a change it then tried to save was refused, yet stored by the cached write; after aput(), the change was stored unvalidated, or dropped without an error when the record was passed with its id (put(id, record)). - Outside
transaction(),update()through that holder from the source method is refused with409, because no commit would save it. Before, it stayed staged, and a laterlock()of the record on the same context failed with409until the holder saved or unlocked. - Inside
transaction(), while acommit()of that transaction is still in flight, re-locking a transaction-scoped lock through the instance that holds it, or upgrading that lock to{ hold: true }through any instance (a source method of that commit included), is refused with409; await the commit, then callTable.lock()again. A{ hold: true }lock stays re-entrant, as does a transaction-scoped one for a source method of that commit that does not ask for a hold; that includes a source method of a write the first one makes, such as to another sourced table. Before, the call returned the lock that commit then released, so the next write through it failed and nothing excluded other callers (harper#4081). - A
lock()in atransaction()after a mid-scopecommit()made while a read iterator was open is refused with409, even once the iterator is returned, until a latercommit()reopens the transaction. Alock()from a source method of a write in another database, for a record in a database the commit has already committed, is refused with409. Both used to fail with500 Transaction was closed while waiting for a record lock. After its firstawaitduring the transaction's final commit, a source method'slock()of any record through the context it is given is refused with409too: that commit has released the context, so the lock used to be taken outside the transaction and held until its lease ran out (harper#4122). Alock()waiting when its transaction finishes or aborts still fails with500.
Source methods should await or catch every write they make; an ignored refusal is an unhandled rejection.
Record Lock Writes Refused When the Transaction Clock Is Fixed Below the Admitted Version
On RocksDB, saving a write to a record held under table.lock() inside an explicit transaction() now fails with 409 whenever the transaction's clock was already fixed (by an earlier unlocked write or a prior saved write) below the record version lock() reloaded at admission (harper#2712). Previously, only a { hold: true } lock whose own earlier write, or admitted version, was stamped ahead of the clock was checked. A write under a lock admitted on a version committed after the transaction's first write was stamped at the older transaction time and lost under last-write-wins (on audit: true tables, merged out of order for disjoint fields and commutative operations).
- Applies to scoped locks (
table.lock(id)), held locks (table.lock(id, { hold: true })), and static table writes (put,patch,update,delete) joined to a lock on the same key. - Read-only scoped locks that stage no record change still commit without error.
- To avoid the
409, acquire record locks before the transaction's first write, or save the locked record in its own transaction.
Storage
RocksDB Memtable Arena Sized From the Write Buffer Budget
RocksDB now sizes each column family's memtable arena block from the write buffer budget: writeBufferManagerSize / 2048, between 64 KiB and 1 MiB (previously a fixed 1 MiB). Every written table holds at least one block against the budget, even while idle, so with many tables on a small instance the fixed size used up most of the write budget and caused constant small flushes. Two new settings override the defaults: storage.rocks.arenaBlockSize (bytes) and storage.rocks.writeBufferManagerFlushPolicy (oldest, largest, or largestAcrossDBs). When writeBufferManagerAllowStall is true, the flush policy now defaults to largestAcrossDBs, so a stalled write no longer waits forever on memory held by idle databases. See RocksDB Memory (harper#3712).
Record history after eviction
Retained audit history stays readable after a record is evicted and across later reinsertion on tables eligible for eviction. Historical reads also reconstruct the complete record after multiple mutations in one transaction.
Plain non-expiring reinsertion on a non-evicting table can still omit older history.
History listings reuse complete audit entries within a bounded read context and build ordinary patch histories incrementally. Unsafe chains and values retain the existing reconstruction behavior. See #3504.
Changing a Plain Index to HNSW While Harper Runs
Changing an attribute from a plain @indexed index to an HNSW index on a running server now takes effect on every thread, including threads that do not declare the table and only reload its schema. Those threads used to keep writing plain index entries, which the HNSW index could not see. Until the Harper process restarts, the new HNSW index reads without its in-memory record cache, and Harper logs a warning naming the index; a restart restores the cache. See Vector Indexing.
Record Cache Size Setting
The new storage.recordCacheSize setting sets how many entries each thread's in-memory record cache holds (RocksDB only; default 32768, the size it already had). Harper now also sizes RocksDB's verification table, which lets a cache hit return without a RocksDB read, from the record cache size and the number of threads it starts. The table gets two slots per record those caches can hold, at least RocksDB's previous 131,072 (1 MB), and grown no further than 1/64 of memory. With 4 or more worker threads the table therefore grows even at the default cache size, to 4 MB on 4 workers; before, cache hits there could fall back to RocksDB reads. On an installation that still has LMDB databases, LMDB's record caches now keep their own default-sized budget instead of sharing one with the RocksDB record cache. The undocumented HARPER_RECORD_CACHE_SIZE environment variable is no longer read. See storage.recordCacheSize.
Removing a Table's Expiration Clears It
@table (and defineTable) now clears an expiration or eviction it no longer names, or sets to 0, after a restart and on threads that load the table without declaring it. Before, the old value stayed in effect, so a table kept expiring records after its schema stopped declaring a TTL. If you relied on that, declare the value again. Records already written keep the expiry they were given. A negative or non-numeric expiration, eviction or scanInterval now fails the declaration with an error naming the table and argument, instead of failing the table later; create_table accepts eviction and rejects a negative or non-numeric expiration or eviction with a 400. Expired records are now removed on schedule after a restart with threads.count: 0, and on a table that has only eviction, without waiting for a write. See @table.
Local Replicas Give Way Under Storage Pressure
A node that keeps local replicas of records owned by other nodes now demotes them back to stubs when free space falls below storage.reclamation.threshold, oldest deadline first, shortening their lifetime as pressure rises. A record written in the last 30 seconds stays at any pressure, because it may be the node's own write that its owner has not received yet. Records the node owns are not touched, and the next read fetches a demoted record again. Space returns when RocksDB compaction rewrites the old rows, not at the moment of demotion; see Local replicas under storage pressure. describe_table and describe_all now report replica_count and replica_size_estimate per table.
Table.deleteHistory on RocksDB Reports What It Removes
On a RocksDB database, Table.deleteHistory(endTime) now returns 0 without reading the audit log. It removes no audit entry: a RocksDB transaction log is shared by every table in its database and loses only whole files, through the retention settings or delete_transaction_logs_before for the whole database. Before, it read the whole log, reported every entry of the table as deleted, and removed the table's tombstones (records left by deletes) as a side effect, which made those deletes invisible to a subscription replaying from before them. To remove tombstones, pass cleanupDeletedRecords: true as the second argument; that still sweeps tombstones older than endTime, which also hides those deletes from a subscription replaying from before them. On LMDB the method is unchanged.
At startup, RocksDB now makes the audit floor durable before it removes aged transaction-log files, as it already did during normal operation. On a full disk, where that step cannot run, startup still removes the files so the node can free space, and logs a warning; until the next flush, a power loss in that window can lose the floor record.
Backups
Restore Checks a Backup Before Deleting the Database
restore_backup no longer deletes a database before checking that the backup can be read. It first verifies the checksum of every database file in the backup, checks its transaction logs, opens each table file to confirm this version can read its format and compression, and, for a backup that captured blobs, checks that the blob snapshot is present. A backup that fails is refused and the database is left as it was (one an earlier restore left incomplete stays blocked); previously the restore deleted the database first and then failed, so a backup taken by a newer version, or one with a corrupt file, destroyed the database it was meant to restore. The check reads the whole backup once, so a restore takes longer, and an online restore keeps the database unavailable while it runs; it needs no extra disk space. A restore interrupted while it is still checking the backup leaves the destination's files and any preexisting incomplete-restore marker unchanged; a previously healthy database loads normally on the next start. Blob file contents are not checksummed; the restore later checks that every blob a record references is present (see below). See restore_backup.
Restores Survive a Crash, and Prove the Database Before Publishing It
Every restore_backup now records its decision on disk before it changes anything, applies and checks the backup in a separate process, and publishes the restored database only after that process has proven it: the transaction log replays completely, every record decodes, and every file-backed blob a record references is present. A crash at any point leaves the database as it was, restored, or blocked until a rerun finishes it; previously a crash mid-copy could leave a mix of old and new files. Only one restore or verification runs at a time on an instance. See if a restore is interrupted (harper#3920).
Restores Refuse to Run on a Node With Replication Peers
restore_backup now fails on a node that lists replication peers in system.hdb_nodes or replication.routes, since replication would undo the restore or spread it to the cluster. Remove the node's peers before restoring and add fresh ones afterwards. While a restore runs, operations that add a peer fail with status code 409. See restoring in a replicated cluster.
New restore_backup Options, and target_database Online
accept_incompatible_backup restores a backup taken by a newer Harper whose archive manifest major is higher than this version understands, logging what it waived. accept_unverified_blobs publishes a restore whose records reference blob files the backup lacks. immutable_source restores from a read-only repository without locking or pinning it. target_database now works with the server running, and verify_backup opens and reads every table, so it fails a backup whose tables this version cannot read, as a restore would. See restore_backup.
Restored Vector Indexes Rebuild Instead of Losing Vectors
A backup does not contain the native file in which an HNSW vector index keeps its graph (the default on an audited table). After restore_backup, if a write reached such an index before any vector search did, the index started a new, empty graph: every vector restored from the backup disappeared from vector search, with no error. Now the restored index rebuilds from the restored records as soon as the database loads. While it rebuilds, vector searches on it return 503.
An index damaged this way by an earlier version stays incomplete after the upgrade. To repair it, stop Harper, delete the index's file from the database directory (<table>%2F<attribute>.hnsw, or <table>%2F<attribute>%40<generation>.hnsw for a table that was dropped and recreated), and start Harper again: the index rebuilds on start. See restoring a backup.
Analytics
The Harper Pro Profiler Samples in Short Windows
Harper Pro's CPU profiler used to sample every worker continuously, so each worker's utilization read about 70% at idle and every worker paid a stack walk every 50 ms. It now samples only for one aggregatePeriod before each capture: from application load until the second capture, two periods later, and then for one period in every thousand. Each hdb_raw_analytics entry whose utilization overlapped sampling carries a new profiler-sampling metric, so a consumer can set that sample aside. Ending each window also stops a wedged V8 profiler thread (harper-pro#1021). analytics.profiling: false still keeps the profiler off. See utilization and the CPU profiler.
Opt-in Full-Text Analytics
Optional analytics.fullText collection reports full-text operation counters, queue occupancy and cached reader observations through the existing fulltext analytics metric. Collection is off by default and requires Harper analytics enabled on a writable node; enabling it applies dynamically without reopening indexes. The implied per-handle cadence is 60 seconds, with histogram recording and reader-byte observation separately opt-in. Rows retain exact decimal counters and the newest observation per worker/index/role. Qualify the cost for your workload; no production overhead bound is claimed. See Full-text analytics and analytics.fullText configuration.
Full-Text Search
Tantivy 0.27 and Existing Index Rebuilds
Harper now uses @harperfast/fulltext 0.9.0 with Tantivy 0.27.0. Harper 5.3.1–5.3.4 ship fulltext 0.5.0. Existing 0.5.0 indexes using current field-based declarations, and prerelease 0.6 indexes, rebuild automatically from authoritative records. Harper 5.3.0 ships fulltext 0.3.0 and is outside this qualified procedure; validate that migration separately. Earlier beta declarations and indexes have no supported migration path. Plan for local full-text unavailability, disk headroom and possible retryable write failures while replay catches up. Stop old processes before starting the new version against the same storage; direct downgrade to fulltext 0.5.x is unsupported.
For the 0.5/0.6 rebuild path, English stemming changes despite the unchanged english@2 analyzer name: added now stems to add rather than ad. Matching and ranking can change. Query modes and schema options are unchanged; benchmark results from the wrapper do not establish application performance parity. Review representative searches and see upgrading existing indexes before restoring traffic.
Harper 5.4.0-alpha.1/alpha.2 indexes using fulltext 0.7.0, and current Harper indexes written by development builds with 0.8.0, can reuse compatible checkpoints with 0.9.0. See upgrading from fulltext 0.7.0 or 0.8.0 for process coordination and the unqualified rollback caveat.
Paging and Search Work Limits
Full-text and autocomplete queries use the existing limit and offset paging options. Harper reads additional native batches when needed after filtering, so the initial native result window no longer silently cuts off eligible results. The new storage.fullText.maxRawHitsPerRequest, storage.fullText.maxResultsPerRequest and storage.fullText.requestBudgetMilliseconds settings bound fetched hits, retained results and execution time. Requests that exceed those budgets return an error. The storage.fullText.runtime section also provides process-wide native-capacity budgets; changes to those budgets require a full Harper process restart. The storage.fullText.query policy configures analyzed terms per leaf, compiled clause count and maximum fuzzy distance; it also requires a full Harper process restart. Defaults remain 64 terms, 256 clauses and distance one. Harper 5.4.0-alpha.1/alpha.2 reject an explicit query-policy block. Separate page requests run independently and can see changes to records or index contents.
See Full-text query execution and storage configuration for limits, restart requirements and recovery guidance.
Full-Text Query Boosts and Optional Scoring
Full-text conditions now accept boost to scale a required text score and should to add optional scoring clauses without expanding the matching records. Use these controls through Table.search(), HTTP QUERY, or search_by_conditions; select $score to return the adjusted relevance score. Optional clauses search the same index and require read access to their source fields. No index rebuild is needed for query-time scoring. See Query-Time Relevance Controls.
Upgrade impact: boost and should are now reserved query-object keys on all tables. Misplaced controls return HTTP 400 instead of being ignored; rename custom keys that previously used these names. Attributes named boost or should remain queryable through the attribute property.
No-Wait Full-Text Searches Wait for Their Reader
A full-text search that passed waitForIndexMilliseconds: 0 could fail with 503 Full-text reader acquisition exceeded the index-wait budget while the node refreshed or reopened its search reader. This happened after index publications and every 30 seconds or so on an idle index. Reader acquisition now has its own 30-second limit, so 0 behaves like an omitted value: the query waits for the reader and then searches. waitForIndexMilliseconds bounds only the wait for index coverage, so a waiting query's total time can exceed it by the time spent acquiring the reader. A query that times out acquiring a reader returns 503 Full-text reader acquisition timed out (#4224).
See freshness controls.
Idle Indexes Keep Their Readers
A full-text index in a database whose transaction log gets no new entries no longer reopens its native reader on every worker about every 30 seconds. Harper's own background commits write nothing to the transaction log. Each one still woke the index's runner, and each wake started a new ownership of the index. A search that arrived while a worker reopened its reader could answer 503. A runner now takes ownership only when the transaction log has moved, a rebuild is requested, or a query waits for coverage. A write to any table in the database moves the log, so the reader is still reopened after a quiet period that ends with such a write. Native vector indexes no longer repeat the cycle either.
The First Search on an Idle Full-Text Index Is Current
A full-text search that did not wait for the index, run on a worker that had not searched that index yet, could return 503 with code: "DERIVED_INDEX_LAGGING" while the index was up to date and had received no writes. This happened with maxIndexLagMilliseconds: 0, and in 5.4.0-beta.3 with the default as well. A retry succeeded. Once a database has gone about 30 seconds without a write to any of its tables, such a search on its full-text indexes now returns its results with Harper-Index-Coverage: current; lag=0. A search with waitForIndexMilliseconds on such a worker no longer makes Harper reopen the index's reader on every worker. A search that arrives just after the quiet period ends, by a write or by a rotation or retention purge of the transaction log, can still return 503 until the index has caught up.
See freshness controls.
Native Addons
Pointer-Compression Runtimes Refuse Incompatible Native Addons
Free and Hobby tier clusters on Harper Fabric run on pointer-compression runtimes; other plans run a standard Node.js. On a Node.js build with V8 pointer compression, Harper now checks native addons before they load, in every thread. It refuses an addon that links V8's C++ API built for the standard ABI, which would otherwise load and then crash the process on first use. The refusal is an IncompatibleNativeAddonError that names the file, its package and the fixes.
These still load as before:
- Node-API addons;
- addons that node-gyp built under the running Node.js, load from the package's
build/directory, and meet theconfig.gypiand binary-age checks in the reference; - packages marked with
.pointer-compression-build.
Standard Node.js builds are unchanged. See Native Addons on Pointer-Compression Runtimes.
Security
An Operations API Source Allowlist for a Fronting Proxy
set_network_access stores a list of allowed source CIDRs for the operations API in the system.hdb_network_access table, which replicates like users and roles, and get_network_access reads it; both are super_user only. Harper publishes the list as allowedSources in the metadata file of each operations API Unix domain socket (tls.unixDomainSockets with operationsApi.network.securePort), and rewrites it on each node the change replicates to. A proxy that reads that file, such as Harper Fabric's, admits only the listed addresses. Harper does not enforce the list on its own listeners. Setting a non-empty list that excludes the request's source address, or when Harper cannot tell that address, fails with 409 unless it passes force: true; on Node.js the local CLI, over the operations domain socket, is never checked. See Network Access (harper#3358).
Intrinsics Are Sealed Instead of Frozen by Default
applications.lockdown has two new modes, seal and seal-after-load, and seal-after-load is the new default for new installs. Sealing stops code from adding or deleting properties on the JavaScript intrinsics (Object, Array, Promise, Map, and the other intrinsics listed under What is locked, plus Function and Function.prototype), which is what a prototype pollution payload such as __proto__.isAdmin needs. Unlike freezing, it leaves existing properties writable: an object can still shadow an inherited method (arr.push = fn), and code can still replace an existing prototype method. That also means a payload that overwrites an existing method, such as __proto__.toString, still succeeds; it mostly breaks behavior rather than granting access, and freeze-after-load remains the mode that blocks it. As under freeze-after-load, components and their dependencies can still extend the intrinsics while they load; once they have loaded, adding a property throws a TypeError in strict-mode code and is ignored in sloppy-mode code.
The seal modes also seal Function.prototype, which the freeze modes leave extensible. Code that adds a property to it after components load can therefore work under freeze-after-load but fail under seal-after-load (throwing in strict-mode code and silently doing nothing in sloppy-mode code).
The default changed because freezing is slow on Node.js. With frozen prototypes, V8 takes a slow path for every store into an array hole on that thread, Harper's own code included: in Harper's benchmark, msgpackr decoding inside a worker ran about four times slower than under seal-after-load. Sealing does not have that cost. freeze-after-load and freeze are unchanged; none and ses are also unchanged. See Intrinsic Lockdown (harper#3552).
WAF Rules After a Full Copy
A node that joined or rejoined a cluster through a full copy could store the copied WAF rules without enforcing them. Nothing reported the gap, and it lasted until the next rule write or a restart. Each worker now recompiles its matcher when the copy completes, and again whenever its rule subscription recovers. A scan or compile that fails now retries with backoff instead of waiting for the next rule write. See harper-pro#889.
A Login Server No Longer Picks the Saved Target
harper login now saves and prints the URL you gave it, and writes that URL to a project .env, never a target named in the server's reply. Before, a login server could choose which target its tokens were saved under and add a line to .env.
Configuration and Key Files Are Readable Only by Harper's User
On Linux and macOS, Harper now creates the configuration file, its backups, the environment-configuration state file, the JWT signing key and its passphrase, and the private keys it generates readable only by the user it runs as (mode 600). Before, they had the mode your umask gave a new file, usually 644, and every configuration change reset it, so a chmod you applied did not last. Now a stricter mode you set on a file Harper owns, such as 400, survives Harper's writes.
Files written by earlier versions keep their modes until Harper next rewrites them, and old backups are never rewritten: run the chmod in File Permissions once after upgrading, and rotate any credential or key other local users could already read. Windows is unchanged (harper#1925).
Subscriptions Follow Attribute Read Permissions
A role with attribute_permissions now receives only the attributes it may read in each subscription message over SSE, WebSocket and MQTT: the current record, live changes, replayed history and published messages. Before, these messages carried every stored attribute. Computed and relationship attributes are left out of these messages, even when the role may read them; a GET can still return them. A message stored as { contentType, data }, which is how Harper stores most MQTT payloads that are not JSON, is filtered like a record, so it arrives as {} unless the role may read an attribute named contentType or data; string and binary values published from code arrive unchanged. For every role, subscription events no longer carry internal storage fields such as metadataFlags and expiresAt, or the audit record's user and nodeId. See Important Rules (harper#3307).
Sealed Operations Through a Relay
encrypted_operation runs an operation that the client sealed to the node's secrets custody key, and seals the result back under a key the client chose, so a proxy or control plane that relays the request sees only ciphertext in both directions. The sealed operation runs as the user who sent the request and is authorized exactly as if sent directly. "mode": "fields" keeps the operation's keys and chosen values readable, so a relay can record what was attempted, and seals the rest. A per-node nonce store and a freshness window refuse replays; operationsApi.encryptedOperation.maxAge and .maxSize size the window and the largest answer. Requires secret custody (Harper Pro). See Encrypted Operations (harper#3288).
Authentication
Local Requests Use a Built-in User
Requests trusted by authorizeLocal, the operations API domain socket (including the local CLI), or raw-TCP MQTT loopback under authorizeLocal now run as a built-in super user named local, independently of stored users. Attribution such as audit entries and job owners now reads local. New users cannot be named local; an existing stored user with that name keeps working. Credential-free token minting still issues tokens for a stored super user and fails with HTTP 409 if none is eligible. Local trust is not saved in sessions. See Local request identity, add_user, and create_authentication_tokens (harper#3755).
Component Bearer Validators
server.registerBearerValidator() lets a component accept Authorization: Bearer tokens Harper did not issue, such as access tokens from its own OAuth authorization server. A validator maps a token to a Harper username; Harper resolves that user through server.getUser() and applies its role on every Harper route on the application port, including REST, GraphQL, WebSocket, MQTT over WebSocket and MCP. See server.registerBearerValidator().
Container Images
Harper Pro Images Run on jemalloc
The harperfast/harper-pro image, its pointer-compression variant (the -pc tags) and the GPU image (harperfast/harper-pro-gpu) now install the distribution's jemalloc package (libjemalloc2, version 5.3.0) and set LD_PRELOAD=libjemalloc.so.2, so every process in the container allocates memory with jemalloc instead of glibc malloc. jemalloc runs with its default settings; the images set no MALLOC_CONF. RocksDB's allocation pattern fragments glibc malloc's heap. In tests on Harper Pro 5.3.4 under a 3 GB memory limit, jemalloc used about 5% less memory (RSS), and its memory grew about half as fast. In short runs without a memory limit it gave YCSB workload A (half reads, half updates) about 5% more throughput, with no measurable difference in peak memory. Expect the memory saving under a memory limit, and not on every workload.
- To run a container on glibc malloc, set
LD_PRELOADto an empty value:docker run -e LD_PRELOAD= harperfast/harper-pro. - A deployment that sets its own
LD_PRELOAD, for example to load an instrumentation library, replaces jemalloc. To keep both, list both:LD_PRELOAD="libjemalloc.so.2 /path/to/other.so". - glibc malloc settings such as
MALLOC_ARENA_MAXhave no effect while jemalloc is preloaded.
The OpenShift image (harperfast/harper-pro-openshift) and the open source harperfast/harper image are unchanged and still use glibc malloc. See harper-pro#1132.
Studio
Component Pages and Cluster Settings for Studio
An application or a plugin can add pages to Studio by declaring them under studio.pages in its config.yaml, and suggest how Studio shows its tabs with studio.navigation. A cluster can hide or tuck away Studio's tabs and choose the page Studio opens on, with the new set_configuration parameters studio_navigation and studio_landing. Studio reads both through the new get_studio_configuration operation, which every signed-in user can call. A role limited to named operations needs it listed, or one of the read_only, standard_user or admin_read groups, which now include it. See Customizing Studio.
studio is now a reserved component name, like sql, because the root configuration's studio entry holds these settings. A deploy named studio now fails with a 400. An application already deployed as studio still loads and can be dropped, which leaves the settings in place; to redeploy it, use another name. (#3656)
Vector Search
Vector thresholds beside other conditions and through relationships
Run HNSW conditions through relationships, and filter a non-leading vector threshold by distance: a vector distance threshold ({ attribute, comparator: 'lt', value, target }) now returns the right records wherever it sits in a query. When another condition was narrower, it compared each coordinate of the stored vector to the threshold instead of the distance, and could return records outside the threshold; it now checks each candidate's exact distance. A threshold on a relationship path (['product', 'embedding']), inside a relationship group (products[...]), or reached through a relationship into another database, used to fail with A target vector must be provided or a transaction error, and now runs. An OR led by a vector condition no longer fails. See Filtering by Distance Threshold.
Upgrade impact: a query whose threshold condition is not its narrowest condition can return fewer records: the ones outside the threshold are gone. An lt or le condition on a vector attribute without target, or a condition with target and another comparator, now returns HTTP 400 wherever it sits, as it already did when it drove the search.
This change is not released yet.
Writes After an Idle Period Are Not Refused
Full-text and native vector indexes no longer refuse writes right after a quiet period. When a database had been idle for longer than an index's lag budget (30 seconds by default), the index counted the idle time as lag at its next commit, so for about a second writes to the indexed table got 503 with code: "DERIVED_INDEX_LAGGING" although the index was caught up. The distance an index measures behind what it has read now starts at the first change it has read but not yet saved, so idle time before that change does not count. See Write backpressure.