Project Haystack

Technical Preview

Overview

Project Haystack is the industry standard for semantic tagging of equipment, points and histories in buildings and industrial sites. Compliant servers (SkyFoundry SkySpark, J2 Innovations FIN Framework, open-source Haxall, Tridium Niagara with nhaystack, Haystack Connect, and others) expose the same HTTP API: ops such as read, hisRead, hisWrite, pointWrite and watchSub exchange grids encoded as Zinc, JSON or Trio. The ProjectHaystackApi connector maps 15 MQTT topics onto that standard ops set (4 meta GETs, 11 POSTs including 3 writes). A Coreflux broker can discover tagged points (point and temp), pull or push histories, command writable setpoints through the priority array, and poll watches for live curVal changes. Each event publishes the server reply on <topic>/response.

Event	Topic	Request	Use
`About`	`energy/haystack/about/get`	`GET /about`	Server / project about grid (haystackVersion, tz, serverName, productName, …); empty request
`ListOps`	`energy/haystack/ops/list`	`GET /ops`	Op defs available on this server (standard + vendor extensions)
`ListFiletypes`	`energy/haystack/filetypes/list`	`GET /filetypes`	Supported MIME encodings (text/zinc, application/json, text/trio, text/csv, …)
`ListLibs`	`energy/haystack/libs/list`	`GET /libs`	Lib defs in the current namespace
`Close`	`energy/haystack/auth/close`	`POST /close`	Invalidate the bearer authToken and close the SCRAM session; body empty grid
`Read`	`energy/haystack/read`	`POST /read`	Entities by filter (rows: filter, optional limit) or by id (rows of id refs); e.g. filter=point and temp
`Nav`	`energy/haystack/nav`	`POST /nav`	Navigation tree children; body row navId (null/omit for root)
`Defs`	`energy/haystack/defs`	`POST /defs`	Def dicts; empty grid for all, or filter/limit row
`HisRead`	`energy/haystack/his/read`	`POST /hisRead`	Historized samples: single (row id+range) or batch (meta.range + id rows); range today	yesterday	{date}	{date},{date}	{dateTime},{dateTime}
`HisWrite`	`energy/haystack/his/write`	`POST /hisWrite`	WRITE: WRITE: post history samples; single (meta.id + ts/val rows) or batch (v0/v1 cols with id in col meta)
`PointWrite`	`energy/haystack/point/write`	`POST /pointWrite`	WRITE: WRITE: read priority array (row id only) or write a level (id, level 1-17, val or null to auto, optional who/duration)
`WatchSub`	`energy/haystack/watch/sub`	`POST /watchSub`	Create watch (meta watchDis, optional lease) or add ids to existing (meta watchId); response meta.watchId
`WatchPoll`	`energy/haystack/watch/poll`	`POST /watchPoll`	Poll changes for watchId (meta); add refresh Marker for full refresh
`WatchUnsub`	`energy/haystack/watch/unsub`	`POST /watchUnsub`	Remove ids from a watch, or close it (meta watchId + close Marker)
`InvokeAction`	`energy/haystack/action/invoke`	`POST /invokeAction`	WRITE: WRITE: invoke a user action; meta id + action name, row = args

Endpoints are relative to BASE_ADDRESS, which already includes the project or service path (see below).

Base URL (hosts differ)

Set WITH BASE_ADDRESS to your server root including the project / service segment. Ops are then /about, /read, /hisRead, …:

Product	Typical base
SkySpark / FIN / Haxall	`https://{host}/api/{project}` e.g. `https://skyspark.example.com/api/demo`
nhaystack (Niagara)	`https://{host}/{service}` e.g. `https://niagara.example.com/haystack`
Generic / Haystack Connect	`https://{host}/haystack`

The placeholder in the template is https://haystack.example.com/api/demo. Trailing slash is optional.

Authentication

Haystack uses a pluggable HTTP auth protocol. Compliant servers must support SCRAM with SHA-256:

Client sends a hello (username) to a protected URI.
Server answers 401 with WWW-Authenticate: SCRAM ....
After the SCRAM exchange the server returns authToken in Authentication-Info.
Subsequent calls send Authorization: BEARER authToken=<token>.

On nhaystack, operators often enable HTTPBasicScheme and send Authorization: Basic <base64(user:pass)> over HTTPS instead. The multi-step SCRAM hello is not a connector event (the REST_API client does one HTTP call per MQTT publish). Obtain a token with a Haystack client library (phable, haystack-java, pyhaystack, …) or a small helper script, store the current header value in Hub secret HAYSTACK_AUTHORIZATION, and map it with GET SECRET. Call Close to invalidate the token when done. Every event sends Authorization: {value.json.authorization}.

Formats (JSON grids)

The connector uses Accept / Content-Type: application/json (Haystack JSON grids with "_kind":"grid"). Zinc (text/zinc) is the HTTP API default; Trio and CSV are also standardized. Change the headers if your tooling prefers Zinc. Request errors that the server can parse still return HTTP 200 with an error grid (meta.err Marker + dis). Always check for err in the reply before treating rows as data.

Request payload

Field	Used by	Example
`authorization`	all events	`(GET SECRET "HAYSTACK_AUTHORIZATION")` → `BEARER authToken=…` or `Basic …`
`body`	all POST events	Haystack JSON grid (string or object the broker serializes)

Example grids

Empty grid (Close, Nav root, Defs all):

{"_kind":"grid","meta":{"ver":"3.0"},"cols":[{"name":"empty"}],"rows":[]}

Read by filter:

{"_kind":"grid","meta":{"ver":"3.0"},"cols":[{"name":"filter"},{"name":"limit"}],"rows":[{"filter":"point and temp","limit":100}]}

HisRead today:

{"_kind":"grid","meta":{"ver":"3.0"},"cols":[{"name":"id"},{"name":"range"}],"rows":[{"id":{"_kind":"ref","val":"zoneTemp"},"range":"today"}]}

PointWrite level 16:

{"_kind":"grid","meta":{"ver":"3.0"},"cols":[{"name":"id"},{"name":"level"},{"name":"val"},{"name":"who"}],"rows":[{"id":{"_kind":"ref","val":"sp"},"level":16,"val":{"_kind":"number","val":21,"unit":"°C"},"who":"Coreflux"}]}

Example actions

# Example actions for the ProjectHaystackApi route (docs page). Not part of the route template.
# Secret: HAYSTACK_AUTHORIZATION = "BEARER authToken=<token>" after SCRAM hello (or "Basic <base64>" on nhaystack).
# Inline JSON object literals are not supported when publishing to a topic, so the request payload is a COLLAPSED
#   model sent with PUBLISH MODEL. For POST ops the body field is the Haystack JSON grid as a string.

DEFINE MODEL HaystackGetRequest COLLAPSED
    ADD STRING "authorization"

DEFINE MODEL HaystackGridRequest COLLAPSED
    ADD STRING "authorization"
    ADD STRING "body"

DEFINE ACTION HaystackAboutHourly
ON EVERY 1 HOUR DO
    PUBLISH MODEL HaystackGetRequest TO "energy/haystack/about/get" WITH
        authorization = (GET SECRET "HAYSTACK_AUTHORIZATION")

DEFINE ACTION HaystackReadPointsEvery5Minutes
ON EVERY 5 MINUTES DO
    PUBLISH MODEL HaystackGridRequest TO "energy/haystack/read" WITH
        authorization = (GET SECRET "HAYSTACK_AUTHORIZATION")
        body = "{\"_kind\":\"grid\",\"meta\":{\"ver\":\"3.0\"},\"cols\":[{\"name\":\"filter\"},{\"name\":\"limit\"}],\"rows\":[{\"filter\":\"point and temp\",\"limit\":100}]}"

DEFINE ACTION HaystackHisReadDaily
ON EVERY 1 DAY DO
    PUBLISH MODEL HaystackGridRequest TO "energy/haystack/his/read" WITH
        authorization = (GET SECRET "HAYSTACK_AUTHORIZATION")
        body = "{\"_kind\":\"grid\",\"meta\":{\"ver\":\"3.0\"},\"cols\":[{\"name\":\"id\"},{\"name\":\"range\"}],\"rows\":[{\"id\":{\"_kind\":\"ref\",\"val\":\"zoneTemp\"},\"range\":\"today\"}]}"

Writes

HisWrite, PointWrite (when the body includes level) and InvokeAction change server state. Restrict which MQTT clients may publish to energy/haystack/his/write, energy/haystack/point/write and energy/haystack/action/invoke. PointWrite with an id-only body only reads the priority array.

Watches

WatchSub with watchDis (+ optional lease) and id rows → response meta.watchId.
Poll with WatchPoll (meta.watchId; add refresh Marker for a full refresh).
WatchUnsub with close Marker (or remove individual ids).

Lease expiry on the server closes the watch if you stop polling.

Not packaged

SkySpark / Haxall extensions such as eval and commit, other vendor ops beyond the standard Ops chapter, and the SCRAM hello handshake as a connector event.

Sources

Ops · HTTP API · Authentication · Ops appendix
Hand-curated machine-readable cache: openapi-cache/project-haystack.json (Project Haystack does not publish an official OpenAPI file)

Status

Blocked beta. Static + mock smoke passed. Unblock with a real host URL + project/service path and SCRAM (or Basic) credentials, then re-run smoke_haystack.py with HAYSTACK_BASE_URL and HAYSTACK_AUTHORIZATION, or complete a broker MQTT round-trip.

Quick Start

Coreflux MQTT Broker

LoT Language

Coreflux HUB

VS Code - LoT Notebooks

Coreflux Portal

Using AI with Coreflux

Product Updates

Overview

Base URL (hosts differ)

Authentication

Formats (JSON grids)

Request payload

Example grids

Example actions

Writes

Watches

Not packaged

Sources

Status

​Overview

​Base URL (hosts differ)

​Authentication

​Formats (JSON grids)

​Request payload

​Example grids

​Example actions

​Writes

​Watches

​Not packaged

​Sources

​Status

Overview

Base URL (hosts differ)

Authentication

Formats (JSON grids)

Request payload

Example grids

Example actions

Writes

Watches

Not packaged

Sources

Status