> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreflux.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Project Haystack

> Client-mode REST API connector for the Project Haystack HTTP API (SkySpark, FIN Framework, Haxall, Niagara nhaystack, Haystack Connect): about, read, hisRead/hisWrite, pointWrite, watches, nav, defs and invokeAction over JSON grids

Technical Preview

## Overview

**Project Haystack** is the industry standard for semantic tagging of equipment, points and histories in buildings and industrial sites. Compliant servers (SkyFoundry **SkySpark**, J2 Innovations **FIN Framework**, open-source **Haxall**, Tridium Niagara with **nhaystack**, **Haystack Connect**, and others) expose the same HTTP API: ops such as `read`, `hisRead`, `hisWrite`, `pointWrite` and `watchSub` exchange **grids** encoded as Zinc, JSON or Trio.

The **ProjectHaystackApi** connector maps 15 MQTT topics onto that standard ops set (4 meta GETs, 11 POSTs including 3 writes). A Coreflux broker can discover tagged points (`point and temp`), pull or push histories, command writable setpoints through the priority array, and poll watches for live `curVal` changes. Each event publishes the server reply on `<topic>/response`.

| Event | Topic | Request | Use | | | | |
| - | - | - | - | - | - | - | - |
| `About` | `energy/haystack/about/get` | `GET /about` | Server / project about grid (haystackVersion, tz, serverName, productName, ...); empty request | | | | |
| `ListOps` | `energy/haystack/ops/list` | `GET /ops` | Op defs available on this server (standard + vendor extensions) | | | | |
| `ListFiletypes` | `energy/haystack/filetypes/list` | `GET /filetypes` | Supported MIME encodings (text/zinc, application/json, text/trio, text/csv, ...) | | | | |
| `ListLibs` | `energy/haystack/libs/list` | `GET /libs` | Lib defs in the current namespace | | | | |
| `Close` | `energy/haystack/auth/close` | `POST /close` | Invalidate the bearer authToken and close the SCRAM session; body empty grid | | | | |
| `Read` | `energy/haystack/read` | `POST /read` | Entities by filter (rows: filter, optional limit) or by id (rows of id refs); e.g. filter=point and temp | | | | |
| `Nav` | `energy/haystack/nav` | `POST /nav` | Navigation tree children; body row navId (null/omit for root) | | | | |
| `Defs` | `energy/haystack/defs` | `POST /defs` | Def dicts; empty grid for all, or filter/limit row | | | | |
| `HisRead` | `energy/haystack/his/read` | `POST /hisRead` | Historized samples: single (row id+range) or batch (meta.range + id rows); range today | yesterday | \{date} | \{date},\{date} | \{dateTime},\{dateTime} |
| `HisWrite` | `energy/haystack/his/write` | `POST /hisWrite` | WRITE: WRITE: post history samples; single (meta.id + ts/val rows) or batch (v0/v1 cols with id in col meta) | | | | |
| `PointWrite` | `energy/haystack/point/write` | `POST /pointWrite` | WRITE: WRITE: read priority array (row id only) or write a level (id, level 1-17, val or null to auto, optional who/duration) | | | | |
| `WatchSub` | `energy/haystack/watch/sub` | `POST /watchSub` | Create watch (meta watchDis, optional lease) or add ids to existing (meta watchId); response meta.watchId | | | | |
| `WatchPoll` | `energy/haystack/watch/poll` | `POST /watchPoll` | Poll changes for watchId (meta); add refresh Marker for full refresh | | | | |
| `WatchUnsub` | `energy/haystack/watch/unsub` | `POST /watchUnsub` | Remove ids from a watch, or close it (meta watchId + close Marker) | | | | |
| `InvokeAction` | `energy/haystack/action/invoke` | `POST /invokeAction` | WRITE: WRITE: invoke a user action; meta id + action name, row = args | | | | |

Endpoints are relative to `BASE_ADDRESS`, which **already includes** the project or service path (see below).

## Base URL (hosts differ)

Set `WITH BASE_ADDRESS` to your server root **including** the project / service segment. Ops are then `/about`, `/read`, `/hisRead`, …:

| Product | Typical base |
| - | - |
| SkySpark / FIN / Haxall | `https://{host}/api/{project}` e.g. `https://skyspark.example.com/api/demo` |
| nhaystack (Niagara) | `https://{host}/{service}` e.g. `https://niagara.example.com/haystack` |
| Generic / Haystack Connect | `https://{host}/haystack` |

The placeholder in the template is `https://haystack.example.com/api/demo`. Trailing slash is optional.

## Authentication

Haystack uses a pluggable HTTP auth protocol. Compliant servers must support **SCRAM with SHA-256**:

1. Client sends a hello (username) to a protected URI.
2. Server answers `401` with `WWW-Authenticate: SCRAM ...`.
3. After the SCRAM exchange the server returns `authToken` in `Authentication-Info`.
4. Subsequent calls send `Authorization: BEARER authToken=<token>`.

On **nhaystack**, operators often enable `HTTPBasicScheme` and send `Authorization: Basic <base64(user:pass)>` over HTTPS instead.

The multi-step SCRAM hello is **not** a connector event (the REST\_API client does one HTTP call per MQTT publish). Obtain a token with a Haystack client library ([phable](https://phable.dev/), haystack-java, pyhaystack, …) or a small helper script, store the current header value in Hub secret `HAYSTACK_AUTHORIZATION`, and map it with `GET SECRET`. Call `Close` to invalidate the token when done.

Every event sends `Authorization: {value.json.authorization}`.

## Formats (JSON grids)

The connector uses `Accept` / `Content-Type: application/json` (Haystack JSON grids with `"_kind":"grid"`). Zinc (`text/zinc`) is the HTTP API default; Trio and CSV are also standardized. Change the headers if your tooling prefers Zinc.

Request errors that the server can parse still return **HTTP 200** with an **error grid** (`meta.err` Marker + `dis`). Always check for `err` in the reply before treating rows as data.

## Request payload

| Field | Used by | Example |
| - | - | - |
| `authorization` | all events | `(GET SECRET "HAYSTACK_AUTHORIZATION")` → `BEARER authToken=…` or `Basic …` |
| `body` | all POST events | Haystack JSON grid (string or object the broker serializes) |

### Example grids

Empty grid (Close, Nav root, Defs all):

```json theme={"theme":"css-variables","languages":{"custom":["/languages/lot.json"]}}
{"_kind":"grid","meta":{"ver":"3.0"},"cols":[{"name":"empty"}],"rows":[]}
```

Read by filter:

```json theme={"theme":"css-variables","languages":{"custom":["/languages/lot.json"]}}
{"_kind":"grid","meta":{"ver":"3.0"},"cols":[{"name":"filter"},{"name":"limit"}],"rows":[{"filter":"point and temp","limit":100}]}
```

HisRead today:

```json theme={"theme":"css-variables","languages":{"custom":["/languages/lot.json"]}}
{"_kind":"grid","meta":{"ver":"3.0"},"cols":[{"name":"id"},{"name":"range"}],"rows":[{"id":{"_kind":"ref","val":"zoneTemp"},"range":"today"}]}
```

PointWrite level 16:

```json theme={"theme":"css-variables","languages":{"custom":["/languages/lot.json"]}}
{"_kind":"grid","meta":{"ver":"3.0"},"cols":[{"name":"id"},{"name":"level"},{"name":"val"},{"name":"who"}],"rows":[{"id":{"_kind":"ref","val":"sp"},"level":16,"val":{"_kind":"number","val":21,"unit":"°C"},"who":"Coreflux"}]}
```

## Example actions

```lot theme={"theme":"css-variables","languages":{"custom":["/languages/lot.json"]}}
# Example actions for the ProjectHaystackApi route (docs page). Not part of the route template.
# Secret: HAYSTACK_AUTHORIZATION = "BEARER authToken=<token>" after SCRAM hello (or "Basic <base64>" on nhaystack).
# Inline JSON object literals are not supported when publishing to a topic, so the request payload is a COLLAPSED
#   model sent with PUBLISH MODEL. For POST ops the body field is the Haystack JSON grid as a string.

DEFINE MODEL HaystackGetRequest COLLAPSED
    ADD STRING "authorization"

DEFINE MODEL HaystackGridRequest COLLAPSED
    ADD STRING "authorization"
    ADD STRING "body"

DEFINE ACTION HaystackAboutHourly
ON EVERY 1 HOUR DO
    PUBLISH MODEL HaystackGetRequest TO "energy/haystack/about/get" WITH
        authorization = (GET SECRET "HAYSTACK_AUTHORIZATION")

DEFINE ACTION HaystackReadPointsEvery5Minutes
ON EVERY 5 MINUTES DO
    PUBLISH MODEL HaystackGridRequest TO "energy/haystack/read" WITH
        authorization = (GET SECRET "HAYSTACK_AUTHORIZATION")
        body = "{\"_kind\":\"grid\",\"meta\":{\"ver\":\"3.0\"},\"cols\":[{\"name\":\"filter\"},{\"name\":\"limit\"}],\"rows\":[{\"filter\":\"point and temp\",\"limit\":100}]}"

DEFINE ACTION HaystackHisReadDaily
ON EVERY 1 DAY DO
    PUBLISH MODEL HaystackGridRequest TO "energy/haystack/his/read" WITH
        authorization = (GET SECRET "HAYSTACK_AUTHORIZATION")
        body = "{\"_kind\":\"grid\",\"meta\":{\"ver\":\"3.0\"},\"cols\":[{\"name\":\"id\"},{\"name\":\"range\"}],\"rows\":[{\"id\":{\"_kind\":\"ref\",\"val\":\"zoneTemp\"},\"range\":\"today\"}]}"
```

## Writes

`HisWrite`, `PointWrite` (when the body includes `level`) and `InvokeAction` change server state. Restrict which MQTT clients may publish to `energy/haystack/his/write`, `energy/haystack/point/write` and `energy/haystack/action/invoke`. `PointWrite` with an id-only body only **reads** the priority array.

## Watches

1. `WatchSub` with `watchDis` (+ optional `lease`) and id rows → response `meta.watchId`.
2. Poll with `WatchPoll` (`meta.watchId`; add `refresh` Marker for a full refresh).
3. `WatchUnsub` with `close` Marker (or remove individual ids).

Lease expiry on the server closes the watch if you stop polling.

## Not packaged

SkySpark / Haxall extensions such as `eval` and `commit`, other vendor ops beyond the [standard Ops chapter](https://project-haystack.org/doc/docHaystack/Ops), and the SCRAM hello handshake as a connector event.

## Sources

* [Ops](https://project-haystack.org/doc/docHaystack/Ops) · [HTTP API](https://project-haystack.org/doc/docHaystack/HttpApi) · [Authentication](https://project-haystack.org/doc/docHaystack/Auth) · [Ops appendix](https://project-haystack.org/doc/appendix/ops)
* Hand-curated machine-readable cache: `openapi-cache/project-haystack.json` (Project Haystack does not publish an official OpenAPI file)

## Status

**Blocked beta.** Static + mock smoke passed. Unblock with a real host URL + project/service path and SCRAM (or Basic) credentials, then re-run `smoke_haystack.py` with `HAYSTACK_BASE_URL` and `HAYSTACK_AUTHORIZATION`, or complete a broker MQTT round-trip.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.