End of day updates, almost at completion of live cron setup with bash script wrapper now as ell

PROJECT-PROGRESS.md

@@ -1,73 +1,158 @@
 # ServiceM8 Project Progress
 
-## Current Quote Template → JobMaterials Pipeline
+## Current Direction — Soft Release Poller + Apply Pipeline
 
-### Live receiver
+We changed tune from relying mainly on ServiceM8 `form.response_created` webhooks to using a scheduled API poller as the operational safety-net/primary soft-release path.
-- `servicem8_webhook_receiver.py`
-  - receives `form.response_created`
-  - stores the raw webhook payload in `servicem8_webhooks.db`
-  - checks incoming form responses for the Quote Template `form_uuid`
-  - if matched, parses and queues the derived jobMaterials payload to:
-    - `quote-template-jobmaterials-queue.jsonl`
-  - does **not** perform live ServiceM8 writes at this stage
 
-### Parser
+Reason: ServiceM8 form webhooks are useful, but during testing some expected form completions did not reliably appear at the FastAPI listener. The ServiceM8 API endpoint `/api_1.0/formresponse.json` can be queried with a timestamp filter, so the safer operational pattern is now:
-- `servicem8-quote-template-parser.py`
+
-  - parses Quote Template `field_data`
+1. poll recent form responses with `timestamp gt 'YYYY-MM-DD HH:MM:SS'`
-  - extracts:
+2. store every returned API row locally
+3. detect Quote Template form responses
+4. parse them into desired jobMaterial rows
+5. apply only previously unapplied Quote Template responses to ServiceM8
+6. track created jobMaterial UUIDs locally to avoid duplicate applies
+
+This keeps the live webhook receiver useful as capture/diagnostics, but no longer depends on webhook delivery for completeness.
+
+## Active Soft-Release Flow
+
+### 1. Poll ServiceM8 form responses
+- Script: `poll_form_responses_since.py`
+- API endpoint: `/api_1.0/formresponse.json`
+- Confirmed filter syntax:
+  - `?$filter=timestamp gt '2026-05-04 10:00:00'`
+- Default operational DB:
+  - `servicem8_formresponse_poll.db`
+- Poll queue/output:
+  - `quote-template-jobmaterials-poll-queue.jsonl`
+- Quote Template form UUID:
+  - `3621b6be-1d19-4756-9ab4-9d5e4120f6d9`
+
+The poller stores all fetched form responses in `form_responses_raw`, then parses Quote Template matches into `quote_template_form_responses`.
+
+### 2. Parse Quote Template responses
+- Parser: `servicem8_quote_template_parser.py`
+- Extracts:
   - description of works
   - `Item 1..12` include lines
   - `Works excluded 1..4` exclude lines
-    - extra descriptive include rows such as labour/scaffolding/equipment fields
+  - extra descriptive include rows such as labour/materials/scaffolding/equipment fields
-  - builds normalized `desired_job_materials`
+- Builds normalized `desired_job_materials` rows.
 
-### Create/apply script
+### 3. Select/apply parsed Quote Template responses
-- `servicem8-create-jobmaterials-from-form-response.py`
+- Script: `apply_polled_quote_template_jobmaterials.py`
-  - standalone script
+- Safe behaviour built in:
-  - consumes a Quote Template form response JSON payload
+  - processes one `form_response_uuid` at a time when called directly
-  - builds ServiceM8 Job Material API payloads
+  - dry-run by default
-  - runs in **dry-run by default**
+  - `--apply` performs live ServiceM8 `jobmaterial.json` creates
-  - supports live create later with `--apply`
+  - refuses duplicate apply when generated material rows already exist for that form response unless `--force` is used
-  - records created/generated mappings into local state DB
+- Apply tracking tables in `servicem8_formresponse_poll.db`:
+  - `quote_template_apply_runs`
+  - `quote_template_apply_run_rows`
+- Created ServiceM8 job material mappings are recorded in:
+  - `servicem8_quote_materials_state.db`
 
-### Local state tracking
+Current apply payload rules:
-- `servicem8_quote_materials_state.db`
+- All rows get:
-  - local SQLite DB for tracking generated jobMaterials
+  - `tax_rate_uuid = 84e4dd28-06b3-452b-a796-1f58a20ac49b`
-  - intended fields include:
+  - `quantity = "0"`
-    - job UUID
+  - `price = ""`
-    - form response UUID
+  - `displayed_amount = ""`
-    - created job material UUID
+  - `displayed_amount_is_tax_inclusive = ""`
-    - kind/source metadata
+- Header material UUIDs:
+  - `include_header` → `1924893b-917f-474a-adaa-2093bd622d4b`
+  - `exclude_header` → `4947bfd7-4875-48f7-9caf-2093b9751b9b`
+- Non-header quote rows currently use:
+  - `f78b1d23-b9fa-40fe-a806-2425fe09cc0b`
 
-### Queue/prepared output
+### 4. Wrapper for scheduled/operational use
-- `quote-template-jobmaterials-queue.jsonl`
+- Script: `poll_and_apply_quote_templates.sh`
-  - lightweight queue/output file written by webhook stage
+- Default behaviour:
-  - contains parsed/prepared `desired_job_materials` objects
+  - polls the last 24 hours from script start
-  - no live update performed yet
+  - stores/parses results
+  - applies any parsed Quote Template responses that are not already marked/applied
+  - logs each run under `logs/poll-and-apply-YYYYMMDD-HHMMSS.log`
+- Examples:
+  - `./poll_and_apply_quote_templates.sh`
+  - `./poll_and_apply_quote_templates.sh --hours 48`
+  - `./poll_and_apply_quote_templates.sh --since '2026-05-04 08:00:00'`
 
-### Inspector
+This is the proposed scheduled entry point for soft release, e.g. every 10–30 minutes.
-- `servicem8_inspector.py`
+
-  - read-only browser for webhook DB
+## Live Webhook Receiver Status
-  - now also includes visibility of generated-materials state DB
+
+### Receiver
+- Script: `servicem8_webhook_receiver.py`
+- Still receives/stores:
+  - event webhooks
+  - object webhooks
+  - form-response webhooks
+- DB:
+  - `servicem8_webhooks.db`
+- Quote Template webhook queue:
+  - `quote-template-jobmaterials-queue.jsonl`
+
+Important: webhook handling still does **not** perform live ServiceM8 writes. Heavy work stays outside the FastAPI webhook request path.
+
+## Inspector / Web Viewer
+
+- Script: `servicem8_inspector.py`
+- Current intended views include:
+  - webhook events
+  - webhook objects
+  - webhook form responses
+  - polled form responses
+  - parsed polled Quote Template responses
+  - poll runs
+  - dry-run/apply runs
+  - generated material state
+
+Note: the code was updated and tested on a temporary localhost port, but the existing live inspector process may need a manual/service restart before all new pages appear in the running viewer.
+
+## Required Files for Current Soft Release
+
+### Scripts
+- `poll_and_apply_quote_templates.sh` — scheduled wrapper / main operational entry point
+- `poll_form_responses_since.py` — polls ServiceM8 and populates poll DB
+- `apply_polled_quote_template_jobmaterials.py` — applies parsed responses to ServiceM8
+- `servicem8_quote_template_parser.py` — parsing logic
+- `servicem8_inspector.py` — web inspection/progress viewer
+- `servicem8_webhook_receiver.py` — still useful for webhook capture/diagnostics
+
+### Databases / state
+- `servicem8_formresponse_poll.db` — poll results, parsed quote responses, apply run status
+- `servicem8_quote_materials_state.db` — created jobMaterial mapping/state to avoid duplicates
+- `servicem8_webhooks.db` — webhook capture/archive/diagnostics
+
+### Queue/log files
+- `quote-template-jobmaterials-poll-queue.jsonl` — poll-derived parsed queue/output
+- `logs/poll-and-apply-*.log` — wrapper run logs
+- `quote-template-jobmaterials-queue.jsonl` — older webhook-derived queue; still useful but not the primary soft-release path
 
 ## Current Status
-Everything is staged and connected up to the point of:
-- webhook receive
-- UUID trigger check
-- form parsing
-- local queueing
-- dry-run jobMaterial payload generation
-- local state DB support
-- inspector visibility
 
-## Not Yet Enabled
+Operational soft-release pieces are now in place:
-- actual live POST creation of Job Materials into ServiceM8 during webhook processing
+- API polling confirmed
-- any automatic update/delete reconciliation against live ServiceM8 records
+- Quote Template detection confirmed
+- parsing confirmed
+- dry-run/apply command path tested
+- payload adjusted for current ServiceM8 requirements
+- duplicate-apply guard in place
+- wrapper script created for scheduled operation
+- inspector updated for progress visibility
+
+## Not Yet Done / Next Steps
+
+- Restart/reload the live inspector process so the new poll/apply pages are available in the active web viewer.
+- Decide schedule interval for `poll_and_apply_quote_templates.sh` — likely every 10 or 30 minutes.
+- Run the wrapper manually for a soft-release smoke test with a controlled recent form response.
+- After confidence builds, wire the wrapper into cron/system scheduling.
+- Future hardening: add reconciliation/update/delete behaviour if ServiceM8 quote form responses are edited after initial apply.
 
 ## Design Notes
-- Heavy lifting is intentionally kept **out of the live webhook handler**.
+
-- The webhook handler is used only for:
+- Webhooks remain lightweight and non-mutating.
-  - capture
+- Polling is now the reliable source of completeness.
-  - UUID gate
+- Applying to ServiceM8 is tracked locally and guarded against duplicates.
-  - parse/prepare/queue
+- The wrapper intentionally skips dry-run for soft release, but the underlying apply script still supports dry-run and duplicate protection.
-- Live ServiceM8 mutation remains a separate step/script for safety.

apply_polled_quote_template_jobmaterials.py

@@ -0,0 +1,399 @@
+#!/usr/bin/env python3
+"""Create ServiceM8 jobMaterials for one selected polled Quote Template response.
+
+Safe-by-default:
+- Requires an explicit form response UUID.
+- Dry-run unless --apply is provided.
+- Refuses to apply twice for the same form response unless --force is used.
+- Records every dry-run/apply attempt in the poll DB for inspector visibility.
+"""
+
+from __future__ import annotations
+
+import argparse
+import importlib.util
+import json
+import os
+import sqlite3
+import sys
+from contextlib import closing
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Dict, List, Optional
+
+import requests
+
+from servicem8_quote_template_parser import STATE_DB_PATH, init_state_db, record_generated_job_material
+
+SCRIPT_DIR = Path(__file__).resolve().parent
+POLL_DB_PATH = Path(os.getenv("WEBHOOK_POLL_DB_PATH", SCRIPT_DIR / "servicem8_formresponse_poll.db"))
+BASE_URL = os.getenv("SERVICEM8_BASE_URL", "https://api.servicem8.com/api_1.0")
+REQUEST_TIMEOUT = int(os.getenv("SERVICEM8_TIMEOUT", "30"))
+DEV_QUOTE_MATERIAL_UUID = "f78b1d23-b9fa-40fe-a806-2425fe09cc0b"
+QUOTE_INCLUDE_HEADER_MATERIAL_UUID = "1924893b-917f-474a-adaa-2093bd622d4b"
+QUOTE_EXCLUDE_HEADER_MATERIAL_UUID = "4947bfd7-4875-48f7-9caf-2093b9751b9b"
+DEV_QUOTE_TAX_RATE_UUID = "84e4dd28-06b3-452b-a796-1f58a20ac49b"
+
+
+def utc_now() -> str:
+    return datetime.now(timezone.utc).isoformat()
+
+
+def material_uuid_for_row(row: dict) -> str:
+    kind = row.get("kind", "")
+    if kind == "include_header":
+        return QUOTE_INCLUDE_HEADER_MATERIAL_UUID
+    if kind == "exclude_header":
+        return QUOTE_EXCLUDE_HEADER_MATERIAL_UUID
+    return DEV_QUOTE_MATERIAL_UUID
+
+
+def build_payload(job_uuid: str, row: dict) -> dict:
+    return {
+        "job_uuid": job_uuid,
+        "material_uuid": material_uuid_for_row(row),
+        "tax_rate_uuid": DEV_QUOTE_TAX_RATE_UUID,
+        "name": row["name"],
+        "quantity": "0",
+        "price": "",
+        "displayed_amount": "",
+        "displayed_amount_is_tax_inclusive": "",
+        "sort_order": row["sort_order"],
+    }
+
+
+def create_job_material(session: requests.Session, payload: dict) -> str:
+    response = session.post(f"{BASE_URL}/jobmaterial.json", json=payload, timeout=REQUEST_TIMEOUT)
+    if not response.ok:
+        raise RuntimeError(f"Create failed: HTTP {response.status_code} :: {response.text}")
+    record_uuid = response.headers.get("x-record-uuid", "")
+    if not record_uuid:
+        raise RuntimeError("Create succeeded but x-record-uuid header was missing")
+    return record_uuid
+
+
+def load_api_key() -> str:
+    for name in ("SERVICEM8_ACCESS_TOKEN", "SERVICEM8_API_KEY"):
+        value = os.getenv(name)
+        if value:
+            return value
+
+    fallback = SCRIPT_DIR / "servicem8-list-webhook-subscriptions-table.py"
+    if fallback.exists():
+        spec = importlib.util.spec_from_file_location("servicem8_subs", fallback)
+        if spec and spec.loader:
+            module = importlib.util.module_from_spec(spec)
+            spec.loader.exec_module(module)  # type: ignore[union-attr]
+            value = getattr(module, "ACCESS_TOKEN", None)
+            if value:
+                return str(value)
+
+    raise RuntimeError("Missing SERVICEM8_ACCESS_TOKEN or SERVICEM8_API_KEY")
+
+
+def get_conn(db_path: Path = POLL_DB_PATH) -> sqlite3.Connection:
+    conn = sqlite3.connect(db_path)
+    conn.row_factory = sqlite3.Row
+    return conn
+
+
+def init_apply_tables(conn: sqlite3.Connection) -> None:
+    conn.execute(
+        """
+        CREATE TABLE IF NOT EXISTS quote_template_apply_runs (
+            id INTEGER PRIMARY KEY AUTOINCREMENT,
+            form_response_uuid TEXT NOT NULL,
+            job_uuid TEXT NOT NULL,
+            mode TEXT NOT NULL,
+            started_at TEXT NOT NULL,
+            finished_at TEXT,
+            desired_count INTEGER NOT NULL DEFAULT 0,
+            created_count INTEGER NOT NULL DEFAULT 0,
+            status TEXT NOT NULL,
+            error TEXT
+        )
+        """
+    )
+    conn.execute(
+        """
+        CREATE TABLE IF NOT EXISTS quote_template_apply_run_rows (
+            id INTEGER PRIMARY KEY AUTOINCREMENT,
+            run_id INTEGER NOT NULL,
+            form_response_uuid TEXT NOT NULL,
+            job_uuid TEXT NOT NULL,
+            row_index INTEGER NOT NULL,
+            kind TEXT,
+            source_question TEXT,
+            name TEXT,
+            api_payload_json TEXT NOT NULL,
+            action TEXT NOT NULL,
+            job_material_uuid TEXT,
+            error TEXT,
+            created_at TEXT NOT NULL,
+            FOREIGN KEY(run_id) REFERENCES quote_template_apply_runs(id)
+        )
+        """
+    )
+    conn.execute("CREATE INDEX IF NOT EXISTS idx_quote_apply_runs_form_response ON quote_template_apply_runs(form_response_uuid)")
+    conn.execute("CREATE INDEX IF NOT EXISTS idx_quote_apply_rows_run ON quote_template_apply_run_rows(run_id)")
+    conn.commit()
+
+
+def load_quote(conn: sqlite3.Connection, form_response_uuid: str) -> sqlite3.Row:
+    row = conn.execute(
+        "select * from quote_template_form_responses where form_response_uuid = ?",
+        (form_response_uuid,),
+    ).fetchone()
+    if not row:
+        raise SystemExit(f"No parsed polled Quote Template response found for UUID: {form_response_uuid}")
+    return row
+
+
+def existing_created_for_form(form_response_uuid: str) -> int:
+    if not STATE_DB_PATH.exists():
+        return 0
+    try:
+        with closing(sqlite3.connect(STATE_DB_PATH)) as conn:
+            row = conn.execute(
+                "select count(*) from generated_job_materials where form_response_uuid = ?",
+                (form_response_uuid,),
+            ).fetchone()
+            return int(row[0] if row else 0)
+    except sqlite3.Error:
+        return 0
+
+
+def create_apply_run(conn: sqlite3.Connection, *, form_response_uuid: str, job_uuid: str, mode: str, desired_count: int) -> int:
+    cur = conn.execute(
+        """
+        INSERT INTO quote_template_apply_runs (
+            form_response_uuid, job_uuid, mode, started_at, desired_count, status
+        ) VALUES (?, ?, ?, ?, ?, ?)
+        """,
+        (form_response_uuid, job_uuid, mode, utc_now(), desired_count, "running"),
+    )
+    conn.commit()
+    return int(cur.lastrowid)
+
+
+def finish_apply_run(conn: sqlite3.Connection, run_id: int, *, status: str, created_count: int, error: Optional[str] = None) -> None:
+    conn.execute(
+        """
+        UPDATE quote_template_apply_runs
+        SET finished_at = ?, status = ?, created_count = ?, error = ?
+        WHERE id = ?
+        """,
+        (utc_now(), status, created_count, error, run_id),
+    )
+    conn.commit()
+
+
+def record_apply_row(
+    conn: sqlite3.Connection,
+    *,
+    run_id: int,
+    form_response_uuid: str,
+    job_uuid: str,
+    row_index: int,
+    row: Dict[str, Any],
+    api_payload: Dict[str, Any],
+    action: str,
+    job_material_uuid: str = "",
+    error: Optional[str] = None,
+) -> None:
+    conn.execute(
+        """
+        INSERT INTO quote_template_apply_run_rows (
+            run_id, form_response_uuid, job_uuid, row_index, kind, source_question,
+            name, api_payload_json, action, job_material_uuid, error, created_at
+        ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+        """,
+        (
+            run_id,
+            form_response_uuid,
+            job_uuid,
+            row_index,
+            row.get("kind", ""),
+            row.get("source_question", ""),
+            row.get("name", ""),
+            json.dumps(api_payload, ensure_ascii=False, sort_keys=True),
+            action,
+            job_material_uuid,
+            error,
+            utc_now(),
+        ),
+    )
+    conn.commit()
+
+
+def list_pending(conn: sqlite3.Connection) -> List[Dict[str, Any]]:
+    rows = conn.execute(
+        """
+        select form_response_uuid, discovered_at, job_uuid, description, queued_at,
+               processed_at, process_status, desired_job_materials_json
+        from quote_template_form_responses
+        order by discovered_at desc
+        """
+    ).fetchall()
+    out = []
+    for row in rows:
+        existing = existing_created_for_form(row["form_response_uuid"])
+        try:
+            desired_count = len(json.loads(row["desired_job_materials_json"] or "[]"))
+        except Exception:
+            desired_count = None
+        out.append(
+            {
+                "form_response_uuid": row["form_response_uuid"],
+                "discovered_at": row["discovered_at"],
+                "job_uuid": row["job_uuid"],
+                "description": row["description"],
+                "desired_count": desired_count,
+                "created_in_state_db": existing,
+                "processed_at": row["processed_at"],
+                "process_status": row["process_status"],
+            }
+        )
+    return out
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description="Dry-run or apply one selected polled Quote Template response to ServiceM8 jobMaterials")
+    parser.add_argument("--uuid", help="Polled Quote Template form_response_uuid to process")
+    parser.add_argument("--db", default=str(POLL_DB_PATH), help="Poll DB path")
+    parser.add_argument("--apply", action="store_true", help="Actually create ServiceM8 jobMaterial records. Default is dry-run.")
+    parser.add_argument("--force", action="store_true", help="Allow apply even if generated_job_materials already contains rows for this form response")
+    parser.add_argument("--list", action="store_true", help="List parsed polled Quote Template responses and exit")
+    parser.add_argument("--pretty", action="store_true", help="Pretty-print JSON output")
+    args = parser.parse_args()
+
+    db_path = Path(args.db)
+    if not db_path.exists():
+        raise SystemExit(f"Poll DB not found: {db_path}")
+
+    with closing(get_conn(db_path)) as conn:
+        init_apply_tables(conn)
+
+        if args.list:
+            print(json.dumps({"ok": True, "rows": list_pending(conn)}, indent=2 if args.pretty else None, ensure_ascii=False))
+            return 0
+
+        if not args.uuid:
+            raise SystemExit("--uuid is required unless using --list")
+
+        quote = load_quote(conn, args.uuid)
+        form_response_uuid = quote["form_response_uuid"]
+        job_uuid = quote["job_uuid"]
+        desired_rows = json.loads(quote["desired_job_materials_json"] or "[]")
+        mode = "apply" if args.apply else "dry-run"
+
+        existing_created = existing_created_for_form(form_response_uuid)
+        if args.apply and existing_created and not args.force:
+            raise SystemExit(
+                f"Refusing duplicate apply: {existing_created} generated_job_materials already recorded for {form_response_uuid}. Use --force only if intentional."
+            )
+
+        run_id = create_apply_run(
+            conn,
+            form_response_uuid=form_response_uuid,
+            job_uuid=job_uuid,
+            mode=mode,
+            desired_count=len(desired_rows),
+        )
+
+        result = {
+            "ok": True,
+            "mode": mode,
+            "run_id": run_id,
+            "form_response_uuid": form_response_uuid,
+            "job_uuid": job_uuid,
+            "description": quote["description"],
+            "desired_count": len(desired_rows),
+            "created_count": 0,
+            "state_db_path": str(STATE_DB_PATH),
+            "rows": [],
+        }
+
+        try:
+            if not args.apply:
+                for idx, row in enumerate(desired_rows, start=1):
+                    api_payload = build_payload(job_uuid, row)
+                    record_apply_row(
+                        conn,
+                        run_id=run_id,
+                        form_response_uuid=form_response_uuid,
+                        job_uuid=job_uuid,
+                        row_index=idx,
+                        row=row,
+                        api_payload=api_payload,
+                        action="would_create",
+                    )
+                    result["rows"].append({"action": "would_create", "kind": row.get("kind"), "payload": api_payload})
+                finish_apply_run(conn, run_id, status="dry-run", created_count=0)
+                conn.execute(
+                    "UPDATE quote_template_form_responses SET process_status = ? WHERE form_response_uuid = ?",
+                    ("dry-run", form_response_uuid),
+                )
+                conn.commit()
+                print(json.dumps(result, indent=2 if args.pretty else None, ensure_ascii=False))
+                return 0
+
+            api_key = load_api_key()
+            init_state_db(STATE_DB_PATH)
+            session = requests.Session()
+            session.headers.update({"X-Api-Key": api_key, "Accept": "application/json", "Content-Type": "application/json"})
+
+            created_count = 0
+            for idx, row in enumerate(desired_rows, start=1):
+                api_payload = build_payload(job_uuid, row)
+                created_uuid = create_job_material(session, api_payload)
+                created_count += 1
+                record_generated_job_material(
+                    job_uuid=job_uuid,
+                    form_response_uuid=form_response_uuid,
+                    job_material_uuid=created_uuid,
+                    kind=row.get("kind", ""),
+                    source_field_uuid=row.get("source_field_uuid", ""),
+                    source_question=row.get("source_question", ""),
+                    source_text=row.get("name", ""),
+                    db_path=STATE_DB_PATH,
+                )
+                record_apply_row(
+                    conn,
+                    run_id=run_id,
+                    form_response_uuid=form_response_uuid,
+                    job_uuid=job_uuid,
+                    row_index=idx,
+                    row=row,
+                    api_payload=api_payload,
+                    action="created",
+                    job_material_uuid=created_uuid,
+                )
+                result["rows"].append({"action": "created", "kind": row.get("kind"), "job_material_uuid": created_uuid, "payload": api_payload})
+
+            result["created_count"] = created_count
+            finish_apply_run(conn, run_id, status="applied", created_count=created_count)
+            conn.execute(
+                "UPDATE quote_template_form_responses SET processed_at = ?, process_status = ?, process_error = NULL WHERE form_response_uuid = ?",
+                (utc_now(), "applied", form_response_uuid),
+            )
+            conn.commit()
+            print(json.dumps(result, indent=2 if args.pretty else None, ensure_ascii=False))
+            return 0
+
+        except Exception as exc:
+            finish_apply_run(conn, run_id, status="error", created_count=result.get("created_count", 0), error=str(exc))
+            conn.execute(
+                "UPDATE quote_template_form_responses SET process_status = ?, process_error = ? WHERE form_response_uuid = ?",
+                ("error", str(exc), form_response_uuid),
+            )
+            conn.commit()
+            raise
+
+
+if __name__ == "__main__":
+    try:
+        raise SystemExit(main())
+    except Exception as exc:
+        print(str(exc), file=sys.stderr)
+        raise SystemExit(1)

docs/list-all-form-responses.md

@@ -0,0 +1,411 @@
+# List all Form Responses
+
+
+			
+#### Filtering
+This endpoint supports result filtering. For more information on how to filter this request, [go here](/docs/filtering).
+			
+			
+#### OAuth Scope
+This endpoint requires the following OAuth scope **read_forms**.
+
+			
+
+# OpenAPI definition
+
+```json
+{
+  "openapi": "3.1.0",
+  "info": {
+    "title": "ServiceM8 API",
+    "description": "Move your app forward with the ServiceM8 API\n\n\n\n## Limits and Throttling\nTo ensure continuous quality of service, API usage can be subject to throttling. The throttle will be applied once an API consumer reaches a certain \nthreshold in terms of a maximum number of requests per minute. Most clients will never hit this threshold, but those that do, will get met by a \nHTTP 429 Too Many Requests response code. \n \nThere is a limit of 180 requests per minute, if you reach this you will receive a HTTP 429 with a text body of \"Number of allowed API requests per minute exceeded\".\nThere is a limit of 20000 requests per day, if you reach this you will receive a HTTP 429 with a text body of \"Number of allowed API requests per day exceeded\".\n\nWe encourage all API developers to anticipate this error, and take appropriate measures like e.g. using a cached value from a previous call, or passing on a message to the end user that gets subjected to this behaviour (if any).\n\nLimits are per Addon per account.\n",
+    "termsOfService": "https://www.servicem8.com/terms-of-service",
+    "version": "1.0.0"
+  },
+  "servers": [
+    {
+      "url": "https://api.servicem8.com/api_1.0"
+    }
+  ],
+  "security": [
+    {
+      "apiKey": []
+    },
+    {
+      "oauth2": []
+    }
+  ],
+  "paths": {
+    "/formresponse.json": {
+      "get": {
+        "tags": [
+          "Form Responses"
+        ],
+        "operationId": "listFormResponses",
+        "summary": "List all Form Responses",
+        "description": "\n\t\t\t\n#### Filtering\nThis endpoint supports result filtering. For more information on how to filter this request, [go here](/docs/filtering).\n\t\t\t\n\t\t\t\n#### OAuth Scope\nThis endpoint requires the following OAuth scope **read_forms**.\n\n\t\t\t",
+        "security": [
+          {
+            "apiKey": []
+          },
+          {
+            "oauth2": [
+              "read_forms"
+            ]
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "An array of Form Responses",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "array",
+                  "items": {
+                    "$ref": "#/components/schemas/FormResponse"
+                  }
+                },
+                "examples": {
+                  "success": {
+                    "value": [
+                      {
+                        "uuid": "123e4567-af16-4618-80c7-23f940cc449b",
+                        "active": 1,
+                        "edit_date": "2026-03-01 12:00:00",
+                        "form_uuid": "123e4567-f8b2-4802-a8db-23f94cf50e1b",
+                        "staff_uuid": "123e4567-5c89-4cb3-8dc0-23f94d75521b",
+                        "regarding_object": "string",
+                        "regarding_object_uuid": "123e4567-016e-482c-a3f7-23f943f6d72b",
+                        "field_data": "string",
+                        "timestamp": "2026-03-01 12:00:00",
+                        "form_by_staff_uuid": "123e4567-44e8-448d-925c-23f9435b205b",
+                        "document_attachment_uuid": "123e4567-2f15-479f-8791-23f94c0fbbbb",
+                        "asset_uuid": "123e4567-89f4-4183-ae1e-23f94832e59b"
+                      }
+                    ]
+                  }
+                }
+              }
+            }
+          },
+          "400": {
+            "description": "Bad Request - The request is malformed or contains invalid parameters",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/Error"
+                },
+                "examples": {
+                  "badRequest": {
+                    "value": {
+                      "errorCode": "1000",
+                      "message": "An error occurred completing your request"
+                    }
+                  }
+                }
+              }
+            }
+          },
+          "401": {
+            "description": "Unauthorized - Authentication credentials are missing or invalid",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/AuthenticationError"
+                },
+                "examples": {
+                  "unauthorized": {
+                    "value": {
+                      "errorCode": "401",
+                      "message": "Authentication failed. Please check your API key or OAuth token."
+                    }
+                  }
+                }
+              }
+            }
+          },
+          "403": {
+            "description": "Forbidden - You don't have permission to access this resource",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/ForbiddenError"
+                },
+                "examples": {
+                  "forbidden": {
+                    "value": {
+                      "errorCode": "403",
+                      "message": "Access forbidden. You don't have permission to access this resource."
+                    }
+                  }
+                }
+              }
+            }
+          },
+          "429": {
+            "description": "Too Many Requests - You have exceeded the rate limit",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/RateLimitError"
+                },
+                "examples": {
+                  "rateLimitMinute": {
+                    "value": {
+                      "errorCode": 429,
+                      "message": "Number of allowed API requests per minute exceeded"
+                    }
+                  },
+                  "rateLimitDay": {
+                    "value": {
+                      "errorCode": 429,
+                      "message": "Number of allowed API requests per day exceeded"
+                    }
+                  }
+                }
+              }
+            }
+          },
+          "500": {
+            "description": "Internal Server Error - An unexpected error occurred on the server",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/Error"
+                },
+                "examples": {
+                  "serverError": {
+                    "value": {
+                      "errorCode": 500,
+                      "message": "An unexpected error occurred. Please try again later."
+                    }
+                  }
+                }
+              }
+            }
+          }
+        }
+      }
+    }
+  },
+  "components": {
+    "securitySchemes": {
+      "apiKey": {
+        "type": "apiKey",
+        "name": "X-Api-Key",
+        "in": "header"
+      },
+      "oauth2": {
+        "type": "oauth2",
+        "flows": {
+          "authorizationCode": {
+            "authorizationUrl": "https://api.servicem8.com/oauth/authorize",
+            "tokenUrl": "https://api.servicem8.com/oauth/access_token",
+            "scopes": {
+              "staff_locations": "Access to real-time GPS information about staff",
+              "staff_activity": "Access to clock on, lunch break and clock off information about staff",
+              "publish_sms": "Access to send SMS messages to customers and/or staff on your behalf. Note sending SMS messages will incur account charges.",
+              "publish_email": "Access to send Email messages to customers and/or staff on your behalf",
+              "vendor": "Access to basic account information",
+              "vendor_logo": "Access to account logo",
+              "vendor_email": "Access to account holder email address",
+              "read_locations": "Read-only access to Location Endpoint",
+              "manage_locations": "Full access to Location Endpoint",
+              "read_staff": "Read-only access to Staff Endpoint",
+              "manage_staff": "Full access to Staff Endpoint",
+              "read_customers": "Read-only access to Company Endpoint",
+              "manage_customers": "Full access to Company Endpoint",
+              "read_customer_contacts": "Read-only access to CompanyContact Endpoint",
+              "manage_customer_contacts": "Full access to CompanyContact Endpoint",
+              "read_jobs": "Read-only access to Job Endpoint",
+              "manage_jobs": "Full access to Job Endpoint",
+              "create_jobs": "Ability to create jobs on behalf of account. Note creating jobs may incur account charges.",
+              "read_job_contacts": "Read-only access to JobContact Endpoint",
+              "manage_job_contacts": "Full access to JobContact Endpoint",
+              "read_job_materials": "Read-only access to JobMaterials Endpoint",
+              "manage_job_materials": "Full access to JobMaterials Endpoint",
+              "read_job_categories": "Read-only access to Categories Endpoint",
+              "manage_job_categories": "Full access to Categories Endpoint",
+              "read_job_queues": "Read-only access to Job Queues Endpoint",
+              "manage_job_queues": "Full access to Job Queues Endpoint",
+              "read_tasks": "Read-only access to Tasks Endpoint",
+              "manage_tasks": "Full access to Tasks Endpoint",
+              "read_schedule": "Read-only access to JobActivity Endpoint",
+              "manage_schedule": "Full access to JobActivity Endpoint",
+              "read_inventory": "Read-only access to Materials Endpoint",
+              "manage_inventory": "Full access to Materials Endpoint",
+              "read_job_notes": "Read-only access to job notes",
+              "publish_job_notes": "Ability to add new job notes",
+              "read_job_photos": "Read-only access to job photos",
+              "publish_job_photos": "Ability to add new job photos",
+              "read_attachments": "Read-only access to Attachments Endpoint",
+              "manage_attachments": "Full access to Attachments Endpoint",
+              "read_inbox": "Read-only access to inbox messages",
+              "read_messages": "Read-only access to staff messages",
+              "manage_notifications": "Ability to read notifications and mark as read",
+              "manage_templates": "Full-access to email, sms and document templates",
+              "manage_badges": "Full-access to create/modify job badges",
+              "read_assets": "Read-only access to Assets Endpoint",
+              "manage_assets": "Full access to Assets Endpoint",
+              "read_knowledge_base": "Read-only access to Knowledge Base Endpoint",
+              "manage_knowledge_base": "Full access to Knowledge Base Endpoint"
+            }
+          }
+        }
+      }
+    },
+    "schemas": {
+      "Error": {
+        "type": "object",
+        "properties": {
+          "errorCode": {
+            "type": "number",
+            "format": "int32",
+            "example": "1000"
+          },
+          "message": {
+            "type": "string",
+            "example": "An error occurred completing your request"
+          }
+        }
+      },
+      "RateLimitError": {
+        "type": "object",
+        "properties": {
+          "errorCode": {
+            "type": "number",
+            "format": "int32",
+            "example": "429"
+          },
+          "message": {
+            "type": "string",
+            "example": "Number of allowed API requests per minute exceeded"
+          }
+        }
+      },
+      "AuthenticationError": {
+        "type": "object",
+        "properties": {
+          "errorCode": {
+            "type": "number",
+            "format": "int32",
+            "example": "401"
+          },
+          "message": {
+            "type": "string",
+            "example": "Authentication failed. Please check your API key or OAuth token."
+          }
+        }
+      },
+      "ForbiddenError": {
+        "type": "object",
+        "properties": {
+          "errorCode": {
+            "type": "number",
+            "format": "int32",
+            "example": "403"
+          },
+          "message": {
+            "type": "string",
+            "example": "Access forbidden. You don't have permission to access this resource."
+          }
+        }
+      },
+      "FormResponse": {
+        "type": "object",
+        "properties": {
+          "form_uuid": {
+            "description": "UUID of the form used to generate this form response. Links to a specific form in the system that defines the fields to be gathered.",
+            "format": "uuid",
+            "example": "123e4567-8f59-4273-9bb2-23f94e3071eb",
+            "type": "string"
+          },
+          "staff_uuid": {
+            "description": "UUID of the staff member who completed this FormResponse.",
+            "format": "uuid",
+            "example": "123e4567-068f-433c-a2e6-23f94e815bdb",
+            "type": "string"
+          },
+          "regarding_object": {
+            "description": "The object type that this form response is associated with. Common values include 'job', 'asset', or 'company'. Works in conjunction with regarding_object_uuid to link this form response to a specific record in the system.",
+            "type": "string"
+          },
+          "regarding_object_uuid": {
+            "description": "UUID of the specific record this form response is linked to. For example, if regarding_object is 'job', this will be the UUID of the specific job. This creates a relationship between the form response and the object it refers to.",
+            "format": "uuid",
+            "example": "123e4567-41ee-4ea3-8ca2-23f94fe918eb",
+            "type": "string"
+          },
+          "field_data": {
+            "description": "JSON array of form answers captured at submission time.",
+            "type": "string"
+          },
+          "timestamp": {
+            "description": "Date and time when the form was submitted/completed. Used for sorting and displaying form responses chronologically. Format is YYYY-MM-DD HH:MM:SS in UTC timezone.",
+            "type": "string",
+            "example": "2026-03-01 12:00:00"
+          },
+          "form_by_staff_uuid": {
+            "description": "UUID of the staff member who completed or submitted this form. Identifies which user filled out the form. Used for tracking form submission history and staff accountability.",
+            "format": "uuid",
+            "example": "123e4567-0230-4fe1-83c9-23f94429362b",
+            "type": "string"
+          },
+          "document_attachment_uuid": {
+            "description": "UUID of the document attachment generated from this form response. When a form is completed, it can generate a PDF document which is stored as an attachment. This field links to that generated document attachment.",
+            "format": "uuid",
+            "example": "123e4567-ff79-499a-8ef4-23f94a981a6b",
+            "type": "string"
+          },
+          "asset_uuid": {
+            "description": "UUID of the Asset this form response is related to. Used when the FormResponsepertains to a specific asset, such as equipment inspections, maintenance checklists, or asset condition reports.",
+            "format": "uuid",
+            "example": "123e4567-fc2c-4a3d-a278-23f942d7f62b",
+            "type": "string"
+          },
+          "uuid": {
+            "format": "uuid",
+            "description": "Unique identifier for this record",
+            "example": "123e4567-faa3-4649-8ea5-23f94011885b",
+            "type": "string"
+          },
+          "active": {
+            "enum": [
+              0,
+              1
+            ],
+            "type": "integer",
+            "default": 1,
+            "description": "Record active/deleted flag.  Valid values are [0,1]"
+          },
+          "edit_date": {
+            "example": "2026-03-01 12:00:00",
+            "readOnly": true,
+            "description": "Timestamp at which record was last modified"
+          }
+        }
+      }
+    }
+  },
+  "x-speakeasy-retries": {
+    "strategy": "backoff",
+    "backoff": {
+      "initialInterval": 500,
+      "maxInterval": 60000,
+      "maxElapsedTime": 3600000,
+      "exponent": 1.5
+    },
+    "statusCodes": [
+      "5XX",
+      "429"
+    ],
+    "retryConnectionErrors": true
+  },
+  "tags": [
+    {
+      "name": "Form Responses",
+      "description": "Operations related to Form Responses"
+    }
+  ]
+}
+```

poll_and_apply_quote_templates.sh

@@ -0,0 +1,138 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Poll ServiceM8 form responses, parse Quote Template responses, then apply any
+# newly/unapplied parsed quote responses to ServiceM8 jobMaterials.
+#
+# Default window: last 24 hours from script start, in local system time.
+# Override with:
+#   --since 'YYYY-MM-DD HH:MM:SS'
+#   --hours 48
+#
+# This wrapper intentionally skips dry-run and calls --apply. The apply script
+# still refuses duplicate applies unless --force is explicitly passed through.
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+POLL_SCRIPT="$SCRIPT_DIR/poll_form_responses_since.py"
+APPLY_SCRIPT="$SCRIPT_DIR/apply_polled_quote_template_jobmaterials.py"
+DB_PATH="${WEBHOOK_POLL_DB_PATH:-$SCRIPT_DIR/servicem8_formresponse_poll.db}"
+LOG_DIR="${WEBHOOK_RUN_LOG_DIR:-$SCRIPT_DIR/logs}"
+QUOTE_TEMPLATE_FORM_UUID="${SERVICEM8_QUOTE_TEMPLATE_FORM_UUID:-3621b6be-1d19-4756-9ab4-9d5e4120f6d9}"
+
+SINCE=""
+HOURS="24"
+FORCE="0"
+
+usage() {
+  cat <<EOF
+Usage: $0 [--since 'YYYY-MM-DD HH:MM:SS'] [--hours N] [--force]
+
+Examples:
+  $0
+  $0 --hours 48
+  $0 --since '2026-05-04 08:00:00'
+
+This will:
+  1. Poll /formresponse.json using timestamp gt SINCE
+  2. Store/parse Quote Template responses into $DB_PATH
+  3. Apply parsed responses that do not already have generated materials recorded
+EOF
+}
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --since)
+      SINCE="${2:-}"
+      shift 2
+      ;;
+    --hours)
+      HOURS="${2:-}"
+      shift 2
+      ;;
+    --force)
+      FORCE="1"
+      shift
+      ;;
+    -h|--help)
+      usage
+      exit 0
+      ;;
+    *)
+      echo "Unknown argument: $1" >&2
+      usage >&2
+      exit 2
+      ;;
+  esac
+done
+
+if [[ -z "$SINCE" ]]; then
+  SINCE="$(date -d "-${HOURS} hours" '+%Y-%m-%d %H:%M:%S')"
+fi
+
+mkdir -p "$LOG_DIR"
+RUN_ID="$(date '+%Y%m%d-%H%M%S')"
+LOG_FILE="$LOG_DIR/poll-and-apply-$RUN_ID.log"
+
+exec > >(tee -a "$LOG_FILE") 2>&1
+
+echo "== ServiceM8 Quote Template poll/apply run =="
+echo "Started: $(date --iso-8601=seconds)"
+echo "Since:   $SINCE"
+echo "DB:      $DB_PATH"
+echo "Log:     $LOG_FILE"
+echo
+
+echo "== Polling form responses =="
+"$POLL_SCRIPT" --since "$SINCE" --summary-limit 20
+
+echo
+echo "== Finding unapplied parsed Quote Template responses =="
+mapfile -t FORM_RESPONSE_UUIDS < <(
+  python3 - "$DB_PATH" "$QUOTE_TEMPLATE_FORM_UUID" <<'PY'
+import sqlite3
+import sys
+
+poll_db, quote_form_uuid = sys.argv[1], sys.argv[2]
+conn = sqlite3.connect(poll_db)
+conn.row_factory = sqlite3.Row
+rows = conn.execute(
+    """
+    select q.form_response_uuid
+    from quote_template_form_responses q
+    left join form_responses_raw r on r.uuid = q.form_response_uuid
+    where q.form_uuid = ?
+      and coalesce(q.process_status, '') != 'applied'
+    order by coalesce(r.timestamp, q.discovered_at) asc, q.form_response_uuid asc
+    """,
+    (quote_form_uuid,),
+).fetchall()
+for row in rows:
+    print(row["form_response_uuid"])
+PY
+)
+
+if [[ ${#FORM_RESPONSE_UUIDS[@]} -eq 0 ]]; then
+  echo "No unapplied Quote Template responses found."
+  echo "Finished: $(date --iso-8601=seconds)"
+  exit 0
+fi
+
+printf 'Found %d unapplied Quote Template response(s):\n' "${#FORM_RESPONSE_UUIDS[@]}"
+printf ' - %s\n' "${FORM_RESPONSE_UUIDS[@]}"
+
+echo
+echo "== Applying to ServiceM8 =="
+APPLY_ARGS=(--apply --pretty)
+if [[ "$FORCE" == "1" ]]; then
+  APPLY_ARGS+=(--force)
+fi
+
+for uuid in "${FORM_RESPONSE_UUIDS[@]}"; do
+  echo
+  echo "-- Applying form_response_uuid=$uuid --"
+  "$APPLY_SCRIPT" --uuid "$uuid" "${APPLY_ARGS[@]}"
+done
+
+echo
+echo "Finished: $(date --iso-8601=seconds)"
+echo "Log: $LOG_FILE"