Closed look on webhook calling ofrm update scripts - no live updated to sevicem8 yet though. Create project-progress as well.

PROJECT-PROGRESS.md

@@ -0,0 +1,73 @@
+# ServiceM8 Project Progress
+
+## Current Quote Template → JobMaterials Pipeline
+
+### Live receiver
+- `servicem8_webhook_receiver.py`
+  - receives `form.response_created`
+  - stores the raw webhook payload in `servicem8_webhooks.db`
+  - checks incoming form responses for the Quote Template `form_uuid`
+  - if matched, parses and queues the derived jobMaterials payload to:
+    - `quote-template-jobmaterials-queue.jsonl`
+  - does **not** perform live ServiceM8 writes at this stage
+
+### Parser
+- `servicem8-quote-template-parser.py`
+  - parses Quote Template `field_data`
+  - extracts:
+    - description of works
+    - `Item 1..12` include lines
+    - `Works excluded 1..4` exclude lines
+    - extra descriptive include rows such as labour/scaffolding/equipment fields
+  - builds normalized `desired_job_materials`
+
+### Create/apply script
+- `servicem8-create-jobmaterials-from-form-response.py`
+  - standalone script
+  - consumes a Quote Template form response JSON payload
+  - builds ServiceM8 Job Material API payloads
+  - runs in **dry-run by default**
+  - supports live create later with `--apply`
+  - records created/generated mappings into local state DB
+
+### Local state tracking
+- `servicem8_quote_materials_state.db`
+  - local SQLite DB for tracking generated jobMaterials
+  - intended fields include:
+    - job UUID
+    - form response UUID
+    - created job material UUID
+    - kind/source metadata
+
+### Queue/prepared output
+- `quote-template-jobmaterials-queue.jsonl`
+  - lightweight queue/output file written by webhook stage
+  - contains parsed/prepared `desired_job_materials` objects
+  - no live update performed yet
+
+### Inspector
+- `servicem8_inspector.py`
+  - read-only browser for webhook DB
+  - now also includes visibility of generated-materials state DB
+
+## Current Status
+Everything is staged and connected up to the point of:
+- webhook receive
+- UUID trigger check
+- form parsing
+- local queueing
+- dry-run jobMaterial payload generation
+- local state DB support
+- inspector visibility
+
+## Not Yet Enabled
+- actual live POST creation of Job Materials into ServiceM8 during webhook processing
+- any automatic update/delete reconciliation against live ServiceM8 records
+
+## Design Notes
+- Heavy lifting is intentionally kept **out of the live webhook handler**.
+- The webhook handler is used only for:
+  - capture
+  - UUID gate
+  - parse/prepare/queue
+- Live ServiceM8 mutation remains a separate step/script for safety.

servicem8_webhook_receiver.py

@@ -7,6 +7,12 @@ from datetime import datetime, timezone
 
 from fastapi import FastAPI, Request
 from fastapi.responses import PlainTextResponse
+from servicem8_quote_template_parser import (
+    QUOTE_TEMPLATE_FORM_UUID,
+    STATE_DB_PATH,
+    init_state_db as init_quote_state_db,
+    parse_quote_template_form_response,
+)
 
 DB_PATH = os.getenv("WEBHOOK_DB_PATH", "./servicem8_webhooks.db")
 APP_HOST = os.getenv("WEBHOOK_HOST", "0.0.0.0")
@@ -75,6 +81,8 @@ def init_db():
 
         conn.commit()
 
+    init_quote_state_db(STATE_DB_PATH)
+
 
 @app.on_event("startup")
 async def startup_event():
@@ -145,6 +153,37 @@ def store_simple_event(table_name, request: Request, client_host: str, received_
         conn.commit()
 
 
+def queue_quote_template_jobmaterials(payload, received_at: str):
+    try:
+        parsed = parse_quote_template_form_response(payload)
+    except Exception as exc:
+        logger.warning("Quote template parser failed: %s", exc)
+        return
+
+    form_uuid = parsed.get("form_uuid", "")
+    if form_uuid != QUOTE_TEMPLATE_FORM_UUID:
+        return
+
+    queue_path = os.path.join(os.path.dirname(DB_PATH), "quote-template-jobmaterials-queue.jsonl")
+    queue_record = {
+        "queued_at": received_at,
+        "form_uuid": form_uuid,
+        "form_response_uuid": parsed.get("form_response_uuid", ""),
+        "job_uuid": parsed.get("job_uuid", ""),
+        "description": parsed.get("description", ""),
+        "desired_job_materials": parsed.get("desired_job_materials", []),
+    }
+    with open(queue_path, "a", encoding="utf-8") as fh:
+        fh.write(json.dumps(queue_record, ensure_ascii=False) + "\n")
+
+    logger.info(
+        "Queued quote-template jobmaterials: form_response_uuid=%s job_uuid=%s rows=%s",
+        queue_record["form_response_uuid"],
+        queue_record["job_uuid"],
+        len(queue_record["desired_job_materials"]),
+    )
+
+
 @app.post("/webhooks/servicem8-job-updated")
 async def servicem8_event_webhook(request: Request):
     payload = await parse_request_payload(request)
@@ -174,6 +213,7 @@ async def servicem8_form_response_webhook(request: Request):
         return challenge_response
 
     store_simple_event("webhook_form_responses", request, client_host, received_at, headers, payload)
+    queue_quote_template_jobmaterials(payload, received_at)
 
     logger.info("Form response webhook received from %s and stored", client_host)
     return PlainTextResponse("OK", status_code=200)