End of day updates, almost at completion of live cron setup with bash script wrapper now as ell

2026-05-04 18:22:39 +10:00
parent 2739606d6a
commit 868632bd21
3 changed files with 295 additions and 61 deletions
@@ -1,73 +1,158 @@
 # ServiceM8 Project Progress

-## Current Quote Template → JobMaterials Pipeline
+## Current Direction — Soft Release Poller + Apply Pipeline

-### Live receiver
- `servicem8_webhook_receiver.py`
-  - receives `form.response_created`
-  - stores the raw webhook payload in `servicem8_webhooks.db`
-  - checks incoming form responses for the Quote Template `form_uuid`
-  - if matched, parses and queues the derived jobMaterials payload to:
-    - `quote-template-jobmaterials-queue.jsonl`
-  - does **not** perform live ServiceM8 writes at this stage
+We changed tune from relying mainly on ServiceM8 `form.response_created` webhooks to using a scheduled API poller as the operational safety-net/primary soft-release path.

-### Parser
- `servicem8-quote-template-parser.py`
-  - parses Quote Template `field_data`
-  - extracts:
-    - description of works
-    - `Item 1..12` include lines
-    - `Works excluded 1..4` exclude lines
-    - extra descriptive include rows such as labour/scaffolding/equipment fields
-  - builds normalized `desired_job_materials`
+Reason: ServiceM8 form webhooks are useful, but during testing some expected form completions did not reliably appear at the FastAPI listener. The ServiceM8 API endpoint `/api_1.0/formresponse.json` can be queried with a timestamp filter, so the safer operational pattern is now:

-### Create/apply script
- `servicem8-create-jobmaterials-from-form-response.py`
-  - standalone script
-  - consumes a Quote Template form response JSON payload
-  - builds ServiceM8 Job Material API payloads
-  - runs in **dry-run by default**
-  - supports live create later with `--apply`
-  - records created/generated mappings into local state DB
+1. poll recent form responses with `timestamp gt 'YYYY-MM-DD HH:MM:SS'`
+2. store every returned API row locally
+3. detect Quote Template form responses
+4. parse them into desired jobMaterial rows
+5. apply only previously unapplied Quote Template responses to ServiceM8
+6. track created jobMaterial UUIDs locally to avoid duplicate applies

-### Local state tracking
- `servicem8_quote_materials_state.db`
-  - local SQLite DB for tracking generated jobMaterials
-  - intended fields include:
-    - job UUID
-    - form response UUID
-    - created job material UUID
-    - kind/source metadata
+This keeps the live webhook receiver useful as capture/diagnostics, but no longer depends on webhook delivery for completeness.

-### Queue/prepared output
- `quote-template-jobmaterials-queue.jsonl`
-  - lightweight queue/output file written by webhook stage
-  - contains parsed/prepared `desired_job_materials` objects
-  - no live update performed yet
+## Active Soft-Release Flow

-### Inspector
- `servicem8_inspector.py`
-  - read-only browser for webhook DB
-  - now also includes visibility of generated-materials state DB
+### 1. Poll ServiceM8 form responses
+- Script: `poll_form_responses_since.py`
+- API endpoint: `/api_1.0/formresponse.json`
+- Confirmed filter syntax:
+  - `?$filter=timestamp gt '2026-05-04 10:00:00'`
+- Default operational DB:
+  - `servicem8_formresponse_poll.db`
+- Poll queue/output:
+  - `quote-template-jobmaterials-poll-queue.jsonl`
+- Quote Template form UUID:
+  - `3621b6be-1d19-4756-9ab4-9d5e4120f6d9`
+
+The poller stores all fetched form responses in `form_responses_raw`, then parses Quote Template matches into `quote_template_form_responses`.
+
+### 2. Parse Quote Template responses
+- Parser: `servicem8_quote_template_parser.py`
+- Extracts:
+  - description of works
+  - `Item 1..12` include lines
+  - `Works excluded 1..4` exclude lines
+  - extra descriptive include rows such as labour/materials/scaffolding/equipment fields
+- Builds normalized `desired_job_materials` rows.
+
+### 3. Select/apply parsed Quote Template responses
+- Script: `apply_polled_quote_template_jobmaterials.py`
+- Safe behaviour built in:
+  - processes one `form_response_uuid` at a time when called directly
+  - dry-run by default
+  - `--apply` performs live ServiceM8 `jobmaterial.json` creates
+  - refuses duplicate apply when generated material rows already exist for that form response unless `--force` is used
+- Apply tracking tables in `servicem8_formresponse_poll.db`:
+  - `quote_template_apply_runs`
+  - `quote_template_apply_run_rows`
+- Created ServiceM8 job material mappings are recorded in:
+  - `servicem8_quote_materials_state.db`
+
+Current apply payload rules:
+- All rows get:
+  - `tax_rate_uuid = 84e4dd28-06b3-452b-a796-1f58a20ac49b`
+  - `quantity = "0"`
+  - `price = ""`
+  - `displayed_amount = ""`
+  - `displayed_amount_is_tax_inclusive = ""`
+- Header material UUIDs:
+  - `include_header` → `1924893b-917f-474a-adaa-2093bd622d4b`
+  - `exclude_header` → `4947bfd7-4875-48f7-9caf-2093b9751b9b`
+- Non-header quote rows currently use:
+  - `f78b1d23-b9fa-40fe-a806-2425fe09cc0b`
+
+### 4. Wrapper for scheduled/operational use
+- Script: `poll_and_apply_quote_templates.sh`
+- Default behaviour:
+  - polls the last 24 hours from script start
+  - stores/parses results
+  - applies any parsed Quote Template responses that are not already marked/applied
+  - logs each run under `logs/poll-and-apply-YYYYMMDD-HHMMSS.log`
+- Examples:
+  - `./poll_and_apply_quote_templates.sh`
+  - `./poll_and_apply_quote_templates.sh --hours 48`
+  - `./poll_and_apply_quote_templates.sh --since '2026-05-04 08:00:00'`
+
+This is the proposed scheduled entry point for soft release, e.g. every 10–30 minutes.
+
+## Live Webhook Receiver Status
+
+### Receiver
+- Script: `servicem8_webhook_receiver.py`
+- Still receives/stores:
+  - event webhooks
+  - object webhooks
+  - form-response webhooks
+- DB:
+  - `servicem8_webhooks.db`
+- Quote Template webhook queue:
+  - `quote-template-jobmaterials-queue.jsonl`
+
+Important: webhook handling still does **not** perform live ServiceM8 writes. Heavy work stays outside the FastAPI webhook request path.
+
+## Inspector / Web Viewer
+
+- Script: `servicem8_inspector.py`
+- Current intended views include:
+  - webhook events
+  - webhook objects
+  - webhook form responses
+  - polled form responses
+  - parsed polled Quote Template responses
+  - poll runs
+  - dry-run/apply runs
+  - generated material state
+
+Note: the code was updated and tested on a temporary localhost port, but the existing live inspector process may need a manual/service restart before all new pages appear in the running viewer.
+
+## Required Files for Current Soft Release
+
+### Scripts
+- `poll_and_apply_quote_templates.sh` — scheduled wrapper / main operational entry point
+- `poll_form_responses_since.py` — polls ServiceM8 and populates poll DB
+- `apply_polled_quote_template_jobmaterials.py` — applies parsed responses to ServiceM8
+- `servicem8_quote_template_parser.py` — parsing logic
+- `servicem8_inspector.py` — web inspection/progress viewer
+- `servicem8_webhook_receiver.py` — still useful for webhook capture/diagnostics
+
+### Databases / state
+- `servicem8_formresponse_poll.db` — poll results, parsed quote responses, apply run status
+- `servicem8_quote_materials_state.db` — created jobMaterial mapping/state to avoid duplicates
+- `servicem8_webhooks.db` — webhook capture/archive/diagnostics
+
+### Queue/log files
+- `quote-template-jobmaterials-poll-queue.jsonl` — poll-derived parsed queue/output
+- `logs/poll-and-apply-*.log` — wrapper run logs
+- `quote-template-jobmaterials-queue.jsonl` — older webhook-derived queue; still useful but not the primary soft-release path

 ## Current Status
-Everything is staged and connected up to the point of:
- webhook receive
- UUID trigger check
- form parsing
- local queueing
- dry-run jobMaterial payload generation
- local state DB support
- inspector visibility

-## Not Yet Enabled
- actual live POST creation of Job Materials into ServiceM8 during webhook processing
- any automatic update/delete reconciliation against live ServiceM8 records
+Operational soft-release pieces are now in place:
+- API polling confirmed
+- Quote Template detection confirmed
+- parsing confirmed
+- dry-run/apply command path tested
+- payload adjusted for current ServiceM8 requirements
+- duplicate-apply guard in place
+- wrapper script created for scheduled operation
+- inspector updated for progress visibility
+
+## Not Yet Done / Next Steps
+
+- Restart/reload the live inspector process so the new poll/apply pages are available in the active web viewer.
+- Decide schedule interval for `poll_and_apply_quote_templates.sh` — likely every 10 or 30 minutes.
+- Run the wrapper manually for a soft-release smoke test with a controlled recent form response.
+- After confidence builds, wire the wrapper into cron/system scheduling.
+- Future hardening: add reconciliation/update/delete behaviour if ServiceM8 quote form responses are edited after initial apply.

 ## Design Notes
- Heavy lifting is intentionally kept **out of the live webhook handler**.
- The webhook handler is used only for:
-  - capture
-  - UUID gate
-  - parse/prepare/queue
- Live ServiceM8 mutation remains a separate step/script for safety.
+
+- Webhooks remain lightweight and non-mutating.
+- Polling is now the reliable source of completeness.
+- Applying to ServiceM8 is tracked locally and guarded against duplicates.
+- The wrapper intentionally skips dry-run for soft release, but the underlying apply script still supports dry-run and duplicate protection.