189 lines
9.0 KiB
Markdown
189 lines
9.0 KiB
Markdown
# ServiceM8 Project Progress
|
||
|
||
## Current Direction — Soft Release Poller + Apply Pipeline
|
||
|
||
We changed tune from relying mainly on ServiceM8 `form.response_created` webhooks to using a scheduled API poller as the operational safety-net/primary soft-release path.
|
||
|
||
Reason: ServiceM8 form webhooks are useful, but during testing some expected form completions did not reliably appear at the FastAPI listener. The ServiceM8 API endpoint `/api_1.0/formresponse.json` can be queried with a timestamp filter, so the safer operational pattern is now:
|
||
|
||
1. poll recent form responses with `timestamp gt 'YYYY-MM-DD HH:MM:SS'`
|
||
2. store every returned API row locally
|
||
3. detect Quote Template form responses
|
||
4. parse them into desired jobMaterial rows
|
||
5. apply only previously unapplied Quote Template responses to ServiceM8
|
||
6. track created jobMaterial UUIDs locally to avoid duplicate applies
|
||
|
||
This keeps the live webhook receiver useful as capture/diagnostics, but no longer depends on webhook delivery for completeness.
|
||
|
||
## Active Soft-Release Flow
|
||
|
||
### 1. Poll ServiceM8 form responses
|
||
- Script: `poll_form_responses_since.py`
|
||
- API endpoint: `/api_1.0/formresponse.json`
|
||
- Confirmed filter syntax:
|
||
- `?$filter=timestamp gt '2026-05-04 10:00:00'`
|
||
- Default operational DB:
|
||
- `servicem8_formresponse_poll.db`
|
||
- Poll queue/output:
|
||
- `quote-template-jobmaterials-poll-queue.jsonl`
|
||
- Quote Template form UUID:
|
||
- `3621b6be-1d19-4756-9ab4-9d5e4120f6d9`
|
||
|
||
The poller stores all fetched form responses in `form_responses_raw`, then parses Quote Template matches into `quote_template_form_responses`.
|
||
|
||
### 2. Parse Quote Template responses
|
||
- Parser: `servicem8_quote_template_parser.py`
|
||
- Extracts:
|
||
- description of works
|
||
- `Item 1..12` include lines
|
||
- `Works excluded 1..4` exclude lines
|
||
- extra descriptive include rows such as labour/materials/scaffolding/equipment fields
|
||
- Builds normalized `desired_job_materials` rows.
|
||
|
||
### 3. Select/apply parsed Quote Template responses
|
||
- Script: `apply_polled_quote_template_jobmaterials.py`
|
||
- Safe behaviour built in:
|
||
- processes one `form_response_uuid` at a time when called directly
|
||
- dry-run by default
|
||
- `--apply` performs live ServiceM8 `jobmaterial.json` creates
|
||
- refuses duplicate apply when generated material rows already exist for that form response unless `--force` is used
|
||
- before any live create, checks remote ServiceM8 `/jobmaterial.json` for existing rows on the target `job_uuid`
|
||
- if remote rows already exist, records a remote-existing incident and creates nothing unless `--force-remote-existing` is explicitly used
|
||
- Apply/incident tracking tables in `servicem8_formresponse_poll.db`:
|
||
- `quote_template_apply_runs`
|
||
- `quote_template_apply_run_rows`
|
||
- `quote_template_remote_existing_incidents`
|
||
- Created ServiceM8 job material mappings are recorded in:
|
||
- `servicem8_quote_materials_state.db`
|
||
|
||
Current apply payload rules:
|
||
- All rows get:
|
||
- `tax_rate_uuid = 84e4dd28-06b3-452b-a796-1f58a20ac49b`
|
||
- `quantity = "0"`
|
||
- `price = ""`
|
||
- `displayed_amount = ""`
|
||
- `displayed_amount_is_tax_inclusive = ""`
|
||
- Header material UUIDs:
|
||
- `include_header` → `1924893b-917f-474a-adaa-2093bd622d4b`
|
||
- `exclude_header` → `4947bfd7-4875-48f7-9caf-2093b9751b9b`
|
||
- Non-header quote rows currently use:
|
||
- `f78b1d23-b9fa-40fe-a806-2425fe09cc0b`
|
||
|
||
### 4. Wrapper for scheduled/operational use
|
||
- Script: `poll_and_apply_quote_templates.sh`
|
||
- Default behaviour:
|
||
- polls the last 24 hours from script start
|
||
- stores/parses results
|
||
- applies any parsed Quote Template responses that are not already marked/applied
|
||
- logs each run under `logs/poll-and-apply-YYYYMMDD-HHMMSS.log`
|
||
- Safety options now available:
|
||
- `--dry-run` runs the same poll/selection flow, but previews the ServiceM8 `jobmaterial` payloads only
|
||
- dry-run does **not** write to ServiceM8
|
||
- dry-run does **not** mark quote responses as applied
|
||
- live apply blocks if remote ServiceM8 already has jobMaterial rows for the job, logging the captured rows/counts to `quote_template_remote_existing_incidents`
|
||
- `--recheck-remote-existing` revisits previously blocked rows without overriding the safety gate
|
||
- `--force-remote-existing` explicitly overrides the remote-existing safety gate and still records a forced incident before creating
|
||
- Examples:
|
||
- `./poll_and_apply_quote_templates.sh`
|
||
- `./poll_and_apply_quote_templates.sh --hours 48`
|
||
- `./poll_and_apply_quote_templates.sh --since '2026-05-04 08:00:00'`
|
||
- `./poll_and_apply_quote_templates.sh --dry-run --hours 48`
|
||
- `./poll_and_apply_quote_templates.sh --recheck-remote-existing --hours 48`
|
||
- `./poll_and_apply_quote_templates.sh --force-remote-existing --hours 48`
|
||
|
||
This is the proposed scheduled entry point for soft release, e.g. every 10–30 minutes. For manual confidence checks, run it with `--dry-run` first, inspect the generated payloads/log, then rerun without `--dry-run` only when ready to apply.
|
||
|
||
## Live Webhook Receiver Status
|
||
|
||
### Receiver
|
||
- Script: `servicem8_webhook_receiver.py`
|
||
- Still receives/stores:
|
||
- event webhooks
|
||
- object webhooks
|
||
- form-response webhooks
|
||
- DB:
|
||
- `servicem8_webhooks.db`
|
||
- Quote Template webhook queue:
|
||
- `quote-template-jobmaterials-queue.jsonl`
|
||
|
||
Important: webhook handling still does **not** perform live ServiceM8 writes. Heavy work stays outside the FastAPI webhook request path.
|
||
|
||
## Inspector / Web Viewer
|
||
|
||
- Script: `servicem8_inspector.py`
|
||
- Current intended views include:
|
||
- webhook events
|
||
- webhook objects
|
||
- webhook form responses
|
||
- polled form responses
|
||
- parsed polled Quote Template responses
|
||
- poll runs
|
||
- dry-run/apply runs
|
||
- generated material state
|
||
|
||
Note: the code was updated and tested on a temporary localhost port, but the existing live inspector process may need a manual/service restart before all new pages appear in the running viewer.
|
||
|
||
## Required Files for Current Soft Release
|
||
|
||
### Scripts
|
||
- `poll_and_apply_quote_templates.sh` — scheduled wrapper / main operational entry point
|
||
- `poll_form_responses_since.py` — polls ServiceM8 and populates poll DB
|
||
- `apply_polled_quote_template_jobmaterials.py` — applies parsed responses to ServiceM8
|
||
- `servicem8_quote_template_parser.py` — parsing logic
|
||
- `servicem8_inspector.py` — web inspection/progress viewer
|
||
- `servicem8_webhook_receiver.py` — still useful for webhook capture/diagnostics
|
||
|
||
### Databases / state
|
||
- `servicem8_formresponse_poll.db` — poll results, parsed quote responses, apply run status
|
||
- `servicem8_quote_materials_state.db` — created jobMaterial mapping/state to avoid duplicates
|
||
- `servicem8_webhooks.db` — webhook capture/archive/diagnostics
|
||
|
||
### Queue/log files
|
||
- `quote-template-jobmaterials-poll-queue.jsonl` — poll-derived parsed queue/output
|
||
- `logs/poll-and-apply-*.log` — wrapper run logs
|
||
- `quote-template-jobmaterials-queue.jsonl` — older webhook-derived queue; still useful but not the primary soft-release path
|
||
|
||
## Current Status
|
||
|
||
Operational soft-release pieces are now in place:
|
||
- API polling confirmed
|
||
- Quote Template detection confirmed
|
||
- parsing confirmed
|
||
- dry-run/apply command path tested
|
||
- payload adjusted for current ServiceM8 requirements
|
||
- duplicate-apply guard in place
|
||
- wrapper script created for scheduled operation
|
||
- wrapper now has a first-class `--dry-run` mode
|
||
- inspector updated for progress visibility
|
||
|
||
### Checkpoint — 2026-05-05
|
||
|
||
Latest verified state:
|
||
- `poll_and_apply_quote_templates.sh --dry-run` added and documented.
|
||
- `bash -n /opt/webhooks/poll_and_apply_quote_templates.sh` passes.
|
||
- `--help` output includes `--dry-run` usage/examples.
|
||
- A future-since dry-run (`--dry-run --since '2099-01-01 00:00:00'`) confirmed:
|
||
- wrapper reports `Mode: dry-run`
|
||
- poll step performs no ServiceM8 writes
|
||
- apply step calls the Python apply tool without `--apply`
|
||
- payload rows are emitted as `would_create`
|
||
- responses are not marked applied by dry-run
|
||
- At checkpoint time, there were still several unapplied parsed Quote Template responses available for preview/apply; this is expected while the soft release remains manual.
|
||
|
||
## Not Yet Done / Next Steps
|
||
|
||
- Restart/reload the live inspector process so the new poll/apply pages are available in the active web viewer.
|
||
- Decide schedule interval for `poll_and_apply_quote_templates.sh` — likely every 10 or 30 minutes.
|
||
- Run the wrapper manually in `--dry-run` mode against a controlled recent form response and inspect the payload/log.
|
||
- If payload is correct, rerun the wrapper without `--dry-run` for a controlled live apply smoke test.
|
||
- After confidence builds, wire the wrapper into cron/system scheduling.
|
||
- Future hardening: add reconciliation/update/delete behaviour if ServiceM8 quote form responses are edited after initial apply.
|
||
|
||
## Design Notes
|
||
|
||
- Webhooks remain lightweight and non-mutating.
|
||
- Polling is now the reliable source of completeness.
|
||
- Applying to ServiceM8 is tracked locally and guarded against duplicates.
|
||
- The wrapper defaults to live apply for scheduled soft release, but now supports `--dry-run` for manual preview/safety checks.
|
||
- The underlying apply script remains dry-run-by-default and provides the duplicate protection used by the wrapper.
|