plumbing/PROJECT-PROGRESS.md

ServiceM8 Project Progress

Current Direction — Soft Release Poller + Apply Pipeline

We changed tune from relying mainly on ServiceM8 form.response_created webhooks to using a scheduled API poller as the operational safety-net/primary soft-release path.

Reason: ServiceM8 form webhooks are useful, but during testing some expected form completions did not reliably appear at the FastAPI listener. The ServiceM8 API endpoint /api_1.0/formresponse.json can be queried with a timestamp filter, so the safer operational pattern is now:

1. poll recent form responses with timestamp gt 'YYYY-MM-DD HH:MM:SS'
2. store every returned API row locally
3. detect Quote Template form responses
4. parse them into desired jobMaterial rows
5. apply only previously unapplied Quote Template responses to ServiceM8
6. track created jobMaterial UUIDs locally to avoid duplicate applies

This keeps the live webhook receiver useful as capture/diagnostics, but no longer depends on webhook delivery for completeness.

Active Soft-Release Flow

1. Poll ServiceM8 form responses

• Script: poll_form_responses_since.py
• API endpoint: /api_1.0/formresponse.json
• Confirmed filter syntax:
  • ?$filter=timestamp gt '2026-05-04 10:00:00'
• Default operational DB:
  • servicem8_formresponse_poll.db
• Poll queue/output:
  • quote-template-jobmaterials-poll-queue.jsonl
• Quote Template form UUID:
  • 3621b6be-1d19-4756-9ab4-9d5e4120f6d9

The poller stores all fetched form responses in form_responses_raw, then parses Quote Template matches into quote_template_form_responses.

2. Parse Quote Template responses

• Parser: servicem8_quote_template_parser.py
• Extracts:
  • description of works
  • Item 1..12 include lines
  • Works excluded 1..4 exclude lines
  • extra descriptive include rows such as labour/materials/scaffolding/equipment fields
• Builds normalized desired_job_materials rows.

3. Select/apply parsed Quote Template responses

• Script: apply_polled_quote_template_jobmaterials.py
• Safe behaviour built in:
  • processes one form_response_uuid at a time when called directly
  • dry-run by default
  • --apply performs live ServiceM8 jobmaterial.json creates
  • refuses duplicate apply when generated material rows already exist for that form response unless --force is used
• Apply tracking tables in servicem8_formresponse_poll.db:
  • quote_template_apply_runs
  • quote_template_apply_run_rows
• Created ServiceM8 job material mappings are recorded in:
  • servicem8_quote_materials_state.db

Current apply payload rules:

• All rows get:
  • tax_rate_uuid = 84e4dd28-06b3-452b-a796-1f58a20ac49b
  • quantity = "0"
  • price = ""
  • displayed_amount = ""
  • displayed_amount_is_tax_inclusive = ""
• Header material UUIDs:
  • include_header → 1924893b-917f-474a-adaa-2093bd622d4b
  • exclude_header → 4947bfd7-4875-48f7-9caf-2093b9751b9b
• Non-header quote rows currently use:
  • f78b1d23-b9fa-40fe-a806-2425fe09cc0b

4. Wrapper for scheduled/operational use

• Script: poll_and_apply_quote_templates.sh
• Default behaviour:
  • polls the last 24 hours from script start
  • stores/parses results
  • applies any parsed Quote Template responses that are not already marked/applied
  • logs each run under logs/poll-and-apply-YYYYMMDD-HHMMSS.log
• Safety option now available:
  • --dry-run runs the same poll/selection flow, but previews the ServiceM8 jobmaterial payloads only
  • dry-run does not write to ServiceM8
  • dry-run does not mark quote responses as applied
• Examples:
  • ./poll_and_apply_quote_templates.sh
  • ./poll_and_apply_quote_templates.sh --hours 48
  • ./poll_and_apply_quote_templates.sh --since '2026-05-04 08:00:00'
  • ./poll_and_apply_quote_templates.sh --dry-run --hours 48

This is the proposed scheduled entry point for soft release, e.g. every 10–30 minutes. For manual confidence checks, run it with --dry-run first, inspect the generated payloads/log, then rerun without --dry-run only when ready to apply.

Live Webhook Receiver Status

Receiver

• Script: servicem8_webhook_receiver.py
• Still receives/stores:
  • event webhooks
  • object webhooks
  • form-response webhooks
• DB:
  • servicem8_webhooks.db
• Quote Template webhook queue:
  • quote-template-jobmaterials-queue.jsonl

Important: webhook handling still does not perform live ServiceM8 writes. Heavy work stays outside the FastAPI webhook request path.

Inspector / Web Viewer

• Script: servicem8_inspector.py
• Current intended views include:
  • webhook events
  • webhook objects
  • webhook form responses
  • polled form responses
  • parsed polled Quote Template responses
  • poll runs
  • dry-run/apply runs
  • generated material state

Note: the code was updated and tested on a temporary localhost port, but the existing live inspector process may need a manual/service restart before all new pages appear in the running viewer.

Required Files for Current Soft Release

Scripts

• poll_and_apply_quote_templates.sh — scheduled wrapper / main operational entry point
• poll_form_responses_since.py — polls ServiceM8 and populates poll DB
• apply_polled_quote_template_jobmaterials.py — applies parsed responses to ServiceM8
• servicem8_quote_template_parser.py — parsing logic
• servicem8_inspector.py — web inspection/progress viewer
• servicem8_webhook_receiver.py — still useful for webhook capture/diagnostics

Databases / state

• servicem8_formresponse_poll.db — poll results, parsed quote responses, apply run status
• servicem8_quote_materials_state.db — created jobMaterial mapping/state to avoid duplicates
• servicem8_webhooks.db — webhook capture/archive/diagnostics

Queue/log files

• quote-template-jobmaterials-poll-queue.jsonl — poll-derived parsed queue/output
• logs/poll-and-apply-*.log — wrapper run logs
• quote-template-jobmaterials-queue.jsonl — older webhook-derived queue; still useful but not the primary soft-release path

Current Status

Operational soft-release pieces are now in place:

• API polling confirmed
• Quote Template detection confirmed
• parsing confirmed
• dry-run/apply command path tested
• payload adjusted for current ServiceM8 requirements
• duplicate-apply guard in place
• wrapper script created for scheduled operation
• wrapper now has a first-class --dry-run mode
• inspector updated for progress visibility

Checkpoint — 2026-05-05

Latest verified state:

• poll_and_apply_quote_templates.sh --dry-run added and documented.
• bash -n /opt/webhooks/poll_and_apply_quote_templates.sh passes.
• --help output includes --dry-run usage/examples.
• A future-since dry-run (--dry-run --since '2099-01-01 00:00:00') confirmed:
  • wrapper reports Mode: dry-run
  • poll step performs no ServiceM8 writes
  • apply step calls the Python apply tool without --apply
  • payload rows are emitted as would_create
  • responses are not marked applied by dry-run
• At checkpoint time, there were still several unapplied parsed Quote Template responses available for preview/apply; this is expected while the soft release remains manual.

Not Yet Done / Next Steps

• Restart/reload the live inspector process so the new poll/apply pages are available in the active web viewer.
• Decide schedule interval for poll_and_apply_quote_templates.sh — likely every 10 or 30 minutes.
• Run the wrapper manually in --dry-run mode against a controlled recent form response and inspect the payload/log.
• If payload is correct, rerun the wrapper without --dry-run for a controlled live apply smoke test.
• After confidence builds, wire the wrapper into cron/system scheduling.
• Future hardening: add reconciliation/update/delete behaviour if ServiceM8 quote form responses are edited after initial apply.

Design Notes

• Webhooks remain lightweight and non-mutating.
• Polling is now the reliable source of completeness.
• Applying to ServiceM8 is tracked locally and guarded against duplicates.
• The wrapper defaults to live apply for scheduled soft release, but now supports --dry-run for manual preview/safety checks.
• The underlying apply script remains dry-run-by-default and provides the duplicate protection used by the wrapper.