Compare commits

...

19 Commits

Author SHA1 Message Date
Soren_Molty 7993756068 Added the crontab to the repo so we can track the settings for the automation tasks 2026-05-21 13:25:03 +09:30
Soren_Molty 16283a64e3 Make dry-run report remote job material blockers 2026-05-18 15:50:11 +09:30
Soren_Molty 49e7a60f98 Fix quote description update field for ServiceM8 jobs 2026-05-18 12:43:21 +09:30
Soren_Molty 9dde6b28a1 I think this one was for the quote description field - but we got it wrong so we need to seatch and find the correct field...... 2026-05-18 11:57:16 +09:30
Soren_Molty 069bf11ec2 Added new docs for the monthly calendar creation 2026-05-13 09:01:51 +10:00
Soren_Molty c4248eba76 Updates to the inspector view to fix matching quote forms to be able to display and also the apply script which now checks for existing jobMaterials before writing. 2026-05-12 20:22:50 +09:30
Soren_Molty c425f45910 Modified the create webhook scripts to create them for the live server at webhook.naroomaplumbing.au 2026-05-12 12:49:41 +09:30
Soren_Molty 28bcee78cd Adding the systemd config files to the docs dir for completeness 2026-05-11 14:58:37 +10:00
Soren_Molty d7dc2ade06 Refactored the quote_push process to check for any existing jobMaterials and fail with a log entry and some intelligence so that we don't overwrite anything that has been manually created. Also added another list in the inspector.py code to allow us to view this when it occurs. 2026-05-11 14:41:26 +10:00
Soren_Molty f03840c574 Added migration notes 2026-05-11 07:19:52 +10:00
Soren_Molty 5993826b79 Add quote template dry-run wrapper checkpoint 2026-05-05 18:56:21 +10:00
Soren_Molty 868632bd21 End of day updates, almost at completion of live cron setup with bash script wrapper now as ell 2026-05-04 18:22:39 +10:00
Soren_Molty 2739606d6a apply_polled_quote_template_jobmaterials.py added to the repo and the docs 2026-05-04 14:31:05 +10:00
Soren_Molty 7e3538e745 quick backup 2026-05-04 13:14:44 +10:00
Soren_Molty 89386909a1 Adding updated inspector.py for human viewable formresponse.json requests 2026-05-04 11:29:48 +10:00
Soren_Molty 22c1040fe8 Intial new scripts for pivot logic approach 2026-05-04 11:18:17 +10:00
Soren_Molty 74c73883c9 Commit prior to form response checking pivot in logic 2026-05-04 11:17:14 +10:00
Soren_Molty cf395091a8 File added for the new name 2026-04-29 13:14:09 +10:00
Soren_Molty 1a7f663d39 File rename for the servicem8_quote_template_parser to use underscores so that the import library works 2026-04-29 13:13:34 +10:00
19 changed files with 4153 additions and 82 deletions
+6
View File
@@ -0,0 +1,6 @@
# The contents of the local user agent crontab
# Used to maintain and document the automated tasks for this system
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# m h dom mon dow command
4 4-23 * * * . /home/npsagent/.config/servicem8-webhooks.env; cd /opt/webhooks && flock -n /tmp/servicem8-quote-template.lock ./poll_and_apply_quote_templates.sh --hours 6 >> /opt/webhooks/logs/cron-driver.log 2>&1
+295
View File
@@ -0,0 +1,295 @@
Yep — I inspected the current setup. The “current bash wrapper” is the ServiceM8 plumbing quote-template pipeline under:
`/opt/webhooks`
In the workspace it appears as a symlink:
`/home/openclaw/.openclaw/workspace/projects/plumbing -> /opt/webhooks`
## Definitive migration set
### Required operational scripts
Copy these from `/opt/webhooks/`:
```text
poll_and_apply_quote_templates.sh
poll_form_responses_since.py
apply_polled_quote_template_jobmaterials.py
servicem8_quote_template_parser.py
servicem8_webhook_receiver.py
servicem8_inspector.py
```
These are the active pieces:
- `poll_and_apply_quote_templates.sh` — main wrapper / scheduled entry point
- `poll_form_responses_since.py` — polls ServiceM8 form responses
- `apply_polled_quote_template_jobmaterials.py` — applies parsed quote rows to ServiceM8 jobMaterials
- `servicem8_quote_template_parser.py` — parsing/state logic
- `servicem8_webhook_receiver.py` — webhook receiver/diagnostics
- `servicem8_inspector.py` — web UI inspector
### Required DB/state files
Copy these, ideally while services are stopped:
```text
servicem8_formresponse_poll.db
servicem8_quote_materials_state.db
servicem8_webhooks.db
```
Also copy any SQLite sidecar files if present at cutover:
```text
*.db-wal
*.db-shm
```
### Required JSONL queue/history files
```text
quote-template-jobmaterials-poll-queue.jsonl
quote-template-jobmaterials-queue.jsonl
```
First one is the current poll-derived queue. Second is older webhook-derived queue but worth keeping for audit/history.
### Useful but not strictly required
```text
PROJECT-PROGRESS.md
docs/
logs/
.git/
```
Id migrate these too unless you want a clean production-only deploy. The logs and docs are useful for future archaeology when something inevitably gets weird.
## Do not blindly migrate
I would **not** migrate the existing virtualenv folders directly:
```text
bin/
lib/
lib64/
include/
pyvenv.cfg
__pycache__/
```
They are host/Python-version sensitive. Recreate the venv on the new server.
## Python / pip packages
Current venv freeze shows:
```text
fastapi==0.136.0
uvicorn==0.45.0
requests==2.33.1
python-dotenv==1.2.2
pydantic==2.13.3
starlette==1.0.0
anyio==4.13.0
click==8.3.3
h11==0.16.0
httptools==0.7.1
uvloop==0.22.1
watchfiles==1.1.1
websockets==16.0
certifi==2026.4.22
charset-normalizer==3.4.7
idna==3.13
urllib3==2.6.3
PyYAML==6.0.3
typing_extensions==4.15.0
typing-inspection==0.4.2
annotated-types==0.7.0
annotated-doc==0.0.4
pydantic_core==2.46.3
```
Practical install line:
```bash
python3.12 -m venv /opt/webhooks
/opt/webhooks/bin/pip install --upgrade pip
/opt/webhooks/bin/pip install fastapi 'uvicorn[standard]' requests python-dotenv
```
Or create a `requirements.txt` from the freeze if you want exact locking.
System packages likely needed:
```bash
python3.12
python3.12-venv
sqlite3
bash
curl
rsync
systemd
```
Important: the wrapper uses GNU `date -d`, so Linux/GNU coreutils is assumed.
## Environment variables required
For polling/applying:
```text
SERVICEM8_ACCESS_TOKEN
# or
SERVICEM8_API_KEY
```
Optional/current defaults:
```text
SERVICEM8_BASE_URL=https://api.servicem8.com/api_1.0
SERVICEM8_TIMEOUT=30
SERVICEM8_QUOTE_TEMPLATE_FORM_UUID=3621b6be-1d19-4756-9ab4-9d5e4120f6d9
WEBHOOK_POLL_DB_PATH=/opt/webhooks/servicem8_formresponse_poll.db
WEBHOOK_RUN_LOG_DIR=/opt/webhooks/logs
```
For webhook receiver:
```text
WEBHOOK_HOST=0.0.0.0
WEBHOOK_PORT=18354
WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db
```
For inspector:
```text
WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db
WEBHOOK_STATE_DB_PATH=/opt/webhooks/servicem8_quote_materials_state.db
WEBHOOK_POLL_DB_PATH=/opt/webhooks/servicem8_formresponse_poll.db
WEBHOOK_INSPECTOR_HOST=0.0.0.0
WEBHOOK_INSPECTOR_PORT=18355
```
## Existing systemd services to recreate
Current services are:
```text
servicem8-dev-webhook.service
servicem8-dev-inspector.service
```
Current paths assume `/opt/webhooks`.
Webhook:
```ini
WorkingDirectory=/opt/webhooks
Environment="WEBHOOK_HOST=0.0.0.0"
Environment="WEBHOOK_PORT=18354"
Environment="WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db"
ExecStart=/opt/webhooks/bin/uvicorn servicem8_webhook_receiver:app --host 0.0.0.0 --port 18354
```
Inspector:
```ini
WorkingDirectory=/opt/webhooks
Environment="WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db"
Environment="WEBHOOK_INSPECTOR_HOST=0.0.0.0"
Environment="WEBHOOK_INSPECTOR_PORT=18355"
ExecStart=/opt/webhooks/bin/uvicorn servicem8_inspector:app --host 0.0.0.0 --port 18355
```
On the new host, probably change:
```ini
User=michael
Group=michael
```
to whatever the production service user is, e.g.:
```ini
User=openclaw
Group=openclaw
```
or create a dedicated `servicem8` user.
## Path/code changes for seamless migration
Best option: keep the same production path:
```text
/opt/webhooks
```
If you do that, almost no code changes are needed.
If you move it elsewhere, update:
1. systemd `WorkingDirectory`
2. systemd `ExecStart`
3. env vars:
- `WEBHOOK_DB_PATH`
- `WEBHOOK_POLL_DB_PATH`
- `WEBHOOK_STATE_DB_PATH`
- `WEBHOOK_RUN_LOG_DIR`
4. any OpenClaw cron or host cron entry that calls:
- `/opt/webhooks/poll_and_apply_quote_templates.sh`
One gotcha: `servicem8_quote_template_parser.py` currently hardcodes the state DB beside the script:
```python
STATE_DB_PATH = Path(__file__).with_name("servicem8_quote_materials_state.db")
```
So keeping DBs beside the scripts is the safest path. If you want DBs somewhere else, Id adjust that code to respect `WEBHOOK_STATE_DB_PATH`.
## External callback / production URL
Current webhook helper scripts still reference dev callback URLs like:
```text
https://nps-dev.coast2cloud.net/...
```
For the new production host, ServiceM8 webhook subscriptions need to point at the new public URL if you want inbound webhooks working.
Polling does **not** depend on inbound webhooks, so the wrapper can operate without this, but the receiver/inspector diagnostics wont capture new webhook events unless ServiceM8 is updated.
## Security note
There are old helper scripts with hardcoded ServiceM8 token fallbacks:
```text
servicem8-create-webhook-form-response.py
servicem8-create-webhook-job-object.py
servicem8-list-webhook-subscriptions-table.py
```
I would not treat those as production-safe as-is. Before migration to prod, remove hardcoded fallback tokens and require env vars only.
## Recommended cutover shape
1. Stop current webhook/inspector services.
2. Copy `/opt/webhooks` excluding venv/cache, including DBs/jsonl/logs.
3. Recreate venv on new host.
4. Install pip packages.
5. Add env/secrets via systemd `EnvironmentFile`, not hardcoded scripts.
6. Recreate systemd services.
7. Run:
```bash
/opt/webhooks/poll_and_apply_quote_templates.sh --dry-run --hours 48
```
8. If clean, run controlled live apply.
9. Only then schedule the wrapper.
At present I found no OpenClaw cron job already running this plumbing wrapper, so scheduling still appears to be a pending production decision rather than something to migrate from OpenClaw cron.
+172 -57
View File
@@ -1,73 +1,188 @@
# ServiceM8 Project Progress
## Current Quote Template → JobMaterials Pipeline
## Current Direction — Soft Release Poller + Apply Pipeline
### Live receiver
- `servicem8_webhook_receiver.py`
- receives `form.response_created`
- stores the raw webhook payload in `servicem8_webhooks.db`
- checks incoming form responses for the Quote Template `form_uuid`
- if matched, parses and queues the derived jobMaterials payload to:
- `quote-template-jobmaterials-queue.jsonl`
- does **not** perform live ServiceM8 writes at this stage
We changed tune from relying mainly on ServiceM8 `form.response_created` webhooks to using a scheduled API poller as the operational safety-net/primary soft-release path.
### Parser
- `servicem8-quote-template-parser.py`
- parses Quote Template `field_data`
- extracts:
Reason: ServiceM8 form webhooks are useful, but during testing some expected form completions did not reliably appear at the FastAPI listener. The ServiceM8 API endpoint `/api_1.0/formresponse.json` can be queried with a timestamp filter, so the safer operational pattern is now:
1. poll recent form responses with `timestamp gt 'YYYY-MM-DD HH:MM:SS'`
2. store every returned API row locally
3. detect Quote Template form responses
4. parse them into desired jobMaterial rows
5. apply only previously unapplied Quote Template responses to ServiceM8
6. track created jobMaterial UUIDs locally to avoid duplicate applies
This keeps the live webhook receiver useful as capture/diagnostics, but no longer depends on webhook delivery for completeness.
## Active Soft-Release Flow
### 1. Poll ServiceM8 form responses
- Script: `poll_form_responses_since.py`
- API endpoint: `/api_1.0/formresponse.json`
- Confirmed filter syntax:
- `?$filter=timestamp gt '2026-05-04 10:00:00'`
- Default operational DB:
- `servicem8_formresponse_poll.db`
- Poll queue/output:
- `quote-template-jobmaterials-poll-queue.jsonl`
- Quote Template form UUID:
- `3621b6be-1d19-4756-9ab4-9d5e4120f6d9`
The poller stores all fetched form responses in `form_responses_raw`, then parses Quote Template matches into `quote_template_form_responses`.
### 2. Parse Quote Template responses
- Parser: `servicem8_quote_template_parser.py`
- Extracts:
- description of works
- `Item 1..12` include lines
- `Works excluded 1..4` exclude lines
- extra descriptive include rows such as labour/scaffolding/equipment fields
- builds normalized `desired_job_materials`
- extra descriptive include rows such as labour/materials/scaffolding/equipment fields
- Builds normalized `desired_job_materials` rows.
### Create/apply script
- `servicem8-create-jobmaterials-from-form-response.py`
- standalone script
- consumes a Quote Template form response JSON payload
- builds ServiceM8 Job Material API payloads
- runs in **dry-run by default**
- supports live create later with `--apply`
- records created/generated mappings into local state DB
### 3. Select/apply parsed Quote Template responses
- Script: `apply_polled_quote_template_jobmaterials.py`
- Safe behaviour built in:
- processes one `form_response_uuid` at a time when called directly
- dry-run by default
- `--apply` performs live ServiceM8 `jobmaterial.json` creates
- refuses duplicate apply when generated material rows already exist for that form response unless `--force` is used
- before any live create, checks remote ServiceM8 `/jobmaterial.json` for existing rows on the target `job_uuid`
- if remote rows already exist, records a remote-existing incident and creates nothing unless `--force-remote-existing` is explicitly used
- Apply/incident tracking tables in `servicem8_formresponse_poll.db`:
- `quote_template_apply_runs`
- `quote_template_apply_run_rows`
- `quote_template_remote_existing_incidents`
- Created ServiceM8 job material mappings are recorded in:
- `servicem8_quote_materials_state.db`
### Local state tracking
- `servicem8_quote_materials_state.db`
- local SQLite DB for tracking generated jobMaterials
- intended fields include:
- job UUID
- form response UUID
- created job material UUID
- kind/source metadata
Current apply payload rules:
- All rows get:
- `tax_rate_uuid = 84e4dd28-06b3-452b-a796-1f58a20ac49b`
- `quantity = "0"`
- `price = ""`
- `displayed_amount = ""`
- `displayed_amount_is_tax_inclusive = ""`
- Header material UUIDs:
- `include_header``1924893b-917f-474a-adaa-2093bd622d4b`
- `exclude_header``4947bfd7-4875-48f7-9caf-2093b9751b9b`
- Non-header quote rows currently use:
- `f78b1d23-b9fa-40fe-a806-2425fe09cc0b`
### Queue/prepared output
- `quote-template-jobmaterials-queue.jsonl`
- lightweight queue/output file written by webhook stage
- contains parsed/prepared `desired_job_materials` objects
- no live update performed yet
### 4. Wrapper for scheduled/operational use
- Script: `poll_and_apply_quote_templates.sh`
- Default behaviour:
- polls the last 24 hours from script start
- stores/parses results
- applies any parsed Quote Template responses that are not already marked/applied
- logs each run under `logs/poll-and-apply-YYYYMMDD-HHMMSS.log`
- Safety options now available:
- `--dry-run` runs the same poll/selection flow, but previews the ServiceM8 `jobmaterial` payloads only
- dry-run does **not** write to ServiceM8
- dry-run does **not** mark quote responses as applied
- live apply blocks if remote ServiceM8 already has jobMaterial rows for the job, logging the captured rows/counts to `quote_template_remote_existing_incidents`
- `--recheck-remote-existing` revisits previously blocked rows without overriding the safety gate
- `--force-remote-existing` explicitly overrides the remote-existing safety gate and still records a forced incident before creating
- Examples:
- `./poll_and_apply_quote_templates.sh`
- `./poll_and_apply_quote_templates.sh --hours 48`
- `./poll_and_apply_quote_templates.sh --since '2026-05-04 08:00:00'`
- `./poll_and_apply_quote_templates.sh --dry-run --hours 48`
- `./poll_and_apply_quote_templates.sh --recheck-remote-existing --hours 48`
- `./poll_and_apply_quote_templates.sh --force-remote-existing --hours 48`
### Inspector
- `servicem8_inspector.py`
- read-only browser for webhook DB
- now also includes visibility of generated-materials state DB
This is the proposed scheduled entry point for soft release, e.g. every 1030 minutes. For manual confidence checks, run it with `--dry-run` first, inspect the generated payloads/log, then rerun without `--dry-run` only when ready to apply.
## Live Webhook Receiver Status
### Receiver
- Script: `servicem8_webhook_receiver.py`
- Still receives/stores:
- event webhooks
- object webhooks
- form-response webhooks
- DB:
- `servicem8_webhooks.db`
- Quote Template webhook queue:
- `quote-template-jobmaterials-queue.jsonl`
Important: webhook handling still does **not** perform live ServiceM8 writes. Heavy work stays outside the FastAPI webhook request path.
## Inspector / Web Viewer
- Script: `servicem8_inspector.py`
- Current intended views include:
- webhook events
- webhook objects
- webhook form responses
- polled form responses
- parsed polled Quote Template responses
- poll runs
- dry-run/apply runs
- generated material state
Note: the code was updated and tested on a temporary localhost port, but the existing live inspector process may need a manual/service restart before all new pages appear in the running viewer.
## Required Files for Current Soft Release
### Scripts
- `poll_and_apply_quote_templates.sh` — scheduled wrapper / main operational entry point
- `poll_form_responses_since.py` — polls ServiceM8 and populates poll DB
- `apply_polled_quote_template_jobmaterials.py` — applies parsed responses to ServiceM8
- `servicem8_quote_template_parser.py` — parsing logic
- `servicem8_inspector.py` — web inspection/progress viewer
- `servicem8_webhook_receiver.py` — still useful for webhook capture/diagnostics
### Databases / state
- `servicem8_formresponse_poll.db` — poll results, parsed quote responses, apply run status
- `servicem8_quote_materials_state.db` — created jobMaterial mapping/state to avoid duplicates
- `servicem8_webhooks.db` — webhook capture/archive/diagnostics
### Queue/log files
- `quote-template-jobmaterials-poll-queue.jsonl` — poll-derived parsed queue/output
- `logs/poll-and-apply-*.log` — wrapper run logs
- `quote-template-jobmaterials-queue.jsonl` — older webhook-derived queue; still useful but not the primary soft-release path
## Current Status
Everything is staged and connected up to the point of:
- webhook receive
- UUID trigger check
- form parsing
- local queueing
- dry-run jobMaterial payload generation
- local state DB support
- inspector visibility
## Not Yet Enabled
- actual live POST creation of Job Materials into ServiceM8 during webhook processing
- any automatic update/delete reconciliation against live ServiceM8 records
Operational soft-release pieces are now in place:
- API polling confirmed
- Quote Template detection confirmed
- parsing confirmed
- dry-run/apply command path tested
- payload adjusted for current ServiceM8 requirements
- duplicate-apply guard in place
- wrapper script created for scheduled operation
- wrapper now has a first-class `--dry-run` mode
- inspector updated for progress visibility
### Checkpoint — 2026-05-05
Latest verified state:
- `poll_and_apply_quote_templates.sh --dry-run` added and documented.
- `bash -n /opt/webhooks/poll_and_apply_quote_templates.sh` passes.
- `--help` output includes `--dry-run` usage/examples.
- A future-since dry-run (`--dry-run --since '2099-01-01 00:00:00'`) confirmed:
- wrapper reports `Mode: dry-run`
- poll step performs no ServiceM8 writes
- apply step calls the Python apply tool without `--apply`
- payload rows are emitted as `would_create`
- responses are not marked applied by dry-run
- At checkpoint time, there were still several unapplied parsed Quote Template responses available for preview/apply; this is expected while the soft release remains manual.
## Not Yet Done / Next Steps
- Restart/reload the live inspector process so the new poll/apply pages are available in the active web viewer.
- Decide schedule interval for `poll_and_apply_quote_templates.sh` — likely every 10 or 30 minutes.
- Run the wrapper manually in `--dry-run` mode against a controlled recent form response and inspect the payload/log.
- If payload is correct, rerun the wrapper without `--dry-run` for a controlled live apply smoke test.
- After confidence builds, wire the wrapper into cron/system scheduling.
- Future hardening: add reconciliation/update/delete behaviour if ServiceM8 quote form responses are edited after initial apply.
## Design Notes
- Heavy lifting is intentionally kept **out of the live webhook handler**.
- The webhook handler is used only for:
- capture
- UUID gate
- parse/prepare/queue
- Live ServiceM8 mutation remains a separate step/script for safety.
- Webhooks remain lightweight and non-mutating.
- Polling is now the reliable source of completeness.
- Applying to ServiceM8 is tracked locally and guarded against duplicates.
- The wrapper defaults to live apply for scheduled soft release, but now supports `--dry-run` for manual preview/safety checks.
- The underlying apply script remains dry-run-by-default and provides the duplicate protection used by the wrapper.
+728
View File
@@ -0,0 +1,728 @@
#!/usr/bin/env python3
"""Create ServiceM8 jobMaterials for one selected polled Quote Template response.
Safe-by-default:
- Requires an explicit form response UUID.
- Dry-run unless --apply is provided.
- Refuses to apply twice for the same form response unless --force is used.
- Records every dry-run/apply attempt in the poll DB for inspector visibility.
"""
from __future__ import annotations
import argparse
import importlib.util
import json
import os
import sqlite3
import sys
from contextlib import closing
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Dict, List, Optional
import requests
from servicem8_quote_template_parser import STATE_DB_PATH, init_state_db, record_generated_job_material
SCRIPT_DIR = Path(__file__).resolve().parent
POLL_DB_PATH = Path(os.getenv("WEBHOOK_POLL_DB_PATH", SCRIPT_DIR / "servicem8_formresponse_poll.db"))
BASE_URL = os.getenv("SERVICEM8_BASE_URL", "https://api.servicem8.com/api_1.0")
REQUEST_TIMEOUT = int(os.getenv("SERVICEM8_TIMEOUT", "30"))
DEV_QUOTE_MATERIAL_UUID = "f78b1d23-b9fa-40fe-a806-2425fe09cc0b"
QUOTE_INCLUDE_HEADER_MATERIAL_UUID = "1924893b-917f-474a-adaa-2093bd622d4b"
QUOTE_EXCLUDE_HEADER_MATERIAL_UUID = "4947bfd7-4875-48f7-9caf-2093b9751b9b"
DEV_QUOTE_TAX_RATE_UUID = "84e4dd28-06b3-452b-a796-1f58a20ac49b"
QUOTE_DESCRIPTION_PREFIX = "Thank you for the opportunity to quote to "
QUOTE_DESCRIPTION_SUFFIX = (
"Please find below a detailed breakdown of the proposed costs included in the quotation. "
"If you have any questions or concerns, please do not hesitate to contact us."
)
def utc_now() -> str:
return datetime.now(timezone.utc).isoformat()
def escape_filter_value(value: str) -> str:
return value.replace("'", "''")
def material_uuid_for_row(row: dict) -> str:
kind = row.get("kind", "")
if kind == "include_header":
return QUOTE_INCLUDE_HEADER_MATERIAL_UUID
if kind == "exclude_header":
return QUOTE_EXCLUDE_HEADER_MATERIAL_UUID
return DEV_QUOTE_MATERIAL_UUID
def build_payload(job_uuid: str, row: dict) -> dict:
return {
"job_uuid": job_uuid,
"material_uuid": material_uuid_for_row(row),
"tax_rate_uuid": DEV_QUOTE_TAX_RATE_UUID,
"name": row["name"],
"quantity": "0",
"price": "",
"displayed_amount": "",
"displayed_amount_is_tax_inclusive": "",
"sort_order": row["sort_order"],
}
def clean_text(value: Any) -> str:
if value is None:
return ""
return str(value).replace("\r\n", "\n").replace("\r", "\n").strip()
def first_text(*values: Any) -> str:
for value in values:
text = clean_text(value)
if text:
return text
return ""
def format_job_address(job: Dict[str, Any]) -> str:
direct = first_text(
job.get("job_address"),
job.get("site_address"),
job.get("address"),
job.get("location_address"),
job.get("billing_address"),
)
if direct:
return direct
parts = [
first_text(job.get("street"), job.get("street_address"), job.get("address_1"), job.get("address1")),
first_text(job.get("suburb"), job.get("city")),
first_text(job.get("state")),
first_text(job.get("postcode"), job.get("postal_code"), job.get("zip")),
]
return " ".join(part for part in parts if part)
def build_quote_description_text(description: str, job: Dict[str, Any]) -> str:
description = clean_text(description)
if not description:
return ""
address = format_job_address(job) or "the job address"
return f"{QUOTE_DESCRIPTION_PREFIX} {description} at {address}. {QUOTE_DESCRIPTION_SUFFIX}"
def build_job_update_payload(description: str, job: Dict[str, Any]) -> dict:
quote_description = build_quote_description_text(description, job)
return {"work_done_description": quote_description} if quote_description else {}
def retrieve_job(session: requests.Session, job_uuid: str) -> Dict[str, Any]:
response = session.get(f"{BASE_URL}/job/{job_uuid}.json", timeout=REQUEST_TIMEOUT)
if not response.ok:
raise RuntimeError(f"Job retrieve failed: HTTP {response.status_code} :: {response.text[:1000]}")
data = response.json()
if not isinstance(data, dict):
raise RuntimeError(f"Job retrieve expected object response, got {type(data).__name__}")
return data
def update_job_description(session: requests.Session, job_uuid: str, payload: dict) -> None:
response = session.post(f"{BASE_URL}/job/{job_uuid}.json", json=payload, timeout=REQUEST_TIMEOUT)
if not response.ok:
raise RuntimeError(f"Job quote description update failed: HTTP {response.status_code} :: {response.text[:1000]}")
def extract_company_name(job: Dict[str, Any]) -> str:
related = job.get("related")
if isinstance(related, dict):
company = related.get("company")
if isinstance(company, dict):
company_name = clean_text(company.get("name"))
if company_name:
return company_name
company = job.get("company")
if isinstance(company, dict):
company_name = clean_text(company.get("name"))
if company_name:
return company_name
return first_text(job.get("company_name"), job.get("customer_name"))
def upsert_job_metadata(conn: sqlite3.Connection, *, job_uuid: str, job: Dict[str, Any], source: str) -> None:
job_uuid = clean_text(job_uuid or job.get("uuid"))
if not job_uuid:
return
now = utc_now()
conn.execute(
"""
INSERT INTO job_metadata (
job_uuid, generated_job_id, job_address, company_name, raw_json,
first_seen_at, last_seen_at, source
) VALUES (?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(job_uuid) DO UPDATE SET
generated_job_id = excluded.generated_job_id,
job_address = excluded.job_address,
company_name = excluded.company_name,
raw_json = excluded.raw_json,
last_seen_at = excluded.last_seen_at,
source = excluded.source
""",
(
job_uuid,
clean_text(job.get("generated_job_id")),
format_job_address(job),
extract_company_name(job),
json.dumps(job, ensure_ascii=False, sort_keys=True),
now,
now,
source,
),
)
conn.commit()
def create_job_material(session: requests.Session, payload: dict) -> str:
response = session.post(f"{BASE_URL}/jobmaterial.json", json=payload, timeout=REQUEST_TIMEOUT)
if not response.ok:
raise RuntimeError(f"Create failed: HTTP {response.status_code} :: {response.text}")
record_uuid = response.headers.get("x-record-uuid", "")
if not record_uuid:
raise RuntimeError("Create succeeded but x-record-uuid header was missing")
return record_uuid
def list_remote_job_materials(session: requests.Session, job_uuid: str) -> List[Dict[str, Any]]:
"""Return existing remote ServiceM8 jobMaterial rows for a job."""
filter_expr = f"job_uuid eq '{escape_filter_value(job_uuid)}'"
response = session.get(f"{BASE_URL}/jobmaterial.json", params={"$filter": filter_expr}, timeout=REQUEST_TIMEOUT)
if not response.ok:
raise RuntimeError(f"Remote jobMaterial preflight failed: HTTP {response.status_code} :: {response.text[:1000]}")
data = response.json()
if not isinstance(data, list):
raise RuntimeError(f"Remote jobMaterial preflight expected list response, got {type(data).__name__}")
return data
def is_active_remote_job_material(row: Dict[str, Any]) -> bool:
value = row.get("active", 1)
return value not in (0, "0", False, "false", "False")
def load_api_key() -> str:
for name in ("SERVICEM8_ACCESS_TOKEN", "SERVICEM8_API_KEY"):
value = os.getenv(name)
if value:
return value
fallback = SCRIPT_DIR / "servicem8-list-webhook-subscriptions-table.py"
if fallback.exists():
spec = importlib.util.spec_from_file_location("servicem8_subs", fallback)
if spec and spec.loader:
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module) # type: ignore[union-attr]
value = getattr(module, "ACCESS_TOKEN", None)
if value:
return str(value)
raise RuntimeError("Missing SERVICEM8_ACCESS_TOKEN or SERVICEM8_API_KEY")
def get_conn(db_path: Path = POLL_DB_PATH) -> sqlite3.Connection:
conn = sqlite3.connect(db_path, timeout=30)
conn.row_factory = sqlite3.Row
return conn
def init_apply_tables(conn: sqlite3.Connection) -> None:
conn.execute(
"""
CREATE TABLE IF NOT EXISTS job_metadata (
job_uuid TEXT PRIMARY KEY,
generated_job_id TEXT,
job_address TEXT,
company_name TEXT,
raw_json TEXT NOT NULL,
first_seen_at TEXT NOT NULL,
last_seen_at TEXT NOT NULL,
source TEXT NOT NULL
)
"""
)
conn.execute("CREATE INDEX IF NOT EXISTS idx_job_metadata_generated_job_id ON job_metadata(generated_job_id)")
conn.execute(
"""
CREATE TABLE IF NOT EXISTS quote_template_apply_runs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
form_response_uuid TEXT NOT NULL,
job_uuid TEXT NOT NULL,
mode TEXT NOT NULL,
started_at TEXT NOT NULL,
finished_at TEXT,
desired_count INTEGER NOT NULL DEFAULT 0,
created_count INTEGER NOT NULL DEFAULT 0,
status TEXT NOT NULL,
error TEXT
)
"""
)
conn.execute(
"""
CREATE TABLE IF NOT EXISTS quote_template_apply_run_rows (
id INTEGER PRIMARY KEY AUTOINCREMENT,
run_id INTEGER NOT NULL,
form_response_uuid TEXT NOT NULL,
job_uuid TEXT NOT NULL,
row_index INTEGER NOT NULL,
kind TEXT,
source_question TEXT,
name TEXT,
api_payload_json TEXT NOT NULL,
action TEXT NOT NULL,
job_material_uuid TEXT,
error TEXT,
created_at TEXT NOT NULL,
FOREIGN KEY(run_id) REFERENCES quote_template_apply_runs(id)
)
"""
)
conn.execute(
"""
CREATE TABLE IF NOT EXISTS quote_template_remote_existing_incidents (
id INTEGER PRIMARY KEY AUTOINCREMENT,
detected_at TEXT NOT NULL,
form_response_uuid TEXT NOT NULL,
job_uuid TEXT NOT NULL,
apply_run_id INTEGER,
desired_count INTEGER NOT NULL DEFAULT 0,
remote_count INTEGER NOT NULL DEFAULT 0,
remote_active_count INTEGER NOT NULL DEFAULT 0,
action TEXT NOT NULL,
reason TEXT,
remote_rows_json TEXT NOT NULL,
FOREIGN KEY(apply_run_id) REFERENCES quote_template_apply_runs(id)
)
"""
)
conn.execute("CREATE INDEX IF NOT EXISTS idx_quote_apply_runs_form_response ON quote_template_apply_runs(form_response_uuid)")
conn.execute("CREATE INDEX IF NOT EXISTS idx_quote_apply_rows_run ON quote_template_apply_run_rows(run_id)")
conn.execute("CREATE INDEX IF NOT EXISTS idx_quote_remote_existing_form_response ON quote_template_remote_existing_incidents(form_response_uuid)")
conn.execute("CREATE INDEX IF NOT EXISTS idx_quote_remote_existing_job_uuid ON quote_template_remote_existing_incidents(job_uuid)")
conn.commit()
def load_quote(conn: sqlite3.Connection, form_response_uuid: str) -> sqlite3.Row:
row = conn.execute(
"select * from quote_template_form_responses where form_response_uuid = ?",
(form_response_uuid,),
).fetchone()
if not row:
raise SystemExit(f"No parsed polled Quote Template response found for UUID: {form_response_uuid}")
return row
def existing_created_for_form(form_response_uuid: str) -> int:
if not STATE_DB_PATH.exists():
return 0
try:
with closing(sqlite3.connect(STATE_DB_PATH, timeout=30)) as conn:
row = conn.execute(
"select count(*) from generated_job_materials where form_response_uuid = ?",
(form_response_uuid,),
).fetchone()
return int(row[0] if row else 0)
except sqlite3.Error:
return 0
def create_apply_run(conn: sqlite3.Connection, *, form_response_uuid: str, job_uuid: str, mode: str, desired_count: int) -> int:
cur = conn.execute(
"""
INSERT INTO quote_template_apply_runs (
form_response_uuid, job_uuid, mode, started_at, desired_count, status
) VALUES (?, ?, ?, ?, ?, ?)
""",
(form_response_uuid, job_uuid, mode, utc_now(), desired_count, "running"),
)
conn.commit()
return int(cur.lastrowid)
def finish_apply_run(conn: sqlite3.Connection, run_id: int, *, status: str, created_count: int, error: Optional[str] = None) -> None:
conn.execute(
"""
UPDATE quote_template_apply_runs
SET finished_at = ?, status = ?, created_count = ?, error = ?
WHERE id = ?
""",
(utc_now(), status, created_count, error, run_id),
)
conn.commit()
def record_apply_row(
conn: sqlite3.Connection,
*,
run_id: int,
form_response_uuid: str,
job_uuid: str,
row_index: int,
row: Dict[str, Any],
api_payload: Dict[str, Any],
action: str,
job_material_uuid: str = "",
error: Optional[str] = None,
) -> None:
conn.execute(
"""
INSERT INTO quote_template_apply_run_rows (
run_id, form_response_uuid, job_uuid, row_index, kind, source_question,
name, api_payload_json, action, job_material_uuid, error, created_at
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(
run_id,
form_response_uuid,
job_uuid,
row_index,
row.get("kind", ""),
row.get("source_question", ""),
row.get("name", ""),
json.dumps(api_payload, ensure_ascii=False, sort_keys=True),
action,
job_material_uuid,
error,
utc_now(),
),
)
conn.commit()
def record_remote_existing_incident(
conn: sqlite3.Connection,
*,
form_response_uuid: str,
job_uuid: str,
apply_run_id: int,
desired_count: int,
remote_rows: List[Dict[str, Any]],
action: str,
reason: str,
) -> int:
remote_active_count = sum(1 for row in remote_rows if is_active_remote_job_material(row))
cur = conn.execute(
"""
INSERT INTO quote_template_remote_existing_incidents (
detected_at, form_response_uuid, job_uuid, apply_run_id, desired_count,
remote_count, remote_active_count, action, reason, remote_rows_json
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(
utc_now(),
form_response_uuid,
job_uuid,
apply_run_id,
desired_count,
len(remote_rows),
remote_active_count,
action,
reason,
json.dumps(remote_rows, ensure_ascii=False, sort_keys=True),
),
)
conn.commit()
return int(cur.lastrowid)
def list_pending(conn: sqlite3.Connection) -> List[Dict[str, Any]]:
rows = conn.execute(
"""
select form_response_uuid, discovered_at, job_uuid, description, queued_at,
processed_at, process_status, desired_job_materials_json
from quote_template_form_responses
order by discovered_at desc
"""
).fetchall()
out = []
for row in rows:
existing = existing_created_for_form(row["form_response_uuid"])
try:
desired_count = len(json.loads(row["desired_job_materials_json"] or "[]"))
except Exception:
desired_count = None
out.append(
{
"form_response_uuid": row["form_response_uuid"],
"discovered_at": row["discovered_at"],
"job_uuid": row["job_uuid"],
"description": row["description"],
"desired_count": desired_count,
"created_in_state_db": existing,
"processed_at": row["processed_at"],
"process_status": row["process_status"],
}
)
return out
def main() -> int:
parser = argparse.ArgumentParser(description="Dry-run or apply one selected polled Quote Template response to ServiceM8 jobMaterials")
parser.add_argument("--uuid", help="Polled Quote Template form_response_uuid to process")
parser.add_argument("--db", default=str(POLL_DB_PATH), help="Poll DB path")
parser.add_argument("--apply", action="store_true", help="Actually create ServiceM8 jobMaterial records. Default is dry-run.")
parser.add_argument("--force", action="store_true", help="Allow apply even if generated_job_materials already contains rows for this form response")
parser.add_argument("--force-remote-existing", action="store_true", help="Allow apply even when ServiceM8 already has jobMaterial rows for the target job; records a forced incident before creating")
parser.add_argument("--list", action="store_true", help="List parsed polled Quote Template responses and exit")
parser.add_argument("--pretty", action="store_true", help="Pretty-print JSON output")
args = parser.parse_args()
db_path = Path(args.db)
if not db_path.exists():
raise SystemExit(f"Poll DB not found: {db_path}")
with closing(get_conn(db_path)) as conn:
init_apply_tables(conn)
if args.list:
print(json.dumps({"ok": True, "rows": list_pending(conn)}, indent=2 if args.pretty else None, ensure_ascii=False))
return 0
if not args.uuid:
raise SystemExit("--uuid is required unless using --list")
quote = load_quote(conn, args.uuid)
form_response_uuid = quote["form_response_uuid"]
job_uuid = quote["job_uuid"]
desired_rows = json.loads(quote["desired_job_materials_json"] or "[]")
mode = "apply" if args.apply else "dry-run"
existing_created = existing_created_for_form(form_response_uuid)
if args.apply and existing_created and not args.force:
raise SystemExit(
f"Refusing duplicate apply: {existing_created} generated_job_materials already recorded for {form_response_uuid}. Use --force only if intentional."
)
run_id = create_apply_run(
conn,
form_response_uuid=form_response_uuid,
job_uuid=job_uuid,
mode=mode,
desired_count=len(desired_rows),
)
result = {
"ok": True,
"mode": mode,
"run_id": run_id,
"form_response_uuid": form_response_uuid,
"job_uuid": job_uuid,
"description": quote["description"],
"desired_count": len(desired_rows),
"created_count": 0,
"state_db_path": str(STATE_DB_PATH),
"rows": [],
}
try:
api_key = load_api_key()
session = requests.Session()
session.headers.update({"X-Api-Key": api_key, "Accept": "application/json", "Content-Type": "application/json"})
quote_description_source = clean_text(quote["description"])
job_details = retrieve_job(session, job_uuid) if quote_description_source else {}
if job_details:
upsert_job_metadata(conn, job_uuid=job_uuid, job=job_details, source=mode)
job_update_payload = build_job_update_payload(quote_description_source, job_details)
job_update_record_payload = {
"endpoint": f"/job/{job_uuid}.json",
"payload": job_update_payload,
"source_description": quote_description_source,
"job_address": format_job_address(job_details) if job_details else "",
}
job_update_row = {
"kind": "work_done_description",
"source_question": "Description of Works to be Quoted",
"name": job_update_payload.get("work_done_description", ""),
}
if not args.apply:
remote_existing_rows = list_remote_job_materials(session, job_uuid)
remote_existing_blocks_apply = bool(remote_existing_rows)
if remote_existing_blocks_apply:
remote_active_count = sum(1 for remote_row in remote_existing_rows if is_active_remote_job_material(remote_row))
reason = (
f"Remote ServiceM8 job already has {len(remote_existing_rows)} jobMaterial row(s) "
f"({remote_active_count} active); apply would be blocked before updates or creates"
)
incident_id = record_remote_existing_incident(
conn,
form_response_uuid=form_response_uuid,
job_uuid=job_uuid,
apply_run_id=run_id,
desired_count=len(desired_rows),
remote_rows=remote_existing_rows,
action="dry_run_would_block",
reason=reason,
)
result["remote_existing"] = {
"incident_id": incident_id,
"action": "would_block_remote_existing",
"remote_count": len(remote_existing_rows),
"remote_active_count": remote_active_count,
"reason": reason,
}
if job_update_payload:
job_update_action = (
"would_update_work_done_description_if_remote_empty"
if remote_existing_blocks_apply
else "would_update_work_done_description"
)
record_apply_row(
conn,
run_id=run_id,
form_response_uuid=form_response_uuid,
job_uuid=job_uuid,
row_index=0,
row=job_update_row,
api_payload=job_update_record_payload,
action=job_update_action,
)
result["job_update"] = {"action": job_update_action, **job_update_record_payload}
else:
result["job_update"] = {"action": "skipped", "reason": "Quote description is empty"}
for idx, row in enumerate(desired_rows, start=1):
api_payload = build_payload(job_uuid, row)
row_action = "would_create_if_remote_empty" if remote_existing_blocks_apply else "would_create"
record_apply_row(
conn,
run_id=run_id,
form_response_uuid=form_response_uuid,
job_uuid=job_uuid,
row_index=idx,
row=row,
api_payload=api_payload,
action=row_action,
)
result["rows"].append({"action": row_action, "kind": row.get("kind"), "payload": api_payload})
finish_apply_run(conn, run_id, status="dry-run", created_count=0)
conn.execute(
"UPDATE quote_template_form_responses SET process_status = ? WHERE form_response_uuid = ?",
("dry-run", form_response_uuid),
)
conn.commit()
print(json.dumps(result, indent=2 if args.pretty else None, ensure_ascii=False))
return 0
init_state_db(STATE_DB_PATH)
remote_existing_rows = list_remote_job_materials(session, job_uuid)
if remote_existing_rows:
remote_active_count = sum(1 for remote_row in remote_existing_rows if is_active_remote_job_material(remote_row))
action = "forced" if args.force_remote_existing else "blocked"
reason = (
f"Remote ServiceM8 job already has {len(remote_existing_rows)} jobMaterial row(s) "
f"({remote_active_count} active); no creates attempted"
if not args.force_remote_existing
else f"Remote ServiceM8 job already has {len(remote_existing_rows)} jobMaterial row(s) "
f"({remote_active_count} active); create forced by --force-remote-existing"
)
incident_id = record_remote_existing_incident(
conn,
form_response_uuid=form_response_uuid,
job_uuid=job_uuid,
apply_run_id=run_id,
desired_count=len(desired_rows),
remote_rows=remote_existing_rows,
action=action,
reason=reason,
)
result["remote_existing"] = {
"incident_id": incident_id,
"action": action,
"remote_count": len(remote_existing_rows),
"remote_active_count": remote_active_count,
"reason": reason,
}
if not args.force_remote_existing:
finish_apply_run(conn, run_id, status="blocked_remote_existing", created_count=0)
conn.execute(
"UPDATE quote_template_form_responses SET processed_at = ?, process_status = ?, process_error = NULL WHERE form_response_uuid = ?",
(utc_now(), "blocked_remote_existing", form_response_uuid),
)
conn.commit()
result["status"] = "blocked_remote_existing"
print(json.dumps(result, indent=2 if args.pretty else None, ensure_ascii=False))
return 0
if job_update_payload:
update_job_description(session, job_uuid, job_update_payload)
record_apply_row(
conn,
run_id=run_id,
form_response_uuid=form_response_uuid,
job_uuid=job_uuid,
row_index=0,
row=job_update_row,
api_payload=job_update_record_payload,
action="updated_work_done_description",
)
result["job_update"] = {"action": "updated_work_done_description", **job_update_record_payload}
else:
result["job_update"] = {"action": "skipped", "reason": "Quote description is empty"}
created_count = 0
for idx, row in enumerate(desired_rows, start=1):
api_payload = build_payload(job_uuid, row)
created_uuid = create_job_material(session, api_payload)
created_count += 1
record_generated_job_material(
job_uuid=job_uuid,
form_response_uuid=form_response_uuid,
job_material_uuid=created_uuid,
kind=row.get("kind", ""),
source_field_uuid=row.get("source_field_uuid", ""),
source_question=row.get("source_question", ""),
source_text=row.get("name", ""),
db_path=STATE_DB_PATH,
)
record_apply_row(
conn,
run_id=run_id,
form_response_uuid=form_response_uuid,
job_uuid=job_uuid,
row_index=idx,
row=row,
api_payload=api_payload,
action="created",
job_material_uuid=created_uuid,
)
result["rows"].append({"action": "created", "kind": row.get("kind"), "job_material_uuid": created_uuid, "payload": api_payload})
result["created_count"] = created_count
finish_apply_run(conn, run_id, status="applied", created_count=created_count)
conn.execute(
"UPDATE quote_template_form_responses SET processed_at = ?, process_status = ?, process_error = NULL WHERE form_response_uuid = ?",
(utc_now(), "applied", form_response_uuid),
)
conn.commit()
print(json.dumps(result, indent=2 if args.pretty else None, ensure_ascii=False))
return 0
except Exception as exc:
finish_apply_run(conn, run_id, status="error", created_count=result.get("created_count", 0), error=str(exc))
conn.execute(
"UPDATE quote_template_form_responses SET process_status = ?, process_error = ? WHERE form_response_uuid = ?",
("error", str(exc), form_response_uuid),
)
conn.commit()
raise
if __name__ == "__main__":
try:
raise SystemExit(main())
except Exception as exc:
print(str(exc), file=sys.stderr)
raise SystemExit(1)
+154
View File
@@ -0,0 +1,154 @@
#!/usr/bin/env python3
import argparse
import json
import sqlite3
import sys
from datetime import datetime
from pathlib import Path
from typing import Any, Dict, Optional
DB_PATH = Path(__file__).with_name("servicem8_webhooks.db")
STATE_PATH = Path(__file__).with_name(".quote_form_response_watch_state.json")
QUOTE_TEMPLATE_FORM_UUID = "3621b6be-1d19-4756-9ab4-9d5e4120f6d9"
def load_state(path: Path) -> Dict[str, Any]:
if not path.exists():
return {}
try:
return json.loads(path.read_text())
except Exception:
return {}
def save_state(path: Path, state: Dict[str, Any]) -> None:
path.write_text(json.dumps(state, indent=2, ensure_ascii=False) + "\n")
def get_conn(db_path: Path) -> sqlite3.Connection:
conn = sqlite3.connect(db_path)
conn.row_factory = sqlite3.Row
return conn
def query_latest_quote_response(db_path: Path) -> Optional[Dict[str, Any]]:
sql = """
SELECT
id,
received_at,
json_extract(payload_json, '$.id') AS webhook_event_id,
json_extract(payload_json, '$.type') AS event_type,
json_extract(payload_json, '$.data.uuid') AS form_response_uuid,
json_extract(payload_json, '$.data.form_uuid') AS form_uuid,
json_extract(payload_json, '$.data.regarding_object_uuid') AS job_uuid,
json_extract(payload_json, '$.data.edit_date') AS edit_date,
json_extract(payload_json, '$.data.created_by_staff_uuid') AS created_by_staff_uuid,
json_extract(payload_json, '$.data.status') AS status,
payload_json
FROM webhook_form_responses
WHERE json_extract(payload_json, '$.data.form_uuid') = ?
ORDER BY datetime(received_at) DESC, id DESC
LIMIT 1
"""
with get_conn(db_path) as conn:
row = conn.execute(sql, (QUOTE_TEMPLATE_FORM_UUID,)).fetchone()
return dict(row) if row else None
def build_summary(row: Dict[str, Any]) -> str:
received_at = row.get("received_at") or "unknown"
edit_date = row.get("edit_date") or "unknown"
job_uuid = row.get("job_uuid") or "unknown"
form_response_uuid = row.get("form_response_uuid") or "unknown"
return (
"New quote template form response received\n"
f"- Received: {received_at}\n"
f"- Edit date: {edit_date}\n"
f"- Job UUID: {job_uuid}\n"
f"- Form response UUID: {form_response_uuid}"
)
def main() -> int:
parser = argparse.ArgumentParser(description="Check for new quote template form responses in the ServiceM8 webhook DB")
parser.add_argument("--db", default=str(DB_PATH), help="Path to servicem8_webhooks.db")
parser.add_argument("--state", default=str(STATE_PATH), help="Path to local state file")
parser.add_argument("--prime", action="store_true", help="Record the current latest event as seen without alerting")
parser.add_argument("--json", action="store_true", help="Print machine-readable JSON")
args = parser.parse_args()
db_path = Path(args.db)
state_path = Path(args.state)
if not db_path.exists():
print(f"Database not found: {db_path}", file=sys.stderr)
return 2
latest = query_latest_quote_response(db_path)
state = load_state(state_path)
if not latest:
result = {
"status": "no_quote_responses_found",
"alert": False,
"db_path": str(db_path),
"state_path": str(state_path),
}
print(json.dumps(result, indent=2) if args.json else "No quote template form responses found in DB")
return 0
current_marker = {
"id": latest.get("id"),
"received_at": latest.get("received_at"),
"form_response_uuid": latest.get("form_response_uuid"),
"job_uuid": latest.get("job_uuid"),
"edit_date": latest.get("edit_date"),
"checked_at": datetime.now().astimezone().isoformat(),
}
if args.prime or not state:
save_state(state_path, current_marker)
result = {
"status": "primed",
"alert": False,
"latest": current_marker,
"db_path": str(db_path),
"state_path": str(state_path),
}
print(json.dumps(result, indent=2) if args.json else f"Primed latest quote template event at {current_marker['received_at']}")
return 0
last_seen_id = state.get("id")
last_seen_form_response_uuid = state.get("form_response_uuid")
is_new = (latest.get("id") != last_seen_id) or (
latest.get("form_response_uuid") and latest.get("form_response_uuid") != last_seen_form_response_uuid
)
if is_new:
save_state(state_path, current_marker)
result = {
"status": "new_quote_response",
"alert": True,
"summary": build_summary(latest),
"latest": current_marker,
"previous": state,
"db_path": str(db_path),
"state_path": str(state_path),
}
print(json.dumps(result, indent=2) if args.json else result["summary"])
return 0
result = {
"status": "no_new_quote_response",
"alert": False,
"latest": current_marker,
"previous": state,
"db_path": str(db_path),
"state_path": str(state_path),
}
print(json.dumps(result, indent=2) if args.json else f"No new quote template form response since {state.get('received_at')}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+411
View File
@@ -0,0 +1,411 @@
# List all Form Responses
#### Filtering
This endpoint supports result filtering. For more information on how to filter this request, [go here](/docs/filtering).
#### OAuth Scope
This endpoint requires the following OAuth scope **read_forms**.
# OpenAPI definition
```json
{
"openapi": "3.1.0",
"info": {
"title": "ServiceM8 API",
"description": "Move your app forward with the ServiceM8 API\n\n\n\n## Limits and Throttling\nTo ensure continuous quality of service, API usage can be subject to throttling. The throttle will be applied once an API consumer reaches a certain \nthreshold in terms of a maximum number of requests per minute. Most clients will never hit this threshold, but those that do, will get met by a \nHTTP 429 Too Many Requests response code. \n \nThere is a limit of 180 requests per minute, if you reach this you will receive a HTTP 429 with a text body of \"Number of allowed API requests per minute exceeded\".\nThere is a limit of 20000 requests per day, if you reach this you will receive a HTTP 429 with a text body of \"Number of allowed API requests per day exceeded\".\n\nWe encourage all API developers to anticipate this error, and take appropriate measures like e.g. using a cached value from a previous call, or passing on a message to the end user that gets subjected to this behaviour (if any).\n\nLimits are per Addon per account.\n",
"termsOfService": "https://www.servicem8.com/terms-of-service",
"version": "1.0.0"
},
"servers": [
{
"url": "https://api.servicem8.com/api_1.0"
}
],
"security": [
{
"apiKey": []
},
{
"oauth2": []
}
],
"paths": {
"/formresponse.json": {
"get": {
"tags": [
"Form Responses"
],
"operationId": "listFormResponses",
"summary": "List all Form Responses",
"description": "\n\t\t\t\n#### Filtering\nThis endpoint supports result filtering. For more information on how to filter this request, [go here](/docs/filtering).\n\t\t\t\n\t\t\t\n#### OAuth Scope\nThis endpoint requires the following OAuth scope **read_forms**.\n\n\t\t\t",
"security": [
{
"apiKey": []
},
{
"oauth2": [
"read_forms"
]
}
],
"responses": {
"200": {
"description": "An array of Form Responses",
"content": {
"application/json": {
"schema": {
"type": "array",
"items": {
"$ref": "#/components/schemas/FormResponse"
}
},
"examples": {
"success": {
"value": [
{
"uuid": "123e4567-af16-4618-80c7-23f940cc449b",
"active": 1,
"edit_date": "2026-03-01 12:00:00",
"form_uuid": "123e4567-f8b2-4802-a8db-23f94cf50e1b",
"staff_uuid": "123e4567-5c89-4cb3-8dc0-23f94d75521b",
"regarding_object": "string",
"regarding_object_uuid": "123e4567-016e-482c-a3f7-23f943f6d72b",
"field_data": "string",
"timestamp": "2026-03-01 12:00:00",
"form_by_staff_uuid": "123e4567-44e8-448d-925c-23f9435b205b",
"document_attachment_uuid": "123e4567-2f15-479f-8791-23f94c0fbbbb",
"asset_uuid": "123e4567-89f4-4183-ae1e-23f94832e59b"
}
]
}
}
}
}
},
"400": {
"description": "Bad Request - The request is malformed or contains invalid parameters",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Error"
},
"examples": {
"badRequest": {
"value": {
"errorCode": "1000",
"message": "An error occurred completing your request"
}
}
}
}
}
},
"401": {
"description": "Unauthorized - Authentication credentials are missing or invalid",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/AuthenticationError"
},
"examples": {
"unauthorized": {
"value": {
"errorCode": "401",
"message": "Authentication failed. Please check your API key or OAuth token."
}
}
}
}
}
},
"403": {
"description": "Forbidden - You don't have permission to access this resource",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ForbiddenError"
},
"examples": {
"forbidden": {
"value": {
"errorCode": "403",
"message": "Access forbidden. You don't have permission to access this resource."
}
}
}
}
}
},
"429": {
"description": "Too Many Requests - You have exceeded the rate limit",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/RateLimitError"
},
"examples": {
"rateLimitMinute": {
"value": {
"errorCode": 429,
"message": "Number of allowed API requests per minute exceeded"
}
},
"rateLimitDay": {
"value": {
"errorCode": 429,
"message": "Number of allowed API requests per day exceeded"
}
}
}
}
}
},
"500": {
"description": "Internal Server Error - An unexpected error occurred on the server",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Error"
},
"examples": {
"serverError": {
"value": {
"errorCode": 500,
"message": "An unexpected error occurred. Please try again later."
}
}
}
}
}
}
}
}
}
},
"components": {
"securitySchemes": {
"apiKey": {
"type": "apiKey",
"name": "X-Api-Key",
"in": "header"
},
"oauth2": {
"type": "oauth2",
"flows": {
"authorizationCode": {
"authorizationUrl": "https://api.servicem8.com/oauth/authorize",
"tokenUrl": "https://api.servicem8.com/oauth/access_token",
"scopes": {
"staff_locations": "Access to real-time GPS information about staff",
"staff_activity": "Access to clock on, lunch break and clock off information about staff",
"publish_sms": "Access to send SMS messages to customers and/or staff on your behalf. Note sending SMS messages will incur account charges.",
"publish_email": "Access to send Email messages to customers and/or staff on your behalf",
"vendor": "Access to basic account information",
"vendor_logo": "Access to account logo",
"vendor_email": "Access to account holder email address",
"read_locations": "Read-only access to Location Endpoint",
"manage_locations": "Full access to Location Endpoint",
"read_staff": "Read-only access to Staff Endpoint",
"manage_staff": "Full access to Staff Endpoint",
"read_customers": "Read-only access to Company Endpoint",
"manage_customers": "Full access to Company Endpoint",
"read_customer_contacts": "Read-only access to CompanyContact Endpoint",
"manage_customer_contacts": "Full access to CompanyContact Endpoint",
"read_jobs": "Read-only access to Job Endpoint",
"manage_jobs": "Full access to Job Endpoint",
"create_jobs": "Ability to create jobs on behalf of account. Note creating jobs may incur account charges.",
"read_job_contacts": "Read-only access to JobContact Endpoint",
"manage_job_contacts": "Full access to JobContact Endpoint",
"read_job_materials": "Read-only access to JobMaterials Endpoint",
"manage_job_materials": "Full access to JobMaterials Endpoint",
"read_job_categories": "Read-only access to Categories Endpoint",
"manage_job_categories": "Full access to Categories Endpoint",
"read_job_queues": "Read-only access to Job Queues Endpoint",
"manage_job_queues": "Full access to Job Queues Endpoint",
"read_tasks": "Read-only access to Tasks Endpoint",
"manage_tasks": "Full access to Tasks Endpoint",
"read_schedule": "Read-only access to JobActivity Endpoint",
"manage_schedule": "Full access to JobActivity Endpoint",
"read_inventory": "Read-only access to Materials Endpoint",
"manage_inventory": "Full access to Materials Endpoint",
"read_job_notes": "Read-only access to job notes",
"publish_job_notes": "Ability to add new job notes",
"read_job_photos": "Read-only access to job photos",
"publish_job_photos": "Ability to add new job photos",
"read_attachments": "Read-only access to Attachments Endpoint",
"manage_attachments": "Full access to Attachments Endpoint",
"read_inbox": "Read-only access to inbox messages",
"read_messages": "Read-only access to staff messages",
"manage_notifications": "Ability to read notifications and mark as read",
"manage_templates": "Full-access to email, sms and document templates",
"manage_badges": "Full-access to create/modify job badges",
"read_assets": "Read-only access to Assets Endpoint",
"manage_assets": "Full access to Assets Endpoint",
"read_knowledge_base": "Read-only access to Knowledge Base Endpoint",
"manage_knowledge_base": "Full access to Knowledge Base Endpoint"
}
}
}
}
},
"schemas": {
"Error": {
"type": "object",
"properties": {
"errorCode": {
"type": "number",
"format": "int32",
"example": "1000"
},
"message": {
"type": "string",
"example": "An error occurred completing your request"
}
}
},
"RateLimitError": {
"type": "object",
"properties": {
"errorCode": {
"type": "number",
"format": "int32",
"example": "429"
},
"message": {
"type": "string",
"example": "Number of allowed API requests per minute exceeded"
}
}
},
"AuthenticationError": {
"type": "object",
"properties": {
"errorCode": {
"type": "number",
"format": "int32",
"example": "401"
},
"message": {
"type": "string",
"example": "Authentication failed. Please check your API key or OAuth token."
}
}
},
"ForbiddenError": {
"type": "object",
"properties": {
"errorCode": {
"type": "number",
"format": "int32",
"example": "403"
},
"message": {
"type": "string",
"example": "Access forbidden. You don't have permission to access this resource."
}
}
},
"FormResponse": {
"type": "object",
"properties": {
"form_uuid": {
"description": "UUID of the form used to generate this form response. Links to a specific form in the system that defines the fields to be gathered.",
"format": "uuid",
"example": "123e4567-8f59-4273-9bb2-23f94e3071eb",
"type": "string"
},
"staff_uuid": {
"description": "UUID of the staff member who completed this FormResponse.",
"format": "uuid",
"example": "123e4567-068f-433c-a2e6-23f94e815bdb",
"type": "string"
},
"regarding_object": {
"description": "The object type that this form response is associated with. Common values include 'job', 'asset', or 'company'. Works in conjunction with regarding_object_uuid to link this form response to a specific record in the system.",
"type": "string"
},
"regarding_object_uuid": {
"description": "UUID of the specific record this form response is linked to. For example, if regarding_object is 'job', this will be the UUID of the specific job. This creates a relationship between the form response and the object it refers to.",
"format": "uuid",
"example": "123e4567-41ee-4ea3-8ca2-23f94fe918eb",
"type": "string"
},
"field_data": {
"description": "JSON array of form answers captured at submission time.",
"type": "string"
},
"timestamp": {
"description": "Date and time when the form was submitted/completed. Used for sorting and displaying form responses chronologically. Format is YYYY-MM-DD HH:MM:SS in UTC timezone.",
"type": "string",
"example": "2026-03-01 12:00:00"
},
"form_by_staff_uuid": {
"description": "UUID of the staff member who completed or submitted this form. Identifies which user filled out the form. Used for tracking form submission history and staff accountability.",
"format": "uuid",
"example": "123e4567-0230-4fe1-83c9-23f94429362b",
"type": "string"
},
"document_attachment_uuid": {
"description": "UUID of the document attachment generated from this form response. When a form is completed, it can generate a PDF document which is stored as an attachment. This field links to that generated document attachment.",
"format": "uuid",
"example": "123e4567-ff79-499a-8ef4-23f94a981a6b",
"type": "string"
},
"asset_uuid": {
"description": "UUID of the Asset this form response is related to. Used when the FormResponsepertains to a specific asset, such as equipment inspections, maintenance checklists, or asset condition reports.",
"format": "uuid",
"example": "123e4567-fc2c-4a3d-a278-23f942d7f62b",
"type": "string"
},
"uuid": {
"format": "uuid",
"description": "Unique identifier for this record",
"example": "123e4567-faa3-4649-8ea5-23f94011885b",
"type": "string"
},
"active": {
"enum": [
0,
1
],
"type": "integer",
"default": 1,
"description": "Record active/deleted flag. Valid values are [0,1]"
},
"edit_date": {
"example": "2026-03-01 12:00:00",
"readOnly": true,
"description": "Timestamp at which record was last modified"
}
}
}
}
},
"x-speakeasy-retries": {
"strategy": "backoff",
"backoff": {
"initialInterval": 500,
"maxInterval": 60000,
"maxElapsedTime": 3600000,
"exponent": 1.5
},
"statusCodes": [
"5XX",
"429"
],
"retryConnectionErrors": true
},
"tags": [
{
"name": "Form Responses",
"description": "Operations related to Form Responses"
}
]
}
```
+440
View File
@@ -0,0 +1,440 @@
# List all Job Allocations
#### Filtering
This endpoint supports result filtering. For more information on how to filter this request, [go here](/docs/filtering).
#### OAuth Scope
This endpoint requires the following OAuth scope **read_schedule**.
# OpenAPI definition
```json
{
"openapi": "3.1.0",
"info": {
"title": "ServiceM8 API",
"description": "Move your app forward with the ServiceM8 API\n\n\n\n## Limits and Throttling\nTo ensure continuous quality of service, API usage can be subject to throttling. The throttle will be applied once an API consumer reaches a certain \nthreshold in terms of a maximum number of requests per minute. Most clients will never hit this threshold, but those that do, will get met by a \nHTTP 429 Too Many Requests response code. \n \nThere is a limit of 180 requests per minute, if you reach this you will receive a HTTP 429 with a text body of \"Number of allowed API requests per minute exceeded\".\nThere is a limit of 20000 requests per day, if you reach this you will receive a HTTP 429 with a text body of \"Number of allowed API requests per day exceeded\".\n\nWe encourage all API developers to anticipate this error, and take appropriate measures like e.g. using a cached value from a previous call, or passing on a message to the end user that gets subjected to this behaviour (if any).\n\nLimits are per Addon per account.\n",
"termsOfService": "https://www.servicem8.com/terms-of-service",
"version": "1.0.0"
},
"servers": [
{
"url": "https://api.servicem8.com/api_1.0"
}
],
"security": [
{
"apiKey": []
},
{
"oauth2": []
}
],
"paths": {
"/joballocation.json": {
"get": {
"tags": [
"Job Allocations"
],
"operationId": "listJobAllocations",
"summary": "List all Job Allocations",
"description": "\n\t\t\t\n#### Filtering\nThis endpoint supports result filtering. For more information on how to filter this request, [go here](/docs/filtering).\n\t\t\t\n\t\t\t\n#### OAuth Scope\nThis endpoint requires the following OAuth scope **read_schedule**.\n\n\t\t\t",
"security": [
{
"apiKey": []
},
{
"oauth2": [
"read_schedule"
]
}
],
"responses": {
"200": {
"description": "An array of Job Allocations",
"content": {
"application/json": {
"schema": {
"type": "array",
"items": {
"$ref": "#/components/schemas/JobAllocation"
}
},
"examples": {
"success": {
"value": [
{
"uuid": "123e4567-aa11-4415-ac38-23f94bc904db",
"active": 1,
"edit_date": "2026-03-01 12:00:00",
"job_uuid": "123e4567-ef33-4570-9929-23f945326d1b",
"queue_uuid": "123e4567-e495-4a8d-a253-23f9408ba0db",
"staff_uuid": "123e4567-e0d8-481f-b4bb-23f944e9c86b",
"allocation_date": "2026-03-01 12:00:00",
"allocation_window_uuid": "123e4567-0f0c-48b9-8570-23f94fcc00db",
"allocated_by_staff_uuid": "123e4567-ab70-4071-8066-23f94537066b",
"allocated_timestamp": "2026-03-01 12:00:00",
"expiry_timestamp": "2026-03-01 12:00:00",
"read_timestamp": "2026-03-01 12:00:00",
"completion_timestamp": "2026-03-01 12:00:00",
"estimated_duration": "string",
"revised_duration": "string",
"sort_priority": "string",
"requires_acceptance": "string",
"acceptance_status": "string",
"acceptance_timestamp": "2026-03-01 12:00:00"
}
]
}
}
}
}
},
"400": {
"description": "Bad Request - The request is malformed or contains invalid parameters",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Error"
},
"examples": {
"badRequest": {
"value": {
"errorCode": "1000",
"message": "An error occurred completing your request"
}
}
}
}
}
},
"401": {
"description": "Unauthorized - Authentication credentials are missing or invalid",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/AuthenticationError"
},
"examples": {
"unauthorized": {
"value": {
"errorCode": "401",
"message": "Authentication failed. Please check your API key or OAuth token."
}
}
}
}
}
},
"403": {
"description": "Forbidden - You don't have permission to access this resource",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ForbiddenError"
},
"examples": {
"forbidden": {
"value": {
"errorCode": "403",
"message": "Access forbidden. You don't have permission to access this resource."
}
}
}
}
}
},
"429": {
"description": "Too Many Requests - You have exceeded the rate limit",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/RateLimitError"
},
"examples": {
"rateLimitMinute": {
"value": {
"errorCode": 429,
"message": "Number of allowed API requests per minute exceeded"
}
},
"rateLimitDay": {
"value": {
"errorCode": 429,
"message": "Number of allowed API requests per day exceeded"
}
}
}
}
}
},
"500": {
"description": "Internal Server Error - An unexpected error occurred on the server",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Error"
},
"examples": {
"serverError": {
"value": {
"errorCode": 500,
"message": "An unexpected error occurred. Please try again later."
}
}
}
}
}
}
}
}
}
},
"components": {
"securitySchemes": {
"apiKey": {
"type": "apiKey",
"name": "X-Api-Key",
"in": "header"
},
"oauth2": {
"type": "oauth2",
"flows": {
"authorizationCode": {
"authorizationUrl": "https://api.servicem8.com/oauth/authorize",
"tokenUrl": "https://api.servicem8.com/oauth/access_token",
"scopes": {
"staff_locations": "Access to real-time GPS information about staff",
"staff_activity": "Access to clock on, lunch break and clock off information about staff",
"publish_sms": "Access to send SMS messages to customers and/or staff on your behalf. Note sending SMS messages will incur account charges.",
"publish_email": "Access to send Email messages to customers and/or staff on your behalf",
"vendor": "Access to basic account information",
"vendor_logo": "Access to account logo",
"vendor_email": "Access to account holder email address",
"read_locations": "Read-only access to Location Endpoint",
"manage_locations": "Full access to Location Endpoint",
"read_staff": "Read-only access to Staff Endpoint",
"manage_staff": "Full access to Staff Endpoint",
"read_customers": "Read-only access to Company Endpoint",
"manage_customers": "Full access to Company Endpoint",
"read_customer_contacts": "Read-only access to CompanyContact Endpoint",
"manage_customer_contacts": "Full access to CompanyContact Endpoint",
"read_jobs": "Read-only access to Job Endpoint",
"manage_jobs": "Full access to Job Endpoint",
"create_jobs": "Ability to create jobs on behalf of account. Note creating jobs may incur account charges.",
"read_job_contacts": "Read-only access to JobContact Endpoint",
"manage_job_contacts": "Full access to JobContact Endpoint",
"read_job_materials": "Read-only access to JobMaterials Endpoint",
"manage_job_materials": "Full access to JobMaterials Endpoint",
"read_job_categories": "Read-only access to Categories Endpoint",
"manage_job_categories": "Full access to Categories Endpoint",
"read_job_queues": "Read-only access to Job Queues Endpoint",
"manage_job_queues": "Full access to Job Queues Endpoint",
"read_tasks": "Read-only access to Tasks Endpoint",
"manage_tasks": "Full access to Tasks Endpoint",
"read_schedule": "Read-only access to JobActivity Endpoint",
"manage_schedule": "Full access to JobActivity Endpoint",
"read_inventory": "Read-only access to Materials Endpoint",
"manage_inventory": "Full access to Materials Endpoint",
"read_job_notes": "Read-only access to job notes",
"publish_job_notes": "Ability to add new job notes",
"read_job_photos": "Read-only access to job photos",
"publish_job_photos": "Ability to add new job photos",
"read_attachments": "Read-only access to Attachments Endpoint",
"manage_attachments": "Full access to Attachments Endpoint",
"read_inbox": "Read-only access to inbox messages",
"read_messages": "Read-only access to staff messages",
"manage_notifications": "Ability to read notifications and mark as read",
"manage_templates": "Full-access to email, sms and document templates",
"manage_badges": "Full-access to create/modify job badges",
"read_assets": "Read-only access to Assets Endpoint",
"manage_assets": "Full access to Assets Endpoint",
"read_knowledge_base": "Read-only access to Knowledge Base Endpoint",
"manage_knowledge_base": "Full access to Knowledge Base Endpoint"
}
}
}
}
},
"schemas": {
"Error": {
"type": "object",
"properties": {
"errorCode": {
"type": "number",
"format": "int32",
"example": "1000"
},
"message": {
"type": "string",
"example": "An error occurred completing your request"
}
}
},
"RateLimitError": {
"type": "object",
"properties": {
"errorCode": {
"type": "number",
"format": "int32",
"example": "429"
},
"message": {
"type": "string",
"example": "Number of allowed API requests per minute exceeded"
}
}
},
"AuthenticationError": {
"type": "object",
"properties": {
"errorCode": {
"type": "number",
"format": "int32",
"example": "401"
},
"message": {
"type": "string",
"example": "Authentication failed. Please check your API key or OAuth token."
}
}
},
"ForbiddenError": {
"type": "object",
"properties": {
"errorCode": {
"type": "number",
"format": "int32",
"example": "403"
},
"message": {
"type": "string",
"example": "Access forbidden. You don't have permission to access this resource."
}
}
},
"JobAllocation": {
"type": "object",
"properties": {
"job_uuid": {
"description": "The UUID of the job that this allocation relates to.",
"format": "uuid",
"example": "123e4567-796e-43d3-89b8-23f942b7f0ab",
"type": "string"
},
"queue_uuid": {
"description": "DEPRECATED"
},
"staff_uuid": {
"description": "The UUID of the staff member this job is allocated to.",
"format": "uuid",
"example": "123e4567-4525-400b-ab3e-23f94473c62b",
"type": "string"
},
"allocation_date": {
"description": "The minimum start date for a job allocation to be completed by a staff member. Setting this date will ensure the job allocation appears in the future on staff schedules.",
"example": "2026-03-01 12:00:00",
"type": "string"
},
"allocation_window_uuid": {
"description": "The UUID of the allocation window that defines when the job should be completed (e.g. Urgent, Early Morning, During Business Hours).",
"format": "uuid",
"example": "123e4567-7fd1-45f3-8068-23f945565e9b",
"type": "string"
},
"allocated_by_staff_uuid": {
"description": "The UUID of the staff member who allocated the job.",
"format": "uuid",
"example": "123e4567-c873-4cca-91e7-23f943de293b",
"type": "string"
},
"allocated_timestamp": {
"description": "The timestamp when the job was allocated.",
"example": "2026-03-01 12:00:00",
"type": "string"
},
"expiry_timestamp": {
"description": "The timestamp when the job allocation expires.",
"example": "2026-03-01 12:00:00",
"type": "string"
},
"read_timestamp": {
"description": "The timestamp when the job allocation was read by the staff member.",
"example": "2026-03-01 12:00:00",
"type": "string"
},
"completion_timestamp": {
"description": "The timestamp when the job allocation was marked as completed.",
"example": "2026-03-01 12:00:00",
"type": "string"
},
"estimated_duration": {
"description": "DEPRECATED"
},
"revised_duration": {
"description": "DEPRECATED"
},
"sort_priority": {
"description": "The sort priority for displaying this job allocation.",
"type": "string"
},
"requires_acceptance": {
"description": "DEPRECATED"
},
"acceptance_status": {
"description": "DEPRECATED"
},
"acceptance_timestamp": {
"description": "DEPRECATED"
},
"uuid": {
"format": "uuid",
"description": "Unique identifier for this record",
"example": "123e4567-8160-4f2d-951a-23f94c5382cb",
"type": "string"
},
"active": {
"enum": [
0,
1
],
"type": "integer",
"default": 1,
"description": "Record active/deleted flag. Valid values are [0,1]"
},
"edit_date": {
"example": "2026-03-01 12:00:00",
"readOnly": true,
"description": "Timestamp at which record was last modified"
}
}
}
}
},
"x-speakeasy-retries": {
"strategy": "backoff",
"backoff": {
"initialInterval": 500,
"maxInterval": 60000,
"maxElapsedTime": 3600000,
"exponent": 1.5
},
"statusCodes": [
"5XX",
"429"
],
"retryConnectionErrors": true
},
"tags": [
{
"name": "Job Allocations",
"description": "Operations related to Job Allocations"
}
]
}
```
+468
View File
@@ -0,0 +1,468 @@
# List all Staff Members
#### Filtering
This endpoint supports result filtering. For more information on how to filter this request, [go here](/docs/filtering).
#### OAuth Scope
This endpoint requires the following OAuth scope **read_staff**.
# OpenAPI definition
```json
{
"openapi": "3.1.0",
"info": {
"title": "ServiceM8 API",
"description": "Move your app forward with the ServiceM8 API\n\n\n\n## Limits and Throttling\nTo ensure continuous quality of service, API usage can be subject to throttling. The throttle will be applied once an API consumer reaches a certain \nthreshold in terms of a maximum number of requests per minute. Most clients will never hit this threshold, but those that do, will get met by a \nHTTP 429 Too Many Requests response code. \n \nThere is a limit of 180 requests per minute, if you reach this you will receive a HTTP 429 with a text body of \"Number of allowed API requests per minute exceeded\".\nThere is a limit of 20000 requests per day, if you reach this you will receive a HTTP 429 with a text body of \"Number of allowed API requests per day exceeded\".\n\nWe encourage all API developers to anticipate this error, and take appropriate measures like e.g. using a cached value from a previous call, or passing on a message to the end user that gets subjected to this behaviour (if any).\n\nLimits are per Addon per account.\n",
"termsOfService": "https://www.servicem8.com/terms-of-service",
"version": "1.0.0"
},
"servers": [
{
"url": "https://api.servicem8.com/api_1.0"
}
],
"security": [
{
"apiKey": []
},
{
"oauth2": []
}
],
"paths": {
"/staff.json": {
"get": {
"tags": [
"Staff Members"
],
"operationId": "listStaffMembers",
"summary": "List all Staff Members",
"description": "\n\t\t\t\n#### Filtering\nThis endpoint supports result filtering. For more information on how to filter this request, [go here](/docs/filtering).\n\t\t\t\n\t\t\t\n#### OAuth Scope\nThis endpoint requires the following OAuth scope **read_staff**.\n\n\t\t\t",
"security": [
{
"apiKey": []
},
{
"oauth2": [
"read_staff"
]
}
],
"responses": {
"200": {
"description": "An array of Staff Members",
"content": {
"application/json": {
"schema": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Staff"
}
},
"examples": {
"success": {
"value": [
{
"first": "string",
"last": "string",
"email": "string",
"mobile": "string",
"lng": "number",
"lat": "number",
"geo_timestamp": "2026-03-01 12:00:00",
"job_title": "string",
"navigating_to_job_uuid": "123e4567-ed86-4242-b22f-23f949021d4b",
"navigating_timestamp": "2026-03-01 12:00:00",
"navigating_expiry_timestamp": "2026-03-01 12:00:00",
"color": "string",
"custom_icon_url": "string",
"status_message": "string",
"status_message_timestamp": "2026-03-01 12:00:00",
"hide_from_schedule": "string",
"uuid": "123e4567-ce52-4b1f-8564-23f94e82d3cb",
"active": 1,
"edit_date": "2026-03-01 12:00:00",
"can_receive_push_notification": "string",
"security_role_uuid": "123e4567-873a-46dc-b21b-23f9497ac49b",
"labour_material_uuid": "123e4567-bd3b-4d18-af43-23f941752a7b"
}
]
}
}
}
}
},
"400": {
"description": "Bad Request - The request is malformed or contains invalid parameters",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Error"
},
"examples": {
"badRequest": {
"value": {
"errorCode": "1000",
"message": "An error occurred completing your request"
}
}
}
}
}
},
"401": {
"description": "Unauthorized - Authentication credentials are missing or invalid",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/AuthenticationError"
},
"examples": {
"unauthorized": {
"value": {
"errorCode": "401",
"message": "Authentication failed. Please check your API key or OAuth token."
}
}
}
}
}
},
"403": {
"description": "Forbidden - You don't have permission to access this resource",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ForbiddenError"
},
"examples": {
"forbidden": {
"value": {
"errorCode": "403",
"message": "Access forbidden. You don't have permission to access this resource."
}
}
}
}
}
},
"429": {
"description": "Too Many Requests - You have exceeded the rate limit",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/RateLimitError"
},
"examples": {
"rateLimitMinute": {
"value": {
"errorCode": 429,
"message": "Number of allowed API requests per minute exceeded"
}
},
"rateLimitDay": {
"value": {
"errorCode": 429,
"message": "Number of allowed API requests per day exceeded"
}
}
}
}
}
},
"500": {
"description": "Internal Server Error - An unexpected error occurred on the server",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Error"
},
"examples": {
"serverError": {
"value": {
"errorCode": 500,
"message": "An unexpected error occurred. Please try again later."
}
}
}
}
}
}
}
}
}
},
"components": {
"securitySchemes": {
"apiKey": {
"type": "apiKey",
"name": "X-Api-Key",
"in": "header"
},
"oauth2": {
"type": "oauth2",
"flows": {
"authorizationCode": {
"authorizationUrl": "https://api.servicem8.com/oauth/authorize",
"tokenUrl": "https://api.servicem8.com/oauth/access_token",
"scopes": {
"staff_locations": "Access to real-time GPS information about staff",
"staff_activity": "Access to clock on, lunch break and clock off information about staff",
"publish_sms": "Access to send SMS messages to customers and/or staff on your behalf. Note sending SMS messages will incur account charges.",
"publish_email": "Access to send Email messages to customers and/or staff on your behalf",
"vendor": "Access to basic account information",
"vendor_logo": "Access to account logo",
"vendor_email": "Access to account holder email address",
"read_locations": "Read-only access to Location Endpoint",
"manage_locations": "Full access to Location Endpoint",
"read_staff": "Read-only access to Staff Endpoint",
"manage_staff": "Full access to Staff Endpoint",
"read_customers": "Read-only access to Company Endpoint",
"manage_customers": "Full access to Company Endpoint",
"read_customer_contacts": "Read-only access to CompanyContact Endpoint",
"manage_customer_contacts": "Full access to CompanyContact Endpoint",
"read_jobs": "Read-only access to Job Endpoint",
"manage_jobs": "Full access to Job Endpoint",
"create_jobs": "Ability to create jobs on behalf of account. Note creating jobs may incur account charges.",
"read_job_contacts": "Read-only access to JobContact Endpoint",
"manage_job_contacts": "Full access to JobContact Endpoint",
"read_job_materials": "Read-only access to JobMaterials Endpoint",
"manage_job_materials": "Full access to JobMaterials Endpoint",
"read_job_categories": "Read-only access to Categories Endpoint",
"manage_job_categories": "Full access to Categories Endpoint",
"read_job_queues": "Read-only access to Job Queues Endpoint",
"manage_job_queues": "Full access to Job Queues Endpoint",
"read_tasks": "Read-only access to Tasks Endpoint",
"manage_tasks": "Full access to Tasks Endpoint",
"read_schedule": "Read-only access to JobActivity Endpoint",
"manage_schedule": "Full access to JobActivity Endpoint",
"read_inventory": "Read-only access to Materials Endpoint",
"manage_inventory": "Full access to Materials Endpoint",
"read_job_notes": "Read-only access to job notes",
"publish_job_notes": "Ability to add new job notes",
"read_job_photos": "Read-only access to job photos",
"publish_job_photos": "Ability to add new job photos",
"read_attachments": "Read-only access to Attachments Endpoint",
"manage_attachments": "Full access to Attachments Endpoint",
"read_inbox": "Read-only access to inbox messages",
"read_messages": "Read-only access to staff messages",
"manage_notifications": "Ability to read notifications and mark as read",
"manage_templates": "Full-access to email, sms and document templates",
"manage_badges": "Full-access to create/modify job badges",
"read_assets": "Read-only access to Assets Endpoint",
"manage_assets": "Full access to Assets Endpoint",
"read_knowledge_base": "Read-only access to Knowledge Base Endpoint",
"manage_knowledge_base": "Full access to Knowledge Base Endpoint"
}
}
}
}
},
"schemas": {
"Error": {
"type": "object",
"properties": {
"errorCode": {
"type": "number",
"format": "int32",
"example": "1000"
},
"message": {
"type": "string",
"example": "An error occurred completing your request"
}
}
},
"RateLimitError": {
"type": "object",
"properties": {
"errorCode": {
"type": "number",
"format": "int32",
"example": "429"
},
"message": {
"type": "string",
"example": "Number of allowed API requests per minute exceeded"
}
}
},
"AuthenticationError": {
"type": "object",
"properties": {
"errorCode": {
"type": "number",
"format": "int32",
"example": "401"
},
"message": {
"type": "string",
"example": "Authentication failed. Please check your API key or OAuth token."
}
}
},
"ForbiddenError": {
"type": "object",
"properties": {
"errorCode": {
"type": "number",
"format": "int32",
"example": "403"
},
"message": {
"type": "string",
"example": "Access forbidden. You don't have permission to access this resource."
}
}
},
"Staff": {
"type": "object",
"properties": {
"first": {
"description": "Staff First Name",
"type": "string",
"maxLength": 30
},
"last": {
"description": "Staff Last Name",
"type": "string",
"maxLength": 30
},
"email": {
"description": "Staff Email Address. This is also your login name.",
"format": "email",
"type": "string"
},
"mobile": {
"description": "Mobile phone number of the staff member. Used for SMS communications and identification when calling.",
"type": "string"
},
"lng": {
"description": "Longitude coordinate of the staff member's current or last known location. Used for tracking staff locations and calculating routes and travel distances.",
"type": "number",
"format": "float"
},
"lat": {
"description": "Latitude coordinate of the staff member's current or last known location. Used for tracking staff locations and calculating routes and travel distances.",
"type": "number",
"format": "float"
},
"geo_timestamp": {
"description": "The date and time when the staff member's geographic location (lat/lng) was last updated. Format is YYYY-MM-DD HH:MM:SS. Used to determine how recent the location data is.",
"example": "2026-03-01 12:00:00",
"type": "string"
},
"job_title": {
"description": "The staff member's job title or role within the organization. Used for organizational purposes and displayed in various places throughout the system.",
"type": "string"
},
"navigating_to_job_uuid": {
"description": "UUID of the job the staff member is currently navigating to. Used to track which job a staff member is traveling toward.",
"format": "uuid",
"example": "123e4567-11b0-4089-8ab2-23f9484d54ab",
"type": "string"
},
"navigating_timestamp": {
"description": "The date and time when the staff member started navigating to a job. Format is YYYY-MM-DD HH:MM:SS. Used to track when navigation began.",
"example": "2026-03-01 12:00:00",
"type": "string"
},
"navigating_expiry_timestamp": {
"description": "The date and time when navigation to a job is expected to complete or expire. Format is YYYY-MM-DD HH:MM:SS. Used to determine if navigation is still active.",
"example": "2026-03-01 12:00:00",
"type": "string"
},
"color": {
"description": "The color assigned to this staff member, represented as a hex color code. Used for visual identification in the schedule, dispatch board, and other interfaces.",
"type": "string"
},
"custom_icon_url": {
"description": "DEPRECATED"
},
"status_message": {
"description": "Short message summarising the staff's current status.",
"type": "string"
},
"status_message_timestamp": {
"description": "The date and time when the staff member's status message was last updated. Format is YYYY-MM-DD HH:MM:SS. Used to determine how recent the status message is.",
"example": "2026-03-01 12:00:00",
"type": "string"
},
"hide_from_schedule": {
"description": "Boolean flag controlling whether this staff member appears in the schedule view. When true (1), the staff member is hidden from the schedule. When false (0), they appear normally in scheduling interfaces.. Valid values are [0,1]",
"type": "integer",
"enum": [
0,
1
]
},
"uuid": {
"format": "uuid",
"description": "Unique identifier for this record",
"example": "123e4567-9fa9-46c2-9e1d-23f94b3418fb",
"type": "string"
},
"active": {
"enum": [
0,
1
],
"type": "integer",
"default": 1,
"description": "Record active/deleted flag. Valid values are [0,1]"
},
"edit_date": {
"example": "2026-03-01 12:00:00",
"readOnly": true,
"description": "Timestamp at which record was last modified"
},
"can_receive_push_notification": {
"type": "string"
},
"security_role_uuid": {
"format": "uuid",
"example": "123e4567-4ff4-4f95-8521-23f9452e9a2b",
"type": "string"
},
"labour_material_uuid": {
"format": "uuid",
"example": "123e4567-5c47-4d54-b8e7-23f94882c18b",
"type": "string"
}
},
"required": [
"first",
"last",
"email"
]
}
}
},
"x-speakeasy-retries": {
"strategy": "backoff",
"backoff": {
"initialInterval": 500,
"maxInterval": 60000,
"maxElapsedTime": 3600000,
"exponent": 1.5
},
"statusCodes": [
"5XX",
"429"
],
"retryConnectionErrors": true
},
"tags": [
{
"name": "Staff Members",
"description": "Operations related to Staff Members"
}
]
}
```
+17
View File
@@ -0,0 +1,17 @@
[Unit]
Description=ServiceM8 FastAPI Inspector
After=network.target
[Service]
User=michael
Group=michael
WorkingDirectory=/opt/webhooks
Environment="WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db"
Environment="WEBHOOK_INSPECTOR_HOST=0.0.0.0"
Environment="WEBHOOK_INSPECTOR_PORT=18355"
ExecStart=/opt/webhooks/bin/uvicorn servicem8_inspector:app --host 0.0.0.0 --port 18355
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
+17
View File
@@ -0,0 +1,17 @@
[Unit]
Description=ServiceM8 FastAPI Webhook Receiver
After=network.target
[Service]
User=michael
Group=michael
WorkingDirectory=/opt/webhooks
Environment="WEBHOOK_HOST=0.0.0.0"
Environment="WEBHOOK_PORT=18354"
Environment="WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db"
ExecStart=/opt/webhooks/bin/uvicorn servicem8_webhook_receiver:app --host 0.0.0.0 --port 18354
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
+186
View File
@@ -0,0 +1,186 @@
#!/usr/bin/env bash
set -euo pipefail
# Poll ServiceM8 form responses, parse Quote Template responses, then apply any
# newly/unapplied parsed quote responses to ServiceM8 jobMaterials.
#
# Default window: last 24 hours from script start, in local system time.
# Override with:
# --since 'YYYY-MM-DD HH:MM:SS'
# --hours 48
#
# By default this wrapper applies unapplied parsed responses. Use --dry-run to
# run the poll and preview each pending apply without writing to ServiceM8.
# The apply script still refuses duplicate applies unless --force is explicitly
# passed through. It also checks ServiceM8 for existing jobMaterial rows on
# the target job and blocks the apply unless --force-remote-existing is passed.
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
POLL_SCRIPT="$SCRIPT_DIR/poll_form_responses_since.py"
APPLY_SCRIPT="$SCRIPT_DIR/apply_polled_quote_template_jobmaterials.py"
DB_PATH="${WEBHOOK_POLL_DB_PATH:-$SCRIPT_DIR/servicem8_formresponse_poll.db}"
LOG_DIR="${WEBHOOK_RUN_LOG_DIR:-$SCRIPT_DIR/logs}"
QUOTE_TEMPLATE_FORM_UUID="${SERVICEM8_QUOTE_TEMPLATE_FORM_UUID:-3621b6be-1d19-4756-9ab4-9d5e4120f6d9}"
SINCE=""
HOURS="24"
FORCE="0"
FORCE_REMOTE_EXISTING="0"
RECHECK_REMOTE_EXISTING="0"
DRY_RUN="0"
usage() {
cat <<EOF
Usage: $0 [--since 'YYYY-MM-DD HH:MM:SS'] [--hours N] [--dry-run] [--force] [--force-remote-existing] [--recheck-remote-existing]
Examples:
$0
$0 --hours 48
$0 --since '2026-05-04 08:00:00'
$0 --dry-run --hours 48
This will:
1. Poll /formresponse.json using timestamp gt SINCE
2. Store/parse Quote Template responses into $DB_PATH
3. Apply parsed responses that do not already have generated materials recorded
With --dry-run, step 3 previews the ServiceM8 jobMaterial payloads only; it does
not write to ServiceM8 or mark responses as applied.
Safety:
--force keeps the existing local duplicate override behaviour.
--force-remote-existing allows creating rows even when the remote ServiceM8
job already has jobMaterial rows; the incident is still recorded.
--recheck-remote-existing revisits rows previously blocked because remote
jobMaterial rows existed. If the remote rows are gone, the apply can proceed.
EOF
}
while [[ $# -gt 0 ]]; do
case "$1" in
--since)
SINCE="${2:-}"
shift 2
;;
--hours)
HOURS="${2:-}"
shift 2
;;
--force)
FORCE="1"
shift
;;
--force-remote-existing)
FORCE_REMOTE_EXISTING="1"
shift
;;
--recheck-remote-existing)
RECHECK_REMOTE_EXISTING="1"
shift
;;
--dry-run)
DRY_RUN="1"
shift
;;
-h|--help)
usage
exit 0
;;
*)
echo "Unknown argument: $1" >&2
usage >&2
exit 2
;;
esac
done
if [[ -z "$SINCE" ]]; then
SINCE="$(date -d "-${HOURS} hours" '+%Y-%m-%d %H:%M:%S')"
fi
mkdir -p "$LOG_DIR"
RUN_ID="$(date '+%Y%m%d-%H%M%S')"
LOG_FILE="$LOG_DIR/poll-and-apply-$RUN_ID.log"
exec > >(tee -a "$LOG_FILE") 2>&1
echo "== ServiceM8 Quote Template poll/apply run =="
echo "Started: $(date --iso-8601=seconds)"
echo "Since: $SINCE"
echo "Mode: $([[ "$DRY_RUN" == "1" ]] && echo "dry-run" || echo "apply")"
echo "Remote existing: $([[ "$FORCE_REMOTE_EXISTING" == "1" ]] && echo "force" || ([[ "$RECHECK_REMOTE_EXISTING" == "1" ]] && echo "recheck" || echo "block"))"
echo "DB: $DB_PATH"
echo "Log: $LOG_FILE"
echo
echo "== Polling form responses =="
"$POLL_SCRIPT" --since "$SINCE" --summary-limit 20
echo
echo "== Finding unapplied parsed Quote Template responses =="
mapfile -t FORM_RESPONSE_UUIDS < <(
python3 - "$DB_PATH" "$QUOTE_TEMPLATE_FORM_UUID" "$RECHECK_REMOTE_EXISTING" "$FORCE_REMOTE_EXISTING" <<'PY'
import sqlite3
import sys
poll_db, quote_form_uuid, recheck_remote_existing, force_remote_existing = sys.argv[1:5]
include_remote_blocked = recheck_remote_existing == "1" or force_remote_existing == "1"
status_clause = "coalesce(q.process_status, '') != 'applied'"
if not include_remote_blocked:
status_clause += " and coalesce(q.process_status, '') != 'blocked_remote_existing'"
conn = sqlite3.connect(poll_db)
conn.row_factory = sqlite3.Row
rows = conn.execute(
f"""
select q.form_response_uuid
from quote_template_form_responses q
left join form_responses_raw r on r.uuid = q.form_response_uuid
where q.form_uuid = ?
and {status_clause}
order by coalesce(r.timestamp, q.discovered_at) asc, q.form_response_uuid asc
""",
(quote_form_uuid,),
).fetchall()
for row in rows:
print(row["form_response_uuid"])
PY
)
if [[ ${#FORM_RESPONSE_UUIDS[@]} -eq 0 ]]; then
echo "No unapplied Quote Template responses found."
echo "Finished: $(date --iso-8601=seconds)"
exit 0
fi
printf 'Found %d unapplied Quote Template response(s):\n' "${#FORM_RESPONSE_UUIDS[@]}"
printf ' - %s\n' "${FORM_RESPONSE_UUIDS[@]}"
echo
if [[ "$DRY_RUN" == "1" ]]; then
echo "== Dry-run preview only; no ServiceM8 writes =="
APPLY_ARGS=(--pretty)
else
echo "== Applying to ServiceM8 =="
APPLY_ARGS=(--apply --pretty)
fi
if [[ "$FORCE" == "1" ]]; then
APPLY_ARGS+=(--force)
fi
if [[ "$FORCE_REMOTE_EXISTING" == "1" ]]; then
APPLY_ARGS+=(--force-remote-existing)
fi
for uuid in "${FORM_RESPONSE_UUIDS[@]}"; do
echo
if [[ "$DRY_RUN" == "1" ]]; then
echo "-- Dry-run form_response_uuid=$uuid --"
else
echo "-- Applying form_response_uuid=$uuid --"
fi
"$APPLY_SCRIPT" --uuid "$uuid" "${APPLY_ARGS[@]}"
done
echo
echo "Finished: $(date --iso-8601=seconds)"
echo "Log: $LOG_FILE"
+660
View File
@@ -0,0 +1,660 @@
#!/usr/bin/env python3
"""Poll ServiceM8 /formresponse.json for responses newer than a timestamp.
This is the safety-net companion to the ServiceM8 form.response_created webhook.
It stores every API hit in a local SQLite DB, then identifies previously unseen
Quote Template form responses and parses them into desired jobMaterial rows using
servicem8_quote_template_parser.py. It does not write anything back to ServiceM8.
"""
from __future__ import annotations
import argparse
import importlib.util
import json
import os
import sqlite3
import sys
from contextlib import closing
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Dict, Iterable, List, Optional, Tuple
import requests
from servicem8_quote_template_parser import (
QUOTE_TEMPLATE_FORM_UUID as PARSER_QUOTE_TEMPLATE_FORM_UUID,
parse_quote_template_form_response,
)
SCRIPT_DIR = Path(__file__).resolve().parent
DEFAULT_DB_PATH = SCRIPT_DIR / "servicem8_formresponse_poll.db"
DEFAULT_QUEUE_PATH = SCRIPT_DIR / "quote-template-jobmaterials-poll-queue.jsonl"
DEFAULT_BASE_URL = "https://api.servicem8.com/api_1.0"
DEFAULT_ENDPOINT = "/formresponse.json"
DEFAULT_TIMEOUT_SECONDS = 30
# Current Quote Template form UUID. Kept as a CLI default so we can adjust without
# editing code if ServiceM8 form/template IDs change again.
DEFAULT_QUOTE_TEMPLATE_FORM_UUID = os.getenv(
"SERVICEM8_QUOTE_TEMPLATE_FORM_UUID",
PARSER_QUOTE_TEMPLATE_FORM_UUID,
)
def utc_now() -> str:
return datetime.now(timezone.utc).isoformat()
def load_api_key() -> str:
"""Load API key from env, falling back to the existing local helper script.
Prefer env vars for any new deployment. The fallback keeps this project script
runnable in the current /opt/webhooks setup without copying credentials.
"""
for name in ("SERVICEM8_ACCESS_TOKEN", "SERVICEM8_API_KEY"):
value = os.getenv(name)
if value:
return value
fallback = SCRIPT_DIR / "servicem8-list-webhook-subscriptions-table.py"
if fallback.exists():
spec = importlib.util.spec_from_file_location("servicem8_subs", fallback)
if spec and spec.loader:
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module) # type: ignore[union-attr]
value = getattr(module, "ACCESS_TOKEN", None)
if value:
return str(value)
raise RuntimeError("Missing SERVICEM8_ACCESS_TOKEN or SERVICEM8_API_KEY")
def get_conn(db_path: Path) -> sqlite3.Connection:
conn = sqlite3.connect(db_path)
conn.row_factory = sqlite3.Row
return conn
def init_db(db_path: Path) -> None:
db_path.parent.mkdir(parents=True, exist_ok=True)
with closing(get_conn(db_path)) as conn:
conn.execute(
"""
CREATE TABLE IF NOT EXISTS poll_runs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
started_at TEXT NOT NULL,
finished_at TEXT,
since_value TEXT NOT NULL,
filter_field TEXT NOT NULL,
filter_expression TEXT NOT NULL,
http_status INTEGER,
fetched_count INTEGER DEFAULT 0,
inserted_count INTEGER DEFAULT 0,
updated_count INTEGER DEFAULT 0,
quote_match_count INTEGER DEFAULT 0,
newly_queued_count INTEGER DEFAULT 0,
error TEXT
)
"""
)
conn.execute(
"""
CREATE TABLE IF NOT EXISTS form_responses_raw (
uuid TEXT PRIMARY KEY,
first_seen_at TEXT NOT NULL,
last_seen_at TEXT NOT NULL,
seen_count INTEGER NOT NULL DEFAULT 1,
active INTEGER,
edit_date TEXT,
form_uuid TEXT,
staff_uuid TEXT,
regarding_object TEXT,
regarding_object_uuid TEXT,
timestamp TEXT,
form_by_staff_uuid TEXT,
document_attachment_uuid TEXT,
asset_uuid TEXT,
is_quote_template INTEGER NOT NULL DEFAULT 0,
parse_status TEXT,
parse_error TEXT,
raw_json TEXT NOT NULL
)
"""
)
conn.execute("CREATE INDEX IF NOT EXISTS idx_form_responses_raw_timestamp ON form_responses_raw(timestamp)")
conn.execute("CREATE INDEX IF NOT EXISTS idx_form_responses_raw_edit_date ON form_responses_raw(edit_date)")
conn.execute("CREATE INDEX IF NOT EXISTS idx_form_responses_raw_form_uuid ON form_responses_raw(form_uuid)")
conn.execute("CREATE INDEX IF NOT EXISTS idx_form_responses_raw_quote ON form_responses_raw(is_quote_template, parse_status)")
conn.execute(
"""
CREATE TABLE IF NOT EXISTS quote_template_form_responses (
form_response_uuid TEXT PRIMARY KEY,
discovered_at TEXT NOT NULL,
job_uuid TEXT,
form_uuid TEXT NOT NULL,
author_name TEXT,
description TEXT,
desired_job_materials_json TEXT NOT NULL,
parsed_json TEXT NOT NULL,
queued_at TEXT,
processed_at TEXT,
process_status TEXT NOT NULL DEFAULT 'parsed',
process_error TEXT
)
"""
)
conn.execute(
"""
CREATE TABLE IF NOT EXISTS job_metadata (
job_uuid TEXT PRIMARY KEY,
generated_job_id TEXT,
job_address TEXT,
company_name TEXT,
raw_json TEXT NOT NULL,
first_seen_at TEXT NOT NULL,
last_seen_at TEXT NOT NULL,
source TEXT NOT NULL
)
"""
)
conn.execute("CREATE INDEX IF NOT EXISTS idx_job_metadata_generated_job_id ON job_metadata(generated_job_id)")
conn.commit()
def create_poll_run(conn: sqlite3.Connection, since: str, filter_field: str, filter_expr: str) -> int:
cur = conn.execute(
"""
INSERT INTO poll_runs (started_at, since_value, filter_field, filter_expression)
VALUES (?, ?, ?, ?)
""",
(utc_now(), since, filter_field, filter_expr),
)
conn.commit()
return int(cur.lastrowid)
def finish_poll_run(conn: sqlite3.Connection, run_id: int, **values: Any) -> None:
allowed = {
"finished_at",
"http_status",
"fetched_count",
"inserted_count",
"updated_count",
"quote_match_count",
"newly_queued_count",
"error",
}
values.setdefault("finished_at", utc_now())
pairs = [(k, v) for k, v in values.items() if k in allowed]
if not pairs:
return
set_sql = ", ".join(f"{k} = ?" for k, _ in pairs)
params = [v for _, v in pairs] + [run_id]
conn.execute(f"UPDATE poll_runs SET {set_sql} WHERE id = ?", params)
conn.commit()
def fetch_form_responses(
*,
api_key: str,
base_url: str,
endpoint: str,
filter_field: str,
since: str,
timeout: int,
) -> Tuple[int, List[Dict[str, Any]], str]:
# ServiceM8 filter syntax confirmed against the live API:
# ?$filter=timestamp gt '2026-05-04 00:00:00'
filter_expr = f"{filter_field} gt '{since}'"
url = f"{base_url.rstrip('/')}{endpoint}"
headers = {"X-Api-Key": api_key, "Accept": "application/json"}
response = requests.get(url, headers=headers, params={"$filter": filter_expr}, timeout=timeout)
if not response.ok:
raise RuntimeError(f"HTTP {response.status_code}: {response.text[:1000]}")
data = response.json()
if not isinstance(data, list):
raise RuntimeError(f"Expected list response, got {type(data).__name__}")
return response.status_code, data, filter_expr
def retrieve_job(
*,
api_key: str,
base_url: str,
job_uuid: str,
timeout: int,
) -> Dict[str, Any]:
response = requests.get(
f"{base_url.rstrip('/')}/job/{job_uuid}.json",
headers={"X-Api-Key": api_key, "Accept": "application/json"},
timeout=timeout,
)
if not response.ok:
raise RuntimeError(f"Job retrieve failed for {job_uuid}: HTTP {response.status_code}: {response.text[:1000]}")
data = response.json()
if not isinstance(data, dict):
raise RuntimeError(f"Job retrieve expected object response, got {type(data).__name__}")
return data
def clean_text(value: Any) -> str:
if value is None:
return ""
return str(value).replace("\r\n", "\n").replace("\r", "\n").strip()
def first_text(*values: Any) -> str:
for value in values:
text = clean_text(value)
if text:
return text
return ""
def format_job_address(job: Dict[str, Any]) -> str:
direct = first_text(
job.get("job_address"),
job.get("site_address"),
job.get("address"),
job.get("location_address"),
job.get("billing_address"),
)
if direct:
return direct
parts = [
first_text(job.get("street"), job.get("street_address"), job.get("address_1"), job.get("address1")),
first_text(job.get("suburb"), job.get("city")),
first_text(job.get("state")),
first_text(job.get("postcode"), job.get("postal_code"), job.get("zip")),
]
return " ".join(part for part in parts if part)
def extract_company_name(job: Dict[str, Any]) -> str:
related = job.get("related")
if isinstance(related, dict):
company = related.get("company")
if isinstance(company, dict):
company_name = clean_text(company.get("name"))
if company_name:
return company_name
company = job.get("company")
if isinstance(company, dict):
company_name = clean_text(company.get("name"))
if company_name:
return company_name
return first_text(job.get("company_name"), job.get("customer_name"))
def upsert_job_metadata(conn: sqlite3.Connection, *, job_uuid: str, job: Dict[str, Any], now: str, source: str) -> None:
job_uuid = clean_text(job_uuid or job.get("uuid"))
if not job_uuid:
return
values = (
job_uuid,
clean_text(job.get("generated_job_id")),
format_job_address(job),
extract_company_name(job),
json.dumps(job, ensure_ascii=False, sort_keys=True),
now,
now,
source,
)
conn.execute(
"""
INSERT INTO job_metadata (
job_uuid, generated_job_id, job_address, company_name, raw_json,
first_seen_at, last_seen_at, source
) VALUES (?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(job_uuid) DO UPDATE SET
generated_job_id = excluded.generated_job_id,
job_address = excluded.job_address,
company_name = excluded.company_name,
raw_json = excluded.raw_json,
last_seen_at = excluded.last_seen_at,
source = excluded.source
""",
values,
)
def insert_or_update_raw(
conn: sqlite3.Connection,
row: Dict[str, Any],
*,
quote_template_form_uuid: str,
now: str,
) -> Tuple[bool, bool]:
uuid = str(row.get("uuid") or "").strip()
if not uuid:
raise ValueError("Form response row missing uuid")
is_quote = 1 if str(row.get("form_uuid") or "").strip() == quote_template_form_uuid else 0
existing = conn.execute("SELECT uuid FROM form_responses_raw WHERE uuid = ?", (uuid,)).fetchone()
raw_json = json.dumps(row, ensure_ascii=False, sort_keys=True)
values = (
uuid,
now,
now,
row.get("active"),
row.get("edit_date"),
row.get("form_uuid"),
row.get("staff_uuid"),
row.get("regarding_object"),
row.get("regarding_object_uuid"),
row.get("timestamp"),
row.get("form_by_staff_uuid"),
row.get("document_attachment_uuid"),
row.get("asset_uuid"),
is_quote,
raw_json,
)
if existing:
conn.execute(
"""
UPDATE form_responses_raw
SET last_seen_at = ?,
seen_count = seen_count + 1,
active = ?,
edit_date = ?,
form_uuid = ?,
staff_uuid = ?,
regarding_object = ?,
regarding_object_uuid = ?,
timestamp = ?,
form_by_staff_uuid = ?,
document_attachment_uuid = ?,
asset_uuid = ?,
is_quote_template = ?,
raw_json = ?
WHERE uuid = ?
""",
(
now,
row.get("active"),
row.get("edit_date"),
row.get("form_uuid"),
row.get("staff_uuid"),
row.get("regarding_object"),
row.get("regarding_object_uuid"),
row.get("timestamp"),
row.get("form_by_staff_uuid"),
row.get("document_attachment_uuid"),
row.get("asset_uuid"),
is_quote,
raw_json,
uuid,
),
)
return False, bool(is_quote)
conn.execute(
"""
INSERT INTO form_responses_raw (
uuid, first_seen_at, last_seen_at, active, edit_date, form_uuid,
staff_uuid, regarding_object, regarding_object_uuid, timestamp,
form_by_staff_uuid, document_attachment_uuid, asset_uuid,
is_quote_template, raw_json
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
values,
)
return True, bool(is_quote)
def ensure_queue_writable(queue_path: Path) -> None:
queue_path.parent.mkdir(parents=True, exist_ok=True)
if queue_path.exists():
with queue_path.open("a", encoding="utf-8"):
pass
else:
with queue_path.open("a", encoding="utf-8"):
pass
def queue_record(queue_path: Path, parsed: Dict[str, Any], discovered_at: str) -> None:
record = {
"queued_at": discovered_at,
"source": "formresponse_poll",
"form_uuid": parsed.get("form_uuid", ""),
"form_response_uuid": parsed.get("form_response_uuid", ""),
"job_uuid": parsed.get("job_uuid", ""),
"author_name": parsed.get("author_name", ""),
"description": parsed.get("description", ""),
"desired_job_materials": parsed.get("desired_job_materials", []),
}
with queue_path.open("a", encoding="utf-8") as fh:
fh.write(json.dumps(record, ensure_ascii=False) + "\n")
def parse_and_store_quote_response(
conn: sqlite3.Connection,
row: Dict[str, Any],
*,
queue_path: Path,
write_queue: bool,
discovered_at: str,
) -> bool:
uuid = str(row.get("uuid") or "").strip()
existing = conn.execute(
"SELECT form_response_uuid, parsed_json, queued_at FROM quote_template_form_responses WHERE form_response_uuid = ?",
(uuid,),
).fetchone()
if existing:
if write_queue and not existing["queued_at"]:
parsed = json.loads(existing["parsed_json"])
queue_record(queue_path, parsed, discovered_at)
conn.execute(
"UPDATE quote_template_form_responses SET queued_at = ? WHERE form_response_uuid = ?",
(discovered_at, uuid),
)
return True
return False
try:
parsed = parse_quote_template_form_response({"data": row})
except Exception as exc:
conn.execute(
"UPDATE form_responses_raw SET parse_status = ?, parse_error = ? WHERE uuid = ?",
("error", str(exc), uuid),
)
return False
parsed_json = json.dumps(parsed, ensure_ascii=False, sort_keys=True)
desired_json = json.dumps(parsed.get("desired_job_materials", []), ensure_ascii=False, sort_keys=True)
queued_at = discovered_at if write_queue else None
conn.execute(
"""
INSERT INTO quote_template_form_responses (
form_response_uuid, discovered_at, job_uuid, form_uuid, author_name,
description, desired_job_materials_json, parsed_json, queued_at
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(
uuid,
discovered_at,
parsed.get("job_uuid", ""),
parsed.get("form_uuid", ""),
parsed.get("author_name", ""),
parsed.get("description", ""),
desired_json,
parsed_json,
queued_at,
),
)
conn.execute(
"UPDATE form_responses_raw SET parse_status = ?, parse_error = NULL WHERE uuid = ?",
("parsed", uuid),
)
if write_queue:
queue_record(queue_path, parsed, discovered_at)
return True
def summarize_rows(rows: Iterable[Dict[str, Any]], limit: int) -> List[Dict[str, Any]]:
summary = []
for row in list(rows)[:limit]:
field_data = row.get("field_data")
field_count: Optional[int] = None
if isinstance(field_data, str):
try:
parsed = json.loads(field_data)
if isinstance(parsed, list):
field_count = len(parsed)
except Exception:
field_count = None
summary.append(
{
"uuid": row.get("uuid"),
"timestamp": row.get("timestamp"),
"edit_date": row.get("edit_date"),
"form_uuid": row.get("form_uuid"),
"regarding_object": row.get("regarding_object"),
"regarding_object_uuid": row.get("regarding_object_uuid"),
"field_data_type": type(field_data).__name__,
"field_count": field_count,
}
)
return summary
def main() -> int:
parser = argparse.ArgumentParser(description="Poll ServiceM8 form responses since a timestamp and store unseen Quote Template responses")
parser.add_argument("--since", required=True, help="Timestamp for ServiceM8 filter, e.g. '2026-05-04 10:00:00'")
parser.add_argument("--filter-field", default="timestamp", choices=("timestamp", "edit_date"), help="Field to apply ServiceM8 gt filter to")
parser.add_argument("--db", default=str(DEFAULT_DB_PATH), help="Local SQLite DB for poll results")
parser.add_argument("--queue", default=str(DEFAULT_QUEUE_PATH), help="JSONL queue path for parsed Quote Template jobMaterials")
parser.add_argument("--quote-template-form-uuid", default=DEFAULT_QUOTE_TEMPLATE_FORM_UUID, help="Quote Template form UUID to process")
parser.add_argument("--base-url", default=DEFAULT_BASE_URL)
parser.add_argument("--endpoint", default=DEFAULT_ENDPOINT)
parser.add_argument("--timeout", type=int, default=DEFAULT_TIMEOUT_SECONDS)
parser.add_argument("--no-store", action="store_true", help="Fetch and print summary only; do not write DB/queue")
parser.add_argument("--no-queue", action="store_true", help="Store/parse matches but do not append to JSONL queue")
parser.add_argument("--dump-json", action="store_true", help="Print full fetched ServiceM8 JSON payload")
parser.add_argument("--summary-limit", type=int, default=10, help="Number of rows to include in summary output")
args = parser.parse_args()
db_path = Path(args.db)
queue_path = Path(args.queue)
api_key = load_api_key()
filter_expr = f"{args.filter_field} gt '{args.since}'"
conn: Optional[sqlite3.Connection] = None
run_id: Optional[int] = None
try:
if not args.no_store and not args.no_queue:
# Fail before DB work if the queue cannot be appended to. This avoids
# marking a response as queued when the JSONL append did not happen.
ensure_queue_writable(queue_path)
if not args.no_store:
init_db(db_path)
conn = get_conn(db_path)
run_id = create_poll_run(conn, args.since, args.filter_field, filter_expr)
http_status, rows, filter_expr = fetch_form_responses(
api_key=api_key,
base_url=args.base_url,
endpoint=args.endpoint,
filter_field=args.filter_field,
since=args.since,
timeout=args.timeout,
)
if args.dump_json:
print(json.dumps(rows, indent=2, ensure_ascii=False))
inserted = updated = quote_matches = newly_queued = 0
now = utc_now()
fetched_job_uuids = set()
if conn is not None:
for row in rows:
was_inserted, is_quote = insert_or_update_raw(
conn,
row,
quote_template_form_uuid=args.quote_template_form_uuid,
now=now,
)
inserted += 1 if was_inserted else 0
updated += 0 if was_inserted else 1
if is_quote:
quote_matches += 1
job_uuid = clean_text(row.get("regarding_object_uuid"))
if job_uuid and job_uuid not in fetched_job_uuids:
try:
job = retrieve_job(
api_key=api_key,
base_url=args.base_url,
job_uuid=job_uuid,
timeout=args.timeout,
)
upsert_job_metadata(conn, job_uuid=job_uuid, job=job, now=now, source="formresponse_poll")
fetched_job_uuids.add(job_uuid)
except Exception as exc:
# Polling/parsing should still proceed if job metadata enrichment fails.
print(
json.dumps(
{"warning": "job_metadata_fetch_failed", "job_uuid": job_uuid, "error": str(exc)},
ensure_ascii=False,
),
file=sys.stderr,
)
if parse_and_store_quote_response(
conn,
row,
queue_path=queue_path,
write_queue=not args.no_queue,
discovered_at=now,
):
newly_queued += 1
conn.commit()
if run_id is not None:
finish_poll_run(
conn,
run_id,
http_status=http_status,
fetched_count=len(rows),
inserted_count=inserted,
updated_count=updated,
quote_match_count=quote_matches,
newly_queued_count=newly_queued,
)
result = {
"ok": True,
"http_status": http_status,
"filter": filter_expr,
"fetched_count": len(rows),
"inserted_count": inserted,
"updated_count": updated,
"quote_template_form_uuid": args.quote_template_form_uuid,
"quote_match_count": quote_matches if conn is not None else sum(1 for r in rows if r.get("form_uuid") == args.quote_template_form_uuid),
"newly_queued_count": newly_queued,
"db_path": None if args.no_store else str(db_path),
"queue_path": None if args.no_queue or args.no_store else str(queue_path),
"sample": summarize_rows(rows, args.summary_limit),
}
print(json.dumps(result, indent=2, ensure_ascii=False))
return 0
except Exception as exc:
if conn is not None and run_id is not None:
finish_poll_run(conn, run_id, error=str(exc))
print(json.dumps({"ok": False, "error": str(exc), "filter": filter_expr}, indent=2), file=sys.stderr)
return 1
finally:
if conn is not None:
conn.close()
if __name__ == "__main__":
raise SystemExit(main())
-5
View File
@@ -1,5 +0,0 @@
home = /usr/bin
include-system-site-packages = false
version = 3.12.3
executable = /usr/bin/python3.12
command = /usr/bin/python3 -m venv /opt/webhooks
+2 -2
View File
@@ -7,8 +7,8 @@ BASE_URL = os.getenv('SERVICEM8_BASE_URL', 'https://api.servicem8.com')
ENDPOINT = '/webhook_subscriptions/event'
ACCESS_TOKEN = os.getenv('SERVICEM8_ACCESS_TOKEN', 'smk-ac525b-99c4b96305a49c7c-fe4dd3e705b647ea')
EVENT_NAME = os.getenv('SERVICEM8_EVENT', 'form.response_created')
CALLBACK_URL = os.getenv('SERVICEM8_CALLBACK_URL', 'https://nps-dev.coast2cloud.net/webhooks/servicem8/form-response')
UNIQUE_ID = os.getenv('SERVICEM8_UNIQUE_ID', 'dev-form-response')
CALLBACK_URL = os.getenv('SERVICEM8_CALLBACK_URL', 'https://webhook.naroomaplumbing.au/webhooks/servicem8/form-response')
UNIQUE_ID = os.getenv('SERVICEM8_UNIQUE_ID', 'au-dev-form-response')
def pretty_print_json(data):
+2 -2
View File
@@ -8,8 +8,8 @@ ENDPOINT = '/webhook_subscriptions'
ACCESS_TOKEN = os.getenv('SERVICEM8_ACCESS_TOKEN', 'smk-ac525b-99c4b96305a49c7c-fe4dd3e705b647ea')
OBJECT_NAME = os.getenv('SERVICEM8_OBJECT', 'job')
FIELDS = os.getenv('SERVICEM8_FIELDS', 'uuid,status,date,queue_uuid,queue_expiry_date,work_order_date,active,edit_date,generated_job_id')
CALLBACK_URL = os.getenv('SERVICEM8_CALLBACK_URL', 'https://nps-dev.coast2cloud.net/webhooks/servicem8-object')
UNIQUE_ID = os.getenv('SERVICEM8_UNIQUE_ID', 'dev-job-object')
CALLBACK_URL = os.getenv('SERVICEM8_CALLBACK_URL', 'https://webhook.naroomaplumbing.au/webhooks/servicem8-object')
UNIQUE_ID = os.getenv('SERVICEM8_UNIQUE_ID', 'au-dev-job-object')
def pretty_print_json(data):
+2 -2
View File
@@ -5,9 +5,9 @@ import requests
BASE_URL = os.getenv('SERVICEM8_BASE_URL', 'https://api.servicem8.com')
ENDPOINT = '/webhook_subscriptions/event'
ACCESS_TOKEN = os.getenv('SERVICEM8_ACCESS_TOKEN', '')
ACCESS_TOKEN = os.getenv('SERVICEM8_ACCESS_TOKEN', 'smk-ac525b-99c4b96305a49c7c-fe4dd3e705b647ea')
EVENT_NAME = os.getenv('SERVICEM8_EVENT', 'job.updated')
CALLBACK_URL = os.getenv('SERVICEM8_CALLBACK_URL', 'https://nps-dev.coast2cloud.net/webhooks/servicem8-job-updated')
CALLBACK_URL = os.getenv('SERVICEM8_CALLBACK_URL', 'https://webhook.naroomaplumbing.au/webhooks/servicem8-job-updated')
UNIQUE_ID = os.getenv('SERVICEM8_UNIQUE_ID', 'dev-job-updated')
+582 -10
View File
@@ -11,7 +11,8 @@ from fastapi.responses import HTMLResponse
DB_PATH = os.getenv("WEBHOOK_DB_PATH", "./servicem8_webhooks.db")
STATE_DB_PATH = os.getenv("WEBHOOK_STATE_DB_PATH", "./servicem8_quote_materials_state.db")
APP_HOST = os.getenv("WEBHOOK_INSPECTOR_HOST", "127.0.0.1")
POLL_DB_PATH = os.getenv("WEBHOOK_POLL_DB_PATH", "./servicem8_formresponse_poll.db")
APP_HOST = os.getenv("WEBHOOK_INSPECTOR_HOST", "0.0.0.0")
APP_PORT = int(os.getenv("WEBHOOK_INSPECTOR_PORT", "18355"))
PAGE_SIZE = 50
@@ -30,13 +31,23 @@ def get_state_conn():
return conn
def get_poll_conn():
conn = sqlite3.connect(POLL_DB_PATH)
conn.row_factory = sqlite3.Row
return conn
def html_page(title: str, body: str) -> HTMLResponse:
nav = """
<nav>
<a href='/'>Dashboard</a>
<a href='/events'>Events</a>
<a href='/objects'>Objects</a>
<a href='/form-responses'>Form responses</a>
<a href='/form-responses'>Webhook form responses</a>
<a href='/poll/form-responses'>Polled form responses</a>
<a href='/poll/quote-template'>Polled quote templates</a>
<a href='/poll/apply-runs'>Apply runs</a>
<a href='/poll/remote-existing-incidents'>Remote existing incidents</a>
<a href='/generated-materials'>Generated materials</a>
</nav>
"""
@@ -57,6 +68,7 @@ def html_page(title: str, body: str) -> HTMLResponse:
code, pre { font-family: ui-monospace, SFMono-Regular, Menlo, monospace; }
pre { white-space: pre-wrap; word-break: break-word; background: #0f172a; color: #e2e8f0; padding: 14px; border-radius: 10px; overflow-x: auto; }
.pill { display: inline-block; padding: 2px 8px; border-radius: 999px; background: #e0f2fe; color: #075985; font-size: 0.85rem; margin: 2px 4px 2px 0; }
.job-id { font-weight: 700; color: #111827; }
.section { margin: 24px 0; }
.toolbar { background: white; border: 1px solid #e5e7eb; border-radius: 10px; padding: 12px; margin-bottom: 16px; }
input[type='text'] { padding: 8px; width: 280px; max-width: 100%; }
@@ -82,6 +94,14 @@ def pretty_json(value):
return str(value)
def html_value(value):
if value is None:
return ""
if isinstance(value, (dict, list)):
return pretty_json(value)
return str(value)
def parse_json_field(value, default=None):
if value is None:
return default
@@ -120,9 +140,66 @@ def link_with_params(path, **params):
return f"{path}?{urlencode(filtered)}" if filtered else path
def resolve_generated_job_id(job_uuid: str) -> str:
job_uuid = str(job_uuid or "").strip()
if not job_uuid:
return ""
try:
with closing(get_poll_conn()) as conn:
row = conn.execute(
"select generated_job_id from job_metadata where job_uuid = ?",
(job_uuid,),
).fetchone()
if row and row["generated_job_id"]:
return str(row["generated_job_id"])
except sqlite3.Error:
pass
try:
with closing(get_conn()) as conn:
row = conn.execute(
"""
with jobs as (
select
json_extract(payload_json, '$.data.uuid') as job_uuid,
json_extract(payload_json, '$.data.generated_job_id') as generated_job_id,
received_at
from webhook_events
where json_extract(payload_json, '$.data.generated_job_id') is not null
union all
select
json_extract(payload_json, '$.related.job.uuid') as job_uuid,
json_extract(payload_json, '$.related.job.generated_job_id') as generated_job_id,
received_at
from webhook_form_responses
where json_extract(payload_json, '$.related.job.generated_job_id') is not null
)
select generated_job_id
from jobs
where job_uuid = ?
and generated_job_id is not null
order by received_at desc
limit 1
""",
(job_uuid,),
).fetchone()
if row and row["generated_job_id"]:
return str(row["generated_job_id"])
except sqlite3.Error:
pass
return ""
def job_id_html(job_uuid: str) -> str:
generated_job_id = resolve_generated_job_id(job_uuid)
return f"<span class='job-id'>{escape(generated_job_id)}</span>" if generated_job_id else ""
@app.get("/health")
def health():
return {"ok": True, "db_path": DB_PATH, "state_db_path": STATE_DB_PATH}
return {"ok": True, "db_path": DB_PATH, "state_db_path": STATE_DB_PATH, "poll_db_path": POLL_DB_PATH}
@app.get("/", response_class=HTMLResponse)
@@ -146,7 +223,34 @@ def dashboard():
except sqlite3.Error:
pass
poll_counts = {
"poll_runs": 0,
"polled_form_responses": 0,
"polled_quote_templates": 0,
"remote_existing_incidents": 0,
}
latest_poll_run = None
latest_polled_form = None
latest_polled_quote = None
latest_remote_existing_incident = None
try:
with closing(get_poll_conn()) as conn:
poll_counts["poll_runs"] = conn.execute("select count(*) from poll_runs").fetchone()[0]
poll_counts["polled_form_responses"] = conn.execute("select count(*) from form_responses_raw").fetchone()[0]
poll_counts["polled_quote_templates"] = conn.execute("select count(*) from quote_template_form_responses").fetchone()[0]
try:
poll_counts["remote_existing_incidents"] = conn.execute("select count(*) from quote_template_remote_existing_incidents").fetchone()[0]
latest_remote_existing_incident = conn.execute("select detected_at from quote_template_remote_existing_incidents order by id desc limit 1").fetchone()
except sqlite3.Error:
pass
latest_poll_run = conn.execute("select finished_at from poll_runs order by id desc limit 1").fetchone()
latest_polled_form = conn.execute("select timestamp from form_responses_raw order by timestamp desc limit 1").fetchone()
latest_polled_quote = conn.execute("select discovered_at from quote_template_form_responses order by discovered_at desc limit 1").fetchone()
except sqlite3.Error:
pass
counts["generated_job_materials"] = state_count
counts.update(poll_counts)
cards = "".join(
f"<div class='card'><div class='muted'>{escape(name)}</div><div class='big'>{count}</div></div>"
@@ -158,10 +262,15 @@ def dashboard():
<div class='summary-grid'>
<div><strong>DB path</strong></div><div><code>{escape(DB_PATH)}</code></div>
<div><strong>State DB path</strong></div><div><code>{escape(STATE_DB_PATH)}</code></div>
<div><strong>Poll DB path</strong></div><div><code>{escape(POLL_DB_PATH)}</code></div>
<div><strong>Latest event</strong></div><div>{escape(latest_event[0] if latest_event else '')}</div>
<div><strong>Latest object</strong></div><div>{escape(latest_object[0] if latest_object else '')}</div>
<div><strong>Latest form response</strong></div><div>{escape(latest_form[0] if latest_form else '')}</div>
<div><strong>Latest generated material row</strong></div><div>{escape(latest_generated[0] if latest_generated else '')}</div>
<div><strong>Latest poll run</strong></div><div>{escape(latest_poll_run[0] if latest_poll_run else '')}</div>
<div><strong>Latest polled form timestamp</strong></div><div>{escape(latest_polled_form[0] if latest_polled_form else '')}</div>
<div><strong>Latest polled quote discovered</strong></div><div>{escape(latest_polled_quote[0] if latest_polled_quote else '')}</div>
<div><strong>Latest remote-existing incident</strong></div><div>{escape(latest_remote_existing_incident[0] if latest_remote_existing_incident else '')}</div>
</div>
</div>
<div class='section'>
@@ -169,7 +278,12 @@ def dashboard():
<ul>
<li><a href='/events'>Browse webhook events</a></li>
<li><a href='/objects'>Browse object webhooks</a></li>
<li><a href='/form-responses'>Browse form responses</a></li>
<li><a href='/form-responses'>Browse webhook form responses</a></li>
<li><a href='/poll/form-responses'>Browse polled form responses</a></li>
<li><a href='/poll/quote-template'>Browse parsed polled Quote Template responses</a></li>
<li><a href='/poll/runs'>Browse poll runs</a></li>
<li><a href='/poll/apply-runs'>Browse dry-run/apply runs</a></li>
<li><a href='/poll/remote-existing-incidents'>Browse remote-existing incidents</a></li>
<li><a href='/generated-materials'>Browse generated job-material state</a></li>
</ul>
</div>
@@ -403,6 +517,7 @@ def list_generated_materials(page: int = Query(1, ge=1)):
table_rows.append(
f"<tr>"
f"<td><a href='/generated-materials/{row['id']}'>{row['id']}</a></td>"
f"<td>{job_id_html(row['job_uuid'])}</td>"
f"<td>{escape(row['job_uuid'] or '')}</td>"
f"<td>{escape(row['form_response_uuid'] or '')}</td>"
f"<td>{escape(row['job_material_uuid'] or '')}</td>"
@@ -414,8 +529,8 @@ def list_generated_materials(page: int = Query(1, ge=1)):
body = f"""
<table>
<thead><tr><th>ID</th><th>Job UUID</th><th>Form response UUID</th><th>Job material UUID</th><th>Kind</th><th>Source question</th><th>Updated</th></tr></thead>
<tbody>{''.join(table_rows) or "<tr><td colspan='7'>No rows found.</td></tr>"}</tbody>
<thead><tr><th>ID</th><th>Job ID</th><th>Job UUID</th><th>Form response UUID</th><th>Job material UUID</th><th>Kind</th><th>Source question</th><th>Updated</th></tr></thead>
<tbody>{''.join(table_rows) or "<tr><td colspan='8'>No rows found.</td></tr>"}</tbody>
</table>
<div class='pagination'>
{f"<a href='{link_with_params('/generated-materials', page=page-1)}'>← Prev</a>" if page > 1 else ''}
@@ -439,6 +554,7 @@ def generated_material_detail(row_id: int):
body = f"""
<div class='card summary-grid'>
<div><strong>ID</strong></div><div>{row['id']}</div>
<div><strong>Job ID</strong></div><div>{job_id_html(row['job_uuid'])}</div>
<div><strong>Job UUID</strong></div><div>{escape(row['job_uuid'] or '')}</div>
<div><strong>Form response UUID</strong></div><div>{escape(row['form_response_uuid'] or '')}</div>
<div><strong>Job material UUID</strong></div><div>{escape(row['job_material_uuid'] or '')}</div>
@@ -472,10 +588,10 @@ def form_response_detail(row_id: int):
for item in sorted(field_data, key=lambda x: x.get("SortOrder", 0)):
field_rows.append(
f"<tr>"
f"<td>{escape(str(item.get('SortOrder', '')))}</td>"
f"<td>{escape(item.get('FieldType', ''))}</td>"
f"<td>{escape(item.get('Question', ''))}</td>"
f"<td>{escape(item.get('Response', ''))}</td>"
f"<td>{escape(html_value(item.get('SortOrder')))}</td>"
f"<td>{escape(html_value(item.get('FieldType')))}</td>"
f"<td>{escape(html_value(item.get('Question')))}</td>"
f"<td>{escape(html_value(item.get('Response')))}</td>"
f"</tr>"
)
@@ -504,6 +620,462 @@ def form_response_detail(row_id: int):
return html_page(f"Form response {row_id}", body)
@app.get("/poll/remote-existing-incidents", response_class=HTMLResponse)
def list_remote_existing_incidents(page: int = Query(1, ge=1)):
offset = (page - 1) * PAGE_SIZE
try:
with closing(get_poll_conn()) as conn:
rows = conn.execute(
"""
select id, detected_at, form_response_uuid, job_uuid, apply_run_id,
desired_count, remote_count, remote_active_count, action, reason
from quote_template_remote_existing_incidents
order by id desc
limit ? offset ?
""",
(PAGE_SIZE, offset),
).fetchall()
except sqlite3.Error as e:
return html_page("Remote existing incidents", f"<div class='card'>Incident table unavailable: {escape(str(e))}</div>")
table_rows = []
for row in rows:
run_link = f"<a href='/poll/apply-runs/{row['apply_run_id']}'>{row['apply_run_id']}</a>" if row['apply_run_id'] else ""
table_rows.append(
f"<tr><td><a href='/poll/remote-existing-incidents/{row['id']}'>{row['id']}</a></td>"
f"<td>{escape(row['detected_at'] or '')}</td><td>{escape(row['action'] or '')}</td>"
f"<td><a href='/poll/quote-template/{escape(row['form_response_uuid'])}'>{escape(row['form_response_uuid'])}</a></td>"
f"<td>{job_id_html(row['job_uuid'])}</td><td>{escape(row['job_uuid'] or '')}</td><td>{run_link}</td>"
f"<td>{row['desired_count']}</td><td>{row['remote_count']}</td><td>{row['remote_active_count']}</td>"
f"<td>{escape((row['reason'] or '')[:180])}</td></tr>"
)
body = f"""
<table>
<thead><tr><th>ID</th><th>Detected</th><th>Action</th><th>Form response UUID</th><th>Job ID</th><th>Job UUID</th><th>Apply run</th><th>Desired</th><th>Remote rows</th><th>Active</th><th>Reason</th></tr></thead>
<tbody>{''.join(table_rows) or "<tr><td colspan='11'>No remote-existing incidents found.</td></tr>"}</tbody>
</table>
<div class='pagination'>
{f"<a href='{link_with_params('/poll/remote-existing-incidents', page=page-1)}'>← Prev</a>" if page > 1 else ''}
<a href='{link_with_params('/poll/remote-existing-incidents', page=page+1)}'>Next →</a>
</div>
"""
return html_page("Remote existing incidents", body)
@app.get("/poll/remote-existing-incidents/{incident_id}", response_class=HTMLResponse)
def remote_existing_incident_detail(incident_id: int):
try:
with closing(get_poll_conn()) as conn:
row = conn.execute("select * from quote_template_remote_existing_incidents where id = ?", (incident_id,)).fetchone()
except sqlite3.Error as e:
return html_page("Remote existing incident", f"<div class='card'>Incident table unavailable: {escape(str(e))}</div>")
if not row:
raise HTTPException(status_code=404, detail="Remote existing incident not found")
remote_rows = parse_json_field(row["remote_rows_json"], []) or []
material_rows = []
if isinstance(remote_rows, list):
for item in remote_rows:
if isinstance(item, dict):
material_rows.append(
f"<tr><td>{escape(str(item.get('uuid', '')))}</td><td>{escape(str(item.get('active', '')))}</td>"
f"<td>{escape(str(item.get('name', '')))}</td><td>{escape(str(item.get('material_uuid', '')))}</td>"
f"<td>{escape(str(item.get('quantity', '')))}</td><td>{escape(str(item.get('price', '')))}</td>"
f"<td>{escape(str(item.get('sort_order', '')))}</td></tr>"
)
run_link = f"<a href='/poll/apply-runs/{row['apply_run_id']}'>{row['apply_run_id']}</a>" if row['apply_run_id'] else ""
body = f"""
<div class='card summary-grid'>
<div><strong>ID</strong></div><div>{row['id']}</div>
<div><strong>Detected</strong></div><div>{escape(row['detected_at'] or '')}</div>
<div><strong>Action</strong></div><div>{escape(row['action'] or '')}</div>
<div><strong>Form response UUID</strong></div><div><a href='/poll/quote-template/{escape(row['form_response_uuid'])}'>{escape(row['form_response_uuid'])}</a></div>
<div><strong>Job ID</strong></div><div>{job_id_html(row['job_uuid'])}</div>
<div><strong>Job UUID</strong></div><div>{escape(row['job_uuid'] or '')}</div>
<div><strong>Apply run</strong></div><div>{run_link}</div>
<div><strong>Desired rows</strong></div><div>{row['desired_count']}</div>
<div><strong>Remote rows</strong></div><div>{row['remote_count']}</div>
<div><strong>Remote active rows</strong></div><div>{row['remote_active_count']}</div>
<div><strong>Reason</strong></div><div>{escape(row['reason'] or '')}</div>
</div>
<div class='section'><h2>Remote jobMaterial rows</h2><table><thead><tr><th>UUID</th><th>Active</th><th>Name</th><th>Material UUID</th><th>Qty</th><th>Price</th><th>Sort</th></tr></thead><tbody>{''.join(material_rows) or "<tr><td colspan='7'>No remote rows captured.</td></tr>"}</tbody></table></div>
<div class='section'><h2>Raw remote rows JSON</h2><pre>{escape(pretty_json(remote_rows))}</pre></div>
"""
return html_page(f"Remote existing incident {incident_id}", body)
@app.get("/poll/apply-runs", response_class=HTMLResponse)
def list_apply_runs(page: int = Query(1, ge=1)):
offset = (page - 1) * PAGE_SIZE
try:
with closing(get_poll_conn()) as conn:
rows = conn.execute(
"""
select id, form_response_uuid, job_uuid, mode, started_at, finished_at,
desired_count, created_count, status, error
from quote_template_apply_runs
order by id desc
limit ? offset ?
""",
(PAGE_SIZE, offset),
).fetchall()
except sqlite3.Error as e:
return html_page("Apply runs", f"<div class='card'>Apply-run table unavailable: {escape(str(e))}</div>")
table_rows = []
for row in rows:
table_rows.append(
f"<tr><td><a href='/poll/apply-runs/{row['id']}'>{row['id']}</a></td>"
f"<td>{escape(row['mode'] or '')}</td><td>{escape(row['status'] or '')}</td>"
f"<td><a href='/poll/quote-template/{escape(row['form_response_uuid'])}'>{escape(row['form_response_uuid'])}</a></td>"
f"<td>{job_id_html(row['job_uuid'])}</td><td>{escape(row['job_uuid'] or '')}</td><td>{escape(row['started_at'] or '')}</td><td>{escape(row['finished_at'] or '')}</td>"
f"<td>{row['desired_count']}</td><td>{row['created_count']}</td><td>{escape((row['error'] or '')[:160])}</td></tr>"
)
body = f"""
<table>
<thead><tr><th>ID</th><th>Mode</th><th>Status</th><th>Form response UUID</th><th>Job ID</th><th>Job UUID</th><th>Started</th><th>Finished</th><th>Desired</th><th>Created</th><th>Error</th></tr></thead>
<tbody>{''.join(table_rows) or "<tr><td colspan='11'>No apply runs found yet.</td></tr>"}</tbody>
</table>
<div class='pagination'>
{f"<a href='{link_with_params('/poll/apply-runs', page=page-1)}'>← Prev</a>" if page > 1 else ''}
<a href='{link_with_params('/poll/apply-runs', page=page+1)}'>Next →</a>
</div>
"""
return html_page("Apply runs", body)
@app.get("/poll/apply-runs/{run_id}", response_class=HTMLResponse)
def apply_run_detail(run_id: int):
try:
with closing(get_poll_conn()) as conn:
run = conn.execute("select * from quote_template_apply_runs where id = ?", (run_id,)).fetchone()
rows = conn.execute(
"select * from quote_template_apply_run_rows where run_id = ? order by row_index asc",
(run_id,),
).fetchall()
incidents = conn.execute(
"select id, detected_at, action, remote_count, remote_active_count, reason from quote_template_remote_existing_incidents where apply_run_id = ? order by id desc",
(run_id,),
).fetchall()
except sqlite3.Error as e:
return html_page("Apply run", f"<div class='card'>Apply-run table unavailable: {escape(str(e))}</div>")
if not run:
raise HTTPException(status_code=404, detail="Apply run not found")
incident_rows = []
for incident in incidents:
incident_rows.append(
f"<tr><td><a href='/poll/remote-existing-incidents/{incident['id']}'>{incident['id']}</a></td>"
f"<td>{escape(incident['detected_at'] or '')}</td><td>{escape(incident['action'] or '')}</td>"
f"<td>{incident['remote_count']}</td><td>{incident['remote_active_count']}</td><td>{escape((incident['reason'] or '')[:180])}</td></tr>"
)
table_rows = []
for row in rows:
payload = parse_json_field(row['api_payload_json'], {}) or {}
table_rows.append(
f"<tr><td>{row['row_index']}</td><td>{escape(row['action'] or '')}</td><td>{escape(row['kind'] or '')}</td>"
f"<td>{escape(row['name'] or '')}</td><td>{escape(row['job_material_uuid'] or '')}</td>"
f"<td>{escape(row['source_question'] or '')}</td><td>{escape(row['error'] or '')}</td></tr>"
f"<tr><td></td><td colspan='6'><details><summary>API payload</summary><pre>{escape(pretty_json(payload))}</pre></details></td></tr>"
)
body = f"""
<div class='card summary-grid'>
<div><strong>Run ID</strong></div><div>{run['id']}</div>
<div><strong>Mode</strong></div><div>{escape(run['mode'] or '')}</div>
<div><strong>Status</strong></div><div>{escape(run['status'] or '')}</div>
<div><strong>Form response UUID</strong></div><div><a href='/poll/quote-template/{escape(run['form_response_uuid'])}'>{escape(run['form_response_uuid'])}</a></div>
<div><strong>Job ID</strong></div><div>{job_id_html(run['job_uuid'])}</div>
<div><strong>Job UUID</strong></div><div>{escape(run['job_uuid'] or '')}</div>
<div><strong>Started</strong></div><div>{escape(run['started_at'] or '')}</div>
<div><strong>Finished</strong></div><div>{escape(run['finished_at'] or '')}</div>
<div><strong>Desired</strong></div><div>{run['desired_count']}</div>
<div><strong>Created</strong></div><div>{run['created_count']}</div>
<div><strong>Error</strong></div><div>{escape(run['error'] or '')}</div>
</div>
<div class='section'><h2>Remote-existing incidents</h2><table><thead><tr><th>ID</th><th>Detected</th><th>Action</th><th>Remote rows</th><th>Active</th><th>Reason</th></tr></thead><tbody>{''.join(incident_rows) or "<tr><td colspan='6'>No remote-existing incidents for this run.</td></tr>"}</tbody></table></div>
<div class='section'><h2>Rows</h2><table><thead><tr><th>#</th><th>Action</th><th>Kind</th><th>Name</th><th>Created UUID</th><th>Source question</th><th>Error</th></tr></thead><tbody>{''.join(table_rows) or "<tr><td colspan='7'>No rows recorded.</td></tr>"}</tbody></table></div>
"""
return html_page(f"Apply run {run_id}", body)
@app.get("/poll/runs", response_class=HTMLResponse)
def list_poll_runs(page: int = Query(1, ge=1)):
offset = (page - 1) * PAGE_SIZE
try:
with closing(get_poll_conn()) as conn:
rows = conn.execute(
"""
select id, started_at, finished_at, since_value, filter_field,
fetched_count, inserted_count, updated_count,
quote_match_count, newly_queued_count, error
from poll_runs order by id desc limit ? offset ?
""",
(PAGE_SIZE, offset),
).fetchall()
except sqlite3.Error as e:
return html_page("Poll runs", f"<div class='card'>Poll DB unavailable: {escape(str(e))}</div>")
table_rows = []
for row in rows:
table_rows.append(
f"<tr><td>{row['id']}</td><td>{escape(row['started_at'] or '')}</td>"
f"<td>{escape(row['finished_at'] or '')}</td>"
f"<td>{escape(row['filter_field'] or '')} gt {escape(row['since_value'] or '')}</td>"
f"<td>{row['fetched_count']}</td><td>{row['inserted_count']}</td><td>{row['updated_count']}</td>"
f"<td>{row['quote_match_count']}</td><td>{row['newly_queued_count']}</td>"
f"<td>{escape((row['error'] or '')[:180])}</td></tr>"
)
body = f"""
<table>
<thead><tr><th>ID</th><th>Started</th><th>Finished</th><th>Filter</th><th>Fetched</th><th>Inserted</th><th>Updated</th><th>Quote matches</th><th>Queued</th><th>Error</th></tr></thead>
<tbody>{''.join(table_rows) or "<tr><td colspan='10'>No poll runs found.</td></tr>"}</tbody>
</table>
<div class='pagination'>
{f"<a href='{link_with_params('/poll/runs', page=page-1)}'>← Prev</a>" if page > 1 else ''}
<a href='{link_with_params('/poll/runs', page=page+1)}'>Next →</a>
</div>
"""
return html_page("Poll runs", body)
@app.get("/poll/form-responses", response_class=HTMLResponse)
def list_polled_form_responses(q: str = Query(""), quote_only: int = Query(0), page: int = Query(1, ge=1)):
offset = (page - 1) * PAGE_SIZE
clauses = []
params = []
if quote_only:
clauses.append("is_quote_template = 1")
if q.strip():
like = f"%{q.strip()}%"
clauses.append("(uuid like ? or form_uuid like ? or regarding_object_uuid like ? or timestamp like ? or edit_date like ? or raw_json like ?)")
params.extend([like, like, like, like, like, like])
where = " where " + " and ".join(clauses) if clauses else ""
try:
with closing(get_poll_conn()) as conn:
rows = conn.execute(
f"""
select uuid, first_seen_at, last_seen_at, seen_count, timestamp, edit_date,
form_uuid, regarding_object, regarding_object_uuid,
is_quote_template, parse_status, parse_error
from form_responses_raw {where}
order by timestamp desc, edit_date desc limit ? offset ?
""",
(*params, PAGE_SIZE, offset),
).fetchall()
except sqlite3.Error as e:
return html_page("Polled form responses", f"<div class='card'>Poll DB unavailable: {escape(str(e))}</div>")
table_rows = []
for row in rows:
quote_pill = "<span class='pill'>Quote Template</span>" if row["is_quote_template"] else ""
table_rows.append(
f"<tr><td><a href='/poll/form-responses/{escape(row['uuid'])}'>{escape(row['uuid'])}</a></td>"
f"<td>{escape(row['timestamp'] or '')}</td><td>{escape(row['edit_date'] or '')}</td>"
f"<td>{escape(row['form_uuid'] or '')}<br>{quote_pill}</td>"
f"<td>{escape(row['regarding_object'] or '')}</td><td>{job_id_html(row['regarding_object_uuid'])}</td><td>{escape(row['regarding_object_uuid'] or '')}</td>"
f"<td>{escape(row['parse_status'] or '')}</td><td>{row['seen_count']}</td><td>{escape(row['last_seen_at'] or '')}</td></tr>"
)
body = f"""
<div class='toolbar'>
<form method='get'>
<label>Search <input type='text' name='q' value='{escape(q)}' placeholder='uuid, job uuid, form uuid, timestamp'></label>
<label><input type='checkbox' name='quote_only' value='1' {'checked' if quote_only else ''}> Quote Template only</label>
<input type='hidden' name='page' value='1'>
<button type='submit'>Filter</button>
</form>
</div>
<table>
<thead><tr><th>UUID</th><th>Timestamp</th><th>Edit date</th><th>Form UUID</th><th>Regarding</th><th>Job ID</th><th>Object UUID</th><th>Parse</th><th>Seen</th><th>Last seen</th></tr></thead>
<tbody>{''.join(table_rows) or "<tr><td colspan='10'>No rows found.</td></tr>"}</tbody>
</table>
<div class='pagination'>
{f"<a href='{link_with_params('/poll/form-responses', q=q, quote_only=quote_only, page=page-1)}'>← Prev</a>" if page > 1 else ''}
<a href='{link_with_params('/poll/form-responses', q=q, quote_only=quote_only, page=page+1)}'>Next →</a>
</div>
"""
return html_page("Polled form responses", body)
@app.get("/poll/form-responses/{form_response_uuid}", response_class=HTMLResponse)
def polled_form_response_detail(form_response_uuid: str):
try:
with closing(get_poll_conn()) as conn:
row = conn.execute("select * from form_responses_raw where uuid = ?", (form_response_uuid,)).fetchone()
quote = conn.execute("select * from quote_template_form_responses where form_response_uuid = ?", (form_response_uuid,)).fetchone()
except sqlite3.Error as e:
return html_page("Polled form response", f"<div class='card'>Poll DB unavailable: {escape(str(e))}</div>")
if not row:
raise HTTPException(status_code=404, detail="Polled form response not found")
raw = parse_json_field(row["raw_json"], {}) or {}
field_data = parse_json_field(raw.get("field_data"), []) or []
field_rows = []
if isinstance(field_data, list):
for item in sorted(field_data, key=lambda x: x.get("SortOrder", 0) if isinstance(x, dict) else 0):
if isinstance(item, dict):
field_rows.append(
f"<tr><td>{escape(html_value(item.get('SortOrder')))}</td><td>{escape(html_value(item.get('FieldType')))}</td>"
f"<td>{escape(html_value(item.get('Question')))}</td><td>{escape(html_value(item.get('Response')))}</td></tr>"
)
quote_link = f"<div><strong>Parsed quote</strong></div><div><a href='/poll/quote-template/{escape(form_response_uuid)}'>Open parsed Quote Template view</a></div>" if quote else ""
body = f"""
<div class='card summary-grid'>
<div><strong>UUID</strong></div><div>{escape(row['uuid'])}</div>
<div><strong>Timestamp</strong></div><div>{escape(row['timestamp'] or '')}</div>
<div><strong>Edit date</strong></div><div>{escape(row['edit_date'] or '')}</div>
<div><strong>Form UUID</strong></div><div>{escape(row['form_uuid'] or '')}</div>
<div><strong>Regarding object</strong></div><div>{escape(row['regarding_object'] or '')}</div>
<div><strong>Job ID</strong></div><div>{job_id_html(row['regarding_object_uuid'])}</div>
<div><strong>Regarding UUID</strong></div><div>{escape(row['regarding_object_uuid'] or '')}</div>
<div><strong>First seen</strong></div><div>{escape(row['first_seen_at'] or '')}</div>
<div><strong>Last seen</strong></div><div>{escape(row['last_seen_at'] or '')}</div>
<div><strong>Seen count</strong></div><div>{row['seen_count']}</div>
<div><strong>Parse status</strong></div><div>{escape(row['parse_status'] or '')}</div>
<div><strong>Parse error</strong></div><div>{escape(row['parse_error'] or '')}</div>
{quote_link}
</div>
<div class='section'><h2>Decoded field data</h2><table><thead><tr><th>Order</th><th>Type</th><th>Question</th><th>Response</th></tr></thead><tbody>{''.join(field_rows) or "<tr><td colspan='4'>No decoded field data.</td></tr>"}</tbody></table></div>
<div class='section'><h2>Raw API row</h2><pre>{escape(pretty_json(raw))}</pre></div>
"""
return html_page(f"Polled form response {form_response_uuid}", body)
@app.get("/poll/quote-template", response_class=HTMLResponse)
def list_polled_quote_templates(page: int = Query(1, ge=1)):
offset = (page - 1) * PAGE_SIZE
try:
with closing(get_poll_conn()) as conn:
rows = conn.execute(
"""
select form_response_uuid, discovered_at, job_uuid, form_uuid, author_name,
description, desired_job_materials_json, queued_at, processed_at,
process_status, process_error
from quote_template_form_responses
order by discovered_at desc, form_response_uuid desc limit ? offset ?
""",
(PAGE_SIZE, offset),
).fetchall()
except sqlite3.Error as e:
return html_page("Polled Quote Template responses", f"<div class='card'>Poll DB unavailable: {escape(str(e))}</div>")
table_rows = []
for row in rows:
try:
material_count = len(json.loads(row["desired_job_materials_json"] or "[]"))
except Exception:
material_count = "?"
table_rows.append(
f"<tr><td><a href='/poll/quote-template/{escape(row['form_response_uuid'])}'>{escape(row['form_response_uuid'])}</a></td>"
f"<td>{escape(row['discovered_at'] or '')}</td><td>{job_id_html(row['job_uuid'])}</td><td>{escape(row['job_uuid'] or '')}</td>"
f"<td>{escape(row['description'] or '')}</td><td>{material_count}</td>"
f"<td>{escape(row['queued_at'] or '')}</td><td>{escape(row['process_status'] or '')}</td></tr>"
)
body = f"""
<table>
<thead><tr><th>Form response UUID</th><th>Discovered</th><th>Job ID</th><th>Job UUID</th><th>Description</th><th>Rows</th><th>Queued</th><th>Status</th></tr></thead>
<tbody>{''.join(table_rows) or "<tr><td colspan='8'>No quote template rows found.</td></tr>"}</tbody>
</table>
<div class='pagination'>
{f"<a href='{link_with_params('/poll/quote-template', page=page-1)}'>← Prev</a>" if page > 1 else ''}
<a href='{link_with_params('/poll/quote-template', page=page+1)}'>Next →</a>
</div>
"""
return html_page("Polled Quote Template responses", body)
@app.get("/poll/quote-template/{form_response_uuid}", response_class=HTMLResponse)
def polled_quote_template_detail(form_response_uuid: str):
try:
with closing(get_poll_conn()) as conn:
row = conn.execute("select * from quote_template_form_responses where form_response_uuid = ?", (form_response_uuid,)).fetchone()
except sqlite3.Error as e:
return html_page("Polled Quote Template response", f"<div class='card'>Poll DB unavailable: {escape(str(e))}</div>")
if not row:
raise HTTPException(status_code=404, detail="Polled quote template response not found")
parsed = parse_json_field(row["parsed_json"], {}) or {}
materials = parse_json_field(row["desired_job_materials_json"], []) or []
material_rows = []
for item in materials:
material_rows.append(
f"<tr><td>{escape(str(item.get('sort_order', '')))}</td><td>{escape(item.get('kind', ''))}</td>"
f"<td>{escape(item.get('name', ''))}</td><td>{escape(item.get('material_uuid', ''))}</td>"
f"<td>{escape(item.get('source_question', ''))}</td></tr>"
)
dry_run_cmd = f"/opt/webhooks/apply_polled_quote_template_jobmaterials.py --uuid {row['form_response_uuid']} --pretty"
apply_cmd = f"/opt/webhooks/apply_polled_quote_template_jobmaterials.py --uuid {row['form_response_uuid']} --apply --pretty"
try:
with closing(get_poll_conn()) as conn:
recent_runs = conn.execute(
"select id, mode, status, started_at, finished_at, desired_count, created_count, error from quote_template_apply_runs where form_response_uuid = ? order by id desc limit 8",
(row['form_response_uuid'],),
).fetchall()
recent_incidents = conn.execute(
"select id, detected_at, action, remote_count, remote_active_count, reason from quote_template_remote_existing_incidents where form_response_uuid = ? order by id desc limit 8",
(row['form_response_uuid'],),
).fetchall()
except sqlite3.Error:
recent_runs = []
recent_incidents = []
recent_run_rows = []
for run in recent_runs:
recent_run_rows.append(
f"<tr><td><a href='/poll/apply-runs/{run['id']}'>{run['id']}</a></td><td>{escape(run['mode'] or '')}</td><td>{escape(run['status'] or '')}</td><td>{escape(run['started_at'] or '')}</td><td>{escape(run['finished_at'] or '')}</td><td>{run['desired_count']}</td><td>{run['created_count']}</td><td>{escape((run['error'] or '')[:120])}</td></tr>"
)
recent_incident_rows = []
for incident in recent_incidents:
recent_incident_rows.append(
f"<tr><td><a href='/poll/remote-existing-incidents/{incident['id']}'>{incident['id']}</a></td><td>{escape(incident['detected_at'] or '')}</td><td>{escape(incident['action'] or '')}</td><td>{incident['remote_count']}</td><td>{incident['remote_active_count']}</td><td>{escape((incident['reason'] or '')[:120])}</td></tr>"
)
body = f"""
<div class='card summary-grid'>
<div><strong>Form response UUID</strong></div><div>{escape(row['form_response_uuid'])}</div>
<div><strong>Discovered</strong></div><div>{escape(row['discovered_at'] or '')}</div>
<div><strong>Job ID</strong></div><div>{job_id_html(row['job_uuid'])}</div>
<div><strong>Job UUID</strong></div><div>{escape(row['job_uuid'] or '')}</div>
<div><strong>Form UUID</strong></div><div>{escape(row['form_uuid'] or '')}</div>
<div><strong>Author</strong></div><div>{escape(row['author_name'] or '')}</div>
<div><strong>Description</strong></div><div>{escape(row['description'] or '')}</div>
<div><strong>Queued</strong></div><div>{escape(row['queued_at'] or '')}</div>
<div><strong>Processed</strong></div><div>{escape(row['processed_at'] or '')}</div>
<div><strong>Status</strong></div><div>{escape(row['process_status'] or '')}</div>
<div><strong>Error</strong></div><div>{escape(row['process_error'] or '')}</div>
<div><strong>Raw polled response</strong></div><div><a href='/poll/form-responses/{escape(row['form_response_uuid'])}'>Open raw polled form response</a></div>
</div>
<div class='section'>
<h2>Selective apply commands</h2>
<div class='card'>
<p><strong>Dry-run first:</strong></p>
<pre>{escape(dry_run_cmd)}</pre>
<p><strong>Apply to ServiceM8 only after checking the dry-run:</strong></p>
<pre>{escape(apply_cmd)}</pre>
</div>
</div>
<div class='section'>
<h2>Recent dry-run/apply runs for this response</h2>
<table><thead><tr><th>ID</th><th>Mode</th><th>Status</th><th>Started</th><th>Finished</th><th>Desired</th><th>Created</th><th>Error</th></tr></thead><tbody>{''.join(recent_run_rows) or "<tr><td colspan='8'>No dry-run/apply runs yet.</td></tr>"}</tbody></table>
</div>
<div class='section'>
<h2>Remote-existing incidents for this response</h2>
<table><thead><tr><th>ID</th><th>Detected</th><th>Action</th><th>Remote rows</th><th>Active</th><th>Reason</th></tr></thead><tbody>{''.join(recent_incident_rows) or "<tr><td colspan='6'>No remote-existing incidents yet.</td></tr>"}</tbody></table>
</div>
<div class='section'><h2>Desired jobMaterial rows</h2><table><thead><tr><th>Sort</th><th>Kind</th><th>Name</th><th>Material UUID</th><th>Source question</th></tr></thead><tbody>{''.join(material_rows) or "<tr><td colspan='5'>No desired jobMaterial rows.</td></tr>"}</tbody></table></div>
<div class='section'><h2>Parsed JSON</h2><pre>{escape(pretty_json(parsed))}</pre></div>
"""
return html_page(f"Polled Quote Template {form_response_uuid}", body)
if __name__ == "__main__":
import uvicorn
@@ -34,7 +34,7 @@ def clean_text(value: Any) -> str:
def get_state_conn(db_path: Path = STATE_DB_PATH):
conn = sqlite3.connect(db_path)
conn = sqlite3.connect(db_path, timeout=30)
conn.row_factory = sqlite3.Row
return conn
@@ -145,6 +145,7 @@ def build_job_material_line(
def parse_quote_template_field_rows(field_rows: List[Dict[str, Any]]) -> Dict[str, Any]:
ordered = sorted(field_rows, key=sort_key)
description = ""
author_name = ""
include_items: List[Dict[str, Any]] = []
exclude_items: List[Dict[str, Any]] = []
extra_include_items: List[Dict[str, Any]] = []
@@ -161,6 +162,10 @@ def parse_quote_template_field_rows(field_rows: List[Dict[str, Any]]) -> Dict[st
description = response
continue
if question == "Name":
author_name = response
continue
if question.startswith("Item ") and response:
include_items.append(
{
@@ -253,6 +258,7 @@ def parse_quote_template_field_rows(field_rows: List[Dict[str, Any]]) -> Dict[st
return {
"description": description,
"author_name": author_name,
"include_items": include_items,
"extra_include_items": extra_include_items,
"exclude_items": exclude_items,
+1
View File
@@ -170,6 +170,7 @@ def queue_quote_template_jobmaterials(payload, received_at: str):
"form_uuid": form_uuid,
"form_response_uuid": parsed.get("form_response_uuid", ""),
"job_uuid": parsed.get("job_uuid", ""),
"author_name": parsed.get("author_name", ""),
"description": parsed.get("description", ""),
"desired_job_materials": parsed.get("desired_job_materials", []),
}