Added migration notes

MIGRATION-NOTES.md

@@ -0,0 +1,295 @@
+Yep — I inspected the current setup. The “current bash wrapper” is the ServiceM8 plumbing quote-template pipeline under:
+
+`/opt/webhooks`
+
+In the workspace it appears as a symlink:
+
+`/home/openclaw/.openclaw/workspace/projects/plumbing -> /opt/webhooks`
+
+## Definitive migration set
+
+### Required operational scripts
+
+Copy these from `/opt/webhooks/`:
+
+```text
+poll_and_apply_quote_templates.sh
+poll_form_responses_since.py
+apply_polled_quote_template_jobmaterials.py
+servicem8_quote_template_parser.py
+servicem8_webhook_receiver.py
+servicem8_inspector.py
+```
+
+These are the active pieces:
+
+- `poll_and_apply_quote_templates.sh` — main wrapper / scheduled entry point
+- `poll_form_responses_since.py` — polls ServiceM8 form responses
+- `apply_polled_quote_template_jobmaterials.py` — applies parsed quote rows to ServiceM8 jobMaterials
+- `servicem8_quote_template_parser.py` — parsing/state logic
+- `servicem8_webhook_receiver.py` — webhook receiver/diagnostics
+- `servicem8_inspector.py` — web UI inspector
+
+### Required DB/state files
+
+Copy these, ideally while services are stopped:
+
+```text
+servicem8_formresponse_poll.db
+servicem8_quote_materials_state.db
+servicem8_webhooks.db
+```
+
+Also copy any SQLite sidecar files if present at cutover:
+
+```text
+*.db-wal
+*.db-shm
+```
+
+### Required JSONL queue/history files
+
+```text
+quote-template-jobmaterials-poll-queue.jsonl
+quote-template-jobmaterials-queue.jsonl
+```
+
+First one is the current poll-derived queue. Second is older webhook-derived queue but worth keeping for audit/history.
+
+### Useful but not strictly required
+
+```text
+PROJECT-PROGRESS.md
+docs/
+logs/
+.git/
+```
+
+I’d migrate these too unless you want a clean production-only deploy. The logs and docs are useful for future archaeology when something inevitably gets weird.
+
+## Do not blindly migrate
+
+I would **not** migrate the existing virtualenv folders directly:
+
+```text
+bin/
+lib/
+lib64/
+include/
+pyvenv.cfg
+__pycache__/
+```
+
+They are host/Python-version sensitive. Recreate the venv on the new server.
+
+## Python / pip packages
+
+Current venv freeze shows:
+
+```text
+fastapi==0.136.0
+uvicorn==0.45.0
+requests==2.33.1
+python-dotenv==1.2.2
+pydantic==2.13.3
+starlette==1.0.0
+anyio==4.13.0
+click==8.3.3
+h11==0.16.0
+httptools==0.7.1
+uvloop==0.22.1
+watchfiles==1.1.1
+websockets==16.0
+certifi==2026.4.22
+charset-normalizer==3.4.7
+idna==3.13
+urllib3==2.6.3
+PyYAML==6.0.3
+typing_extensions==4.15.0
+typing-inspection==0.4.2
+annotated-types==0.7.0
+annotated-doc==0.0.4
+pydantic_core==2.46.3
+```
+
+Practical install line:
+
+```bash
+python3.12 -m venv /opt/webhooks
+/opt/webhooks/bin/pip install --upgrade pip
+/opt/webhooks/bin/pip install fastapi 'uvicorn[standard]' requests python-dotenv
+```
+
+Or create a `requirements.txt` from the freeze if you want exact locking.
+
+System packages likely needed:
+
+```bash
+python3.12
+python3.12-venv
+sqlite3
+bash
+curl
+rsync
+systemd
+```
+
+Important: the wrapper uses GNU `date -d`, so Linux/GNU coreutils is assumed.
+
+## Environment variables required
+
+For polling/applying:
+
+```text
+SERVICEM8_ACCESS_TOKEN
+# or
+SERVICEM8_API_KEY
+```
+
+Optional/current defaults:
+
+```text
+SERVICEM8_BASE_URL=https://api.servicem8.com/api_1.0
+SERVICEM8_TIMEOUT=30
+SERVICEM8_QUOTE_TEMPLATE_FORM_UUID=3621b6be-1d19-4756-9ab4-9d5e4120f6d9
+WEBHOOK_POLL_DB_PATH=/opt/webhooks/servicem8_formresponse_poll.db
+WEBHOOK_RUN_LOG_DIR=/opt/webhooks/logs
+```
+
+For webhook receiver:
+
+```text
+WEBHOOK_HOST=0.0.0.0
+WEBHOOK_PORT=18354
+WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db
+```
+
+For inspector:
+
+```text
+WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db
+WEBHOOK_STATE_DB_PATH=/opt/webhooks/servicem8_quote_materials_state.db
+WEBHOOK_POLL_DB_PATH=/opt/webhooks/servicem8_formresponse_poll.db
+WEBHOOK_INSPECTOR_HOST=0.0.0.0
+WEBHOOK_INSPECTOR_PORT=18355
+```
+
+## Existing systemd services to recreate
+
+Current services are:
+
+```text
+servicem8-dev-webhook.service
+servicem8-dev-inspector.service
+```
+
+Current paths assume `/opt/webhooks`.
+
+Webhook:
+
+```ini
+WorkingDirectory=/opt/webhooks
+Environment="WEBHOOK_HOST=0.0.0.0"
+Environment="WEBHOOK_PORT=18354"
+Environment="WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db"
+ExecStart=/opt/webhooks/bin/uvicorn servicem8_webhook_receiver:app --host 0.0.0.0 --port 18354
+```
+
+Inspector:
+
+```ini
+WorkingDirectory=/opt/webhooks
+Environment="WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db"
+Environment="WEBHOOK_INSPECTOR_HOST=0.0.0.0"
+Environment="WEBHOOK_INSPECTOR_PORT=18355"
+ExecStart=/opt/webhooks/bin/uvicorn servicem8_inspector:app --host 0.0.0.0 --port 18355
+```
+
+On the new host, probably change:
+
+```ini
+User=michael
+Group=michael
+```
+
+to whatever the production service user is, e.g.:
+
+```ini
+User=openclaw
+Group=openclaw
+```
+
+or create a dedicated `servicem8` user.
+
+## Path/code changes for seamless migration
+
+Best option: keep the same production path:
+
+```text
+/opt/webhooks
+```
+
+If you do that, almost no code changes are needed.
+
+If you move it elsewhere, update:
+
+1. systemd `WorkingDirectory`
+2. systemd `ExecStart`
+3. env vars:
+   - `WEBHOOK_DB_PATH`
+   - `WEBHOOK_POLL_DB_PATH`
+   - `WEBHOOK_STATE_DB_PATH`
+   - `WEBHOOK_RUN_LOG_DIR`
+4. any OpenClaw cron or host cron entry that calls:
+   - `/opt/webhooks/poll_and_apply_quote_templates.sh`
+
+One gotcha: `servicem8_quote_template_parser.py` currently hardcodes the state DB beside the script:
+
+```python
+STATE_DB_PATH = Path(__file__).with_name("servicem8_quote_materials_state.db")
+```
+
+So keeping DBs beside the scripts is the safest path. If you want DBs somewhere else, I’d adjust that code to respect `WEBHOOK_STATE_DB_PATH`.
+
+## External callback / production URL
+
+Current webhook helper scripts still reference dev callback URLs like:
+
+```text
+https://nps-dev.coast2cloud.net/...
+```
+
+For the new production host, ServiceM8 webhook subscriptions need to point at the new public URL if you want inbound webhooks working.
+
+Polling does **not** depend on inbound webhooks, so the wrapper can operate without this, but the receiver/inspector diagnostics won’t capture new webhook events unless ServiceM8 is updated.
+
+## Security note
+
+There are old helper scripts with hardcoded ServiceM8 token fallbacks:
+
+```text
+servicem8-create-webhook-form-response.py
+servicem8-create-webhook-job-object.py
+servicem8-list-webhook-subscriptions-table.py
+```
+
+I would not treat those as production-safe as-is. Before migration to prod, remove hardcoded fallback tokens and require env vars only.
+
+## Recommended cutover shape
+
+1. Stop current webhook/inspector services.
+2. Copy `/opt/webhooks` excluding venv/cache, including DBs/jsonl/logs.
+3. Recreate venv on new host.
+4. Install pip packages.
+5. Add env/secrets via systemd `EnvironmentFile`, not hardcoded scripts.
+6. Recreate systemd services.
+7. Run:
+
+```bash
+/opt/webhooks/poll_and_apply_quote_templates.sh --dry-run --hours 48
+```
+
+8. If clean, run controlled live apply.
+9. Only then schedule the wrapper.
+
+At present I found no OpenClaw cron job already running this plumbing wrapper, so scheduling still appears to be a pending production decision rather than something to migrate from OpenClaw cron.

PROJECT-PROGRESS.md

@@ -73,12 +73,17 @@ Current apply payload rules:
   - stores/parses results
   - applies any parsed Quote Template responses that are not already marked/applied
   - logs each run under `logs/poll-and-apply-YYYYMMDD-HHMMSS.log`
+- Safety option now available:
+  - `--dry-run` runs the same poll/selection flow, but previews the ServiceM8 `jobmaterial` payloads only
+  - dry-run does **not** write to ServiceM8
+  - dry-run does **not** mark quote responses as applied
 - Examples:
   - `./poll_and_apply_quote_templates.sh`
   - `./poll_and_apply_quote_templates.sh --hours 48`
   - `./poll_and_apply_quote_templates.sh --since '2026-05-04 08:00:00'`
+  - `./poll_and_apply_quote_templates.sh --dry-run --hours 48`
 
-This is the proposed scheduled entry point for soft release, e.g. every 10–30 minutes.
+This is the proposed scheduled entry point for soft release, e.g. every 10–30 minutes. For manual confidence checks, run it with `--dry-run` first, inspect the generated payloads/log, then rerun without `--dry-run` only when ready to apply.
 
 ## Live Webhook Receiver Status
 
@@ -140,13 +145,29 @@ Operational soft-release pieces are now in place:
 - payload adjusted for current ServiceM8 requirements
 - duplicate-apply guard in place
 - wrapper script created for scheduled operation
+- wrapper now has a first-class `--dry-run` mode
 - inspector updated for progress visibility
 
+### Checkpoint — 2026-05-05
+
+Latest verified state:
+- `poll_and_apply_quote_templates.sh --dry-run` added and documented.
+- `bash -n /opt/webhooks/poll_and_apply_quote_templates.sh` passes.
+- `--help` output includes `--dry-run` usage/examples.
+- A future-since dry-run (`--dry-run --since '2099-01-01 00:00:00'`) confirmed:
+  - wrapper reports `Mode: dry-run`
+  - poll step performs no ServiceM8 writes
+  - apply step calls the Python apply tool without `--apply`
+  - payload rows are emitted as `would_create`
+  - responses are not marked applied by dry-run
+- At checkpoint time, there were still several unapplied parsed Quote Template responses available for preview/apply; this is expected while the soft release remains manual.
+
 ## Not Yet Done / Next Steps
 
 - Restart/reload the live inspector process so the new poll/apply pages are available in the active web viewer.
 - Decide schedule interval for `poll_and_apply_quote_templates.sh` — likely every 10 or 30 minutes.
-- Run the wrapper manually for a soft-release smoke test with a controlled recent form response.
+- Run the wrapper manually in `--dry-run` mode against a controlled recent form response and inspect the payload/log.
+- If payload is correct, rerun the wrapper without `--dry-run` for a controlled live apply smoke test.
 - After confidence builds, wire the wrapper into cron/system scheduling.
 - Future hardening: add reconciliation/update/delete behaviour if ServiceM8 quote form responses are edited after initial apply.
 
@@ -155,4 +176,5 @@ Operational soft-release pieces are now in place:
 - Webhooks remain lightweight and non-mutating.
 - Polling is now the reliable source of completeness.
 - Applying to ServiceM8 is tracked locally and guarded against duplicates.
-- The wrapper intentionally skips dry-run for soft release, but the underlying apply script still supports dry-run and duplicate protection.
+- The wrapper defaults to live apply for scheduled soft release, but now supports `--dry-run` for manual preview/safety checks.
+- The underlying apply script remains dry-run-by-default and provides the duplicate protection used by the wrapper.

poll_and_apply_quote_templates.sh

@@ -9,8 +9,10 @@ set -euo pipefail
 #   --since 'YYYY-MM-DD HH:MM:SS'
 #   --hours 48
 #
-# This wrapper intentionally skips dry-run and calls --apply. The apply script
-# still refuses duplicate applies unless --force is explicitly passed through.
+# By default this wrapper applies unapplied parsed responses. Use --dry-run to
+# run the poll and preview each pending apply without writing to ServiceM8.
+# The apply script still refuses duplicate applies unless --force is explicitly
+# passed through.
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 POLL_SCRIPT="$SCRIPT_DIR/poll_form_responses_since.py"
@@ -22,20 +24,25 @@ QUOTE_TEMPLATE_FORM_UUID="${SERVICEM8_QUOTE_TEMPLATE_FORM_UUID:-3621b6be-1d19-47
 SINCE=""
 HOURS="24"
 FORCE="0"
+DRY_RUN="0"
 
 usage() {
   cat <<EOF
-Usage: $0 [--since 'YYYY-MM-DD HH:MM:SS'] [--hours N] [--force]
+Usage: $0 [--since 'YYYY-MM-DD HH:MM:SS'] [--hours N] [--dry-run] [--force]
 
 Examples:
   $0
   $0 --hours 48
   $0 --since '2026-05-04 08:00:00'
+  $0 --dry-run --hours 48
 
 This will:
   1. Poll /formresponse.json using timestamp gt SINCE
   2. Store/parse Quote Template responses into $DB_PATH
   3. Apply parsed responses that do not already have generated materials recorded
+
+With --dry-run, step 3 previews the ServiceM8 jobMaterial payloads only; it does
+not write to ServiceM8 or mark responses as applied.
 EOF
 }
 
@@ -53,6 +60,10 @@ while [[ $# -gt 0 ]]; do
       FORCE="1"
       shift
       ;;
+    --dry-run)
+      DRY_RUN="1"
+      shift
+      ;;
     -h|--help)
       usage
       exit 0
@@ -78,6 +89,7 @@ exec > >(tee -a "$LOG_FILE") 2>&1
 echo "== ServiceM8 Quote Template poll/apply run =="
 echo "Started: $(date --iso-8601=seconds)"
 echo "Since:   $SINCE"
+echo "Mode:    $([[ "$DRY_RUN" == "1" ]] && echo "dry-run" || echo "apply")"
 echo "DB:      $DB_PATH"
 echo "Log:     $LOG_FILE"
 echo
@@ -121,15 +133,24 @@ printf 'Found %d unapplied Quote Template response(s):\n' "${#FORM_RESPONSE_UUID
 printf ' - %s\n' "${FORM_RESPONSE_UUIDS[@]}"
 
 echo
-echo "== Applying to ServiceM8 =="
-APPLY_ARGS=(--apply --pretty)
+if [[ "$DRY_RUN" == "1" ]]; then
+  echo "== Dry-run preview only; no ServiceM8 writes =="
+  APPLY_ARGS=(--pretty)
+else
+  echo "== Applying to ServiceM8 =="
+  APPLY_ARGS=(--apply --pretty)
+fi
 if [[ "$FORCE" == "1" ]]; then
   APPLY_ARGS+=(--force)
 fi
 
 for uuid in "${FORM_RESPONSE_UUIDS[@]}"; do
   echo
-  echo "-- Applying form_response_uuid=$uuid --"
+  if [[ "$DRY_RUN" == "1" ]]; then
+    echo "-- Dry-run form_response_uuid=$uuid --"
+  else
+    echo "-- Applying form_response_uuid=$uuid --"
+  fi
   "$APPLY_SCRIPT" --uuid "$uuid" "${APPLY_ARGS[@]}"
 done
 

pyvenv.cfg

@@ -1,5 +0,0 @@
-home = /usr/bin
-include-system-site-packages = false
-version = 3.12.3
-executable = /usr/bin/python3.12
-command = /usr/bin/python3 -m venv /opt/webhooks