plumbing/MIGRATION-NOTES.md

Yep — I inspected the current setup. The “current bash wrapper” is the ServiceM8 plumbing quote-template pipeline under:

/opt/webhooks

In the workspace it appears as a symlink:

/home/openclaw/.openclaw/workspace/projects/plumbing -> /opt/webhooks

Definitive migration set

Required operational scripts

Copy these from /opt/webhooks/:

poll_and_apply_quote_templates.sh
poll_form_responses_since.py
apply_polled_quote_template_jobmaterials.py
servicem8_quote_template_parser.py
servicem8_webhook_receiver.py
servicem8_inspector.py

These are the active pieces:

• poll_and_apply_quote_templates.sh — main wrapper / scheduled entry point
• poll_form_responses_since.py — polls ServiceM8 form responses
• apply_polled_quote_template_jobmaterials.py — applies parsed quote rows to ServiceM8 jobMaterials
• servicem8_quote_template_parser.py — parsing/state logic
• servicem8_webhook_receiver.py — webhook receiver/diagnostics
• servicem8_inspector.py — web UI inspector

Required DB/state files

Copy these, ideally while services are stopped:

servicem8_formresponse_poll.db
servicem8_quote_materials_state.db
servicem8_webhooks.db

Also copy any SQLite sidecar files if present at cutover:

*.db-wal
*.db-shm

Required JSONL queue/history files

quote-template-jobmaterials-poll-queue.jsonl
quote-template-jobmaterials-queue.jsonl

First one is the current poll-derived queue. Second is older webhook-derived queue but worth keeping for audit/history.

Useful but not strictly required

PROJECT-PROGRESS.md
docs/
logs/
.git/

I’d migrate these too unless you want a clean production-only deploy. The logs and docs are useful for future archaeology when something inevitably gets weird.

Do not blindly migrate

I would not migrate the existing virtualenv folders directly:

bin/
lib/
lib64/
include/
pyvenv.cfg
__pycache__/

They are host/Python-version sensitive. Recreate the venv on the new server.

Python / pip packages

Current venv freeze shows:

fastapi==0.136.0
uvicorn==0.45.0
requests==2.33.1
python-dotenv==1.2.2
pydantic==2.13.3
starlette==1.0.0
anyio==4.13.0
click==8.3.3
h11==0.16.0
httptools==0.7.1
uvloop==0.22.1
watchfiles==1.1.1
websockets==16.0
certifi==2026.4.22
charset-normalizer==3.4.7
idna==3.13
urllib3==2.6.3
PyYAML==6.0.3
typing_extensions==4.15.0
typing-inspection==0.4.2
annotated-types==0.7.0
annotated-doc==0.0.4
pydantic_core==2.46.3

Practical install line:

python3.12 -m venv /opt/webhooks
/opt/webhooks/bin/pip install --upgrade pip
/opt/webhooks/bin/pip install fastapi 'uvicorn[standard]' requests python-dotenv

Or create a requirements.txt from the freeze if you want exact locking.

System packages likely needed:

python3.12
python3.12-venv
sqlite3
bash
curl
rsync
systemd

Important: the wrapper uses GNU date -d, so Linux/GNU coreutils is assumed.

Environment variables required

For polling/applying:

SERVICEM8_ACCESS_TOKEN
# or
SERVICEM8_API_KEY

Optional/current defaults:

SERVICEM8_BASE_URL=https://api.servicem8.com/api_1.0
SERVICEM8_TIMEOUT=30
SERVICEM8_QUOTE_TEMPLATE_FORM_UUID=3621b6be-1d19-4756-9ab4-9d5e4120f6d9
WEBHOOK_POLL_DB_PATH=/opt/webhooks/servicem8_formresponse_poll.db
WEBHOOK_RUN_LOG_DIR=/opt/webhooks/logs

For webhook receiver:

WEBHOOK_HOST=0.0.0.0
WEBHOOK_PORT=18354
WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db

For inspector:

WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db
WEBHOOK_STATE_DB_PATH=/opt/webhooks/servicem8_quote_materials_state.db
WEBHOOK_POLL_DB_PATH=/opt/webhooks/servicem8_formresponse_poll.db
WEBHOOK_INSPECTOR_HOST=0.0.0.0
WEBHOOK_INSPECTOR_PORT=18355

Existing systemd services to recreate

Current services are:

servicem8-dev-webhook.service
servicem8-dev-inspector.service

Current paths assume /opt/webhooks.

Webhook:

WorkingDirectory=/opt/webhooks
Environment="WEBHOOK_HOST=0.0.0.0"
Environment="WEBHOOK_PORT=18354"
Environment="WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db"
ExecStart=/opt/webhooks/bin/uvicorn servicem8_webhook_receiver:app --host 0.0.0.0 --port 18354

Inspector:

WorkingDirectory=/opt/webhooks
Environment="WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db"
Environment="WEBHOOK_INSPECTOR_HOST=0.0.0.0"
Environment="WEBHOOK_INSPECTOR_PORT=18355"
ExecStart=/opt/webhooks/bin/uvicorn servicem8_inspector:app --host 0.0.0.0 --port 18355

On the new host, probably change:

User=michael
Group=michael

to whatever the production service user is, e.g.:

User=openclaw
Group=openclaw

or create a dedicated servicem8 user.

Path/code changes for seamless migration

Best option: keep the same production path:

/opt/webhooks

If you do that, almost no code changes are needed.

If you move it elsewhere, update:

1. systemd WorkingDirectory
2. systemd ExecStart
3. env vars:
   • WEBHOOK_DB_PATH
   • WEBHOOK_POLL_DB_PATH
   • WEBHOOK_STATE_DB_PATH
   • WEBHOOK_RUN_LOG_DIR
4. any OpenClaw cron or host cron entry that calls:
   • /opt/webhooks/poll_and_apply_quote_templates.sh

One gotcha: servicem8_quote_template_parser.py currently hardcodes the state DB beside the script:

STATE_DB_PATH = Path(__file__).with_name("servicem8_quote_materials_state.db")

So keeping DBs beside the scripts is the safest path. If you want DBs somewhere else, I’d adjust that code to respect WEBHOOK_STATE_DB_PATH.

External callback / production URL

Current webhook helper scripts still reference dev callback URLs like:

https://nps-dev.coast2cloud.net/...

For the new production host, ServiceM8 webhook subscriptions need to point at the new public URL if you want inbound webhooks working.

Polling does not depend on inbound webhooks, so the wrapper can operate without this, but the receiver/inspector diagnostics won’t capture new webhook events unless ServiceM8 is updated.

Security note

There are old helper scripts with hardcoded ServiceM8 token fallbacks:

servicem8-create-webhook-form-response.py
servicem8-create-webhook-job-object.py
servicem8-list-webhook-subscriptions-table.py

I would not treat those as production-safe as-is. Before migration to prod, remove hardcoded fallback tokens and require env vars only.

Recommended cutover shape

1. Stop current webhook/inspector services.
2. Copy /opt/webhooks excluding venv/cache, including DBs/jsonl/logs.
3. Recreate venv on new host.
4. Install pip packages.
5. Add env/secrets via systemd EnvironmentFile, not hardcoded scripts.
6. Recreate systemd services.
7. Run:

/opt/webhooks/poll_and_apply_quote_templates.sh --dry-run --hours 48

8. If clean, run controlled live apply.
9. Only then schedule the wrapper.

At present I found no OpenClaw cron job already running this plumbing wrapper, so scheduling still appears to be a pending production decision rather than something to migrate from OpenClaw cron.