From f03840c5744f0e54f33b0002053933b6df59a5a0 Mon Sep 17 00:00:00 2001 From: "Soren (Molty)" Date: Mon, 11 May 2026 07:19:52 +1000 Subject: [PATCH] Added migration notes --- MIGRATION-NOTES.md | 295 +++++++++++++++++++++++++++++++++++++++++++++ pyvenv.cfg | 5 - 2 files changed, 295 insertions(+), 5 deletions(-) create mode 100644 MIGRATION-NOTES.md delete mode 100755 pyvenv.cfg diff --git a/MIGRATION-NOTES.md b/MIGRATION-NOTES.md new file mode 100644 index 0000000..5137c1c --- /dev/null +++ b/MIGRATION-NOTES.md @@ -0,0 +1,295 @@ +Yep — I inspected the current setup. The “current bash wrapper” is the ServiceM8 plumbing quote-template pipeline under: + +`/opt/webhooks` + +In the workspace it appears as a symlink: + +`/home/openclaw/.openclaw/workspace/projects/plumbing -> /opt/webhooks` + +## Definitive migration set + +### Required operational scripts + +Copy these from `/opt/webhooks/`: + +```text +poll_and_apply_quote_templates.sh +poll_form_responses_since.py +apply_polled_quote_template_jobmaterials.py +servicem8_quote_template_parser.py +servicem8_webhook_receiver.py +servicem8_inspector.py +``` + +These are the active pieces: + +- `poll_and_apply_quote_templates.sh` — main wrapper / scheduled entry point +- `poll_form_responses_since.py` — polls ServiceM8 form responses +- `apply_polled_quote_template_jobmaterials.py` — applies parsed quote rows to ServiceM8 jobMaterials +- `servicem8_quote_template_parser.py` — parsing/state logic +- `servicem8_webhook_receiver.py` — webhook receiver/diagnostics +- `servicem8_inspector.py` — web UI inspector + +### Required DB/state files + +Copy these, ideally while services are stopped: + +```text +servicem8_formresponse_poll.db +servicem8_quote_materials_state.db +servicem8_webhooks.db +``` + +Also copy any SQLite sidecar files if present at cutover: + +```text +*.db-wal +*.db-shm +``` + +### Required JSONL queue/history files + +```text +quote-template-jobmaterials-poll-queue.jsonl +quote-template-jobmaterials-queue.jsonl +``` + +First one is the current poll-derived queue. Second is older webhook-derived queue but worth keeping for audit/history. + +### Useful but not strictly required + +```text +PROJECT-PROGRESS.md +docs/ +logs/ +.git/ +``` + +I’d migrate these too unless you want a clean production-only deploy. The logs and docs are useful for future archaeology when something inevitably gets weird. + +## Do not blindly migrate + +I would **not** migrate the existing virtualenv folders directly: + +```text +bin/ +lib/ +lib64/ +include/ +pyvenv.cfg +__pycache__/ +``` + +They are host/Python-version sensitive. Recreate the venv on the new server. + +## Python / pip packages + +Current venv freeze shows: + +```text +fastapi==0.136.0 +uvicorn==0.45.0 +requests==2.33.1 +python-dotenv==1.2.2 +pydantic==2.13.3 +starlette==1.0.0 +anyio==4.13.0 +click==8.3.3 +h11==0.16.0 +httptools==0.7.1 +uvloop==0.22.1 +watchfiles==1.1.1 +websockets==16.0 +certifi==2026.4.22 +charset-normalizer==3.4.7 +idna==3.13 +urllib3==2.6.3 +PyYAML==6.0.3 +typing_extensions==4.15.0 +typing-inspection==0.4.2 +annotated-types==0.7.0 +annotated-doc==0.0.4 +pydantic_core==2.46.3 +``` + +Practical install line: + +```bash +python3.12 -m venv /opt/webhooks +/opt/webhooks/bin/pip install --upgrade pip +/opt/webhooks/bin/pip install fastapi 'uvicorn[standard]' requests python-dotenv +``` + +Or create a `requirements.txt` from the freeze if you want exact locking. + +System packages likely needed: + +```bash +python3.12 +python3.12-venv +sqlite3 +bash +curl +rsync +systemd +``` + +Important: the wrapper uses GNU `date -d`, so Linux/GNU coreutils is assumed. + +## Environment variables required + +For polling/applying: + +```text +SERVICEM8_ACCESS_TOKEN +# or +SERVICEM8_API_KEY +``` + +Optional/current defaults: + +```text +SERVICEM8_BASE_URL=https://api.servicem8.com/api_1.0 +SERVICEM8_TIMEOUT=30 +SERVICEM8_QUOTE_TEMPLATE_FORM_UUID=3621b6be-1d19-4756-9ab4-9d5e4120f6d9 +WEBHOOK_POLL_DB_PATH=/opt/webhooks/servicem8_formresponse_poll.db +WEBHOOK_RUN_LOG_DIR=/opt/webhooks/logs +``` + +For webhook receiver: + +```text +WEBHOOK_HOST=0.0.0.0 +WEBHOOK_PORT=18354 +WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db +``` + +For inspector: + +```text +WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db +WEBHOOK_STATE_DB_PATH=/opt/webhooks/servicem8_quote_materials_state.db +WEBHOOK_POLL_DB_PATH=/opt/webhooks/servicem8_formresponse_poll.db +WEBHOOK_INSPECTOR_HOST=0.0.0.0 +WEBHOOK_INSPECTOR_PORT=18355 +``` + +## Existing systemd services to recreate + +Current services are: + +```text +servicem8-dev-webhook.service +servicem8-dev-inspector.service +``` + +Current paths assume `/opt/webhooks`. + +Webhook: + +```ini +WorkingDirectory=/opt/webhooks +Environment="WEBHOOK_HOST=0.0.0.0" +Environment="WEBHOOK_PORT=18354" +Environment="WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db" +ExecStart=/opt/webhooks/bin/uvicorn servicem8_webhook_receiver:app --host 0.0.0.0 --port 18354 +``` + +Inspector: + +```ini +WorkingDirectory=/opt/webhooks +Environment="WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db" +Environment="WEBHOOK_INSPECTOR_HOST=0.0.0.0" +Environment="WEBHOOK_INSPECTOR_PORT=18355" +ExecStart=/opt/webhooks/bin/uvicorn servicem8_inspector:app --host 0.0.0.0 --port 18355 +``` + +On the new host, probably change: + +```ini +User=michael +Group=michael +``` + +to whatever the production service user is, e.g.: + +```ini +User=openclaw +Group=openclaw +``` + +or create a dedicated `servicem8` user. + +## Path/code changes for seamless migration + +Best option: keep the same production path: + +```text +/opt/webhooks +``` + +If you do that, almost no code changes are needed. + +If you move it elsewhere, update: + +1. systemd `WorkingDirectory` +2. systemd `ExecStart` +3. env vars: + - `WEBHOOK_DB_PATH` + - `WEBHOOK_POLL_DB_PATH` + - `WEBHOOK_STATE_DB_PATH` + - `WEBHOOK_RUN_LOG_DIR` +4. any OpenClaw cron or host cron entry that calls: + - `/opt/webhooks/poll_and_apply_quote_templates.sh` + +One gotcha: `servicem8_quote_template_parser.py` currently hardcodes the state DB beside the script: + +```python +STATE_DB_PATH = Path(__file__).with_name("servicem8_quote_materials_state.db") +``` + +So keeping DBs beside the scripts is the safest path. If you want DBs somewhere else, I’d adjust that code to respect `WEBHOOK_STATE_DB_PATH`. + +## External callback / production URL + +Current webhook helper scripts still reference dev callback URLs like: + +```text +https://nps-dev.coast2cloud.net/... +``` + +For the new production host, ServiceM8 webhook subscriptions need to point at the new public URL if you want inbound webhooks working. + +Polling does **not** depend on inbound webhooks, so the wrapper can operate without this, but the receiver/inspector diagnostics won’t capture new webhook events unless ServiceM8 is updated. + +## Security note + +There are old helper scripts with hardcoded ServiceM8 token fallbacks: + +```text +servicem8-create-webhook-form-response.py +servicem8-create-webhook-job-object.py +servicem8-list-webhook-subscriptions-table.py +``` + +I would not treat those as production-safe as-is. Before migration to prod, remove hardcoded fallback tokens and require env vars only. + +## Recommended cutover shape + +1. Stop current webhook/inspector services. +2. Copy `/opt/webhooks` excluding venv/cache, including DBs/jsonl/logs. +3. Recreate venv on new host. +4. Install pip packages. +5. Add env/secrets via systemd `EnvironmentFile`, not hardcoded scripts. +6. Recreate systemd services. +7. Run: + +```bash +/opt/webhooks/poll_and_apply_quote_templates.sh --dry-run --hours 48 +``` + +8. If clean, run controlled live apply. +9. Only then schedule the wrapper. + +At present I found no OpenClaw cron job already running this plumbing wrapper, so scheduling still appears to be a pending production decision rather than something to migrate from OpenClaw cron. diff --git a/pyvenv.cfg b/pyvenv.cfg deleted file mode 100755 index f50c12c..0000000 --- a/pyvenv.cfg +++ /dev/null @@ -1,5 +0,0 @@ -home = /usr/bin -include-system-site-packages = false -version = 3.12.3 -executable = /usr/bin/python3.12 -command = /usr/bin/python3 -m venv /opt/webhooks