296 lines
6.9 KiB
Markdown
296 lines
6.9 KiB
Markdown
Yep — I inspected the current setup. The “current bash wrapper” is the ServiceM8 plumbing quote-template pipeline under:
|
||
|
||
`/opt/webhooks`
|
||
|
||
In the workspace it appears as a symlink:
|
||
|
||
`/home/openclaw/.openclaw/workspace/projects/plumbing -> /opt/webhooks`
|
||
|
||
## Definitive migration set
|
||
|
||
### Required operational scripts
|
||
|
||
Copy these from `/opt/webhooks/`:
|
||
|
||
```text
|
||
poll_and_apply_quote_templates.sh
|
||
poll_form_responses_since.py
|
||
apply_polled_quote_template_jobmaterials.py
|
||
servicem8_quote_template_parser.py
|
||
servicem8_webhook_receiver.py
|
||
servicem8_inspector.py
|
||
```
|
||
|
||
These are the active pieces:
|
||
|
||
- `poll_and_apply_quote_templates.sh` — main wrapper / scheduled entry point
|
||
- `poll_form_responses_since.py` — polls ServiceM8 form responses
|
||
- `apply_polled_quote_template_jobmaterials.py` — applies parsed quote rows to ServiceM8 jobMaterials
|
||
- `servicem8_quote_template_parser.py` — parsing/state logic
|
||
- `servicem8_webhook_receiver.py` — webhook receiver/diagnostics
|
||
- `servicem8_inspector.py` — web UI inspector
|
||
|
||
### Required DB/state files
|
||
|
||
Copy these, ideally while services are stopped:
|
||
|
||
```text
|
||
servicem8_formresponse_poll.db
|
||
servicem8_quote_materials_state.db
|
||
servicem8_webhooks.db
|
||
```
|
||
|
||
Also copy any SQLite sidecar files if present at cutover:
|
||
|
||
```text
|
||
*.db-wal
|
||
*.db-shm
|
||
```
|
||
|
||
### Required JSONL queue/history files
|
||
|
||
```text
|
||
quote-template-jobmaterials-poll-queue.jsonl
|
||
quote-template-jobmaterials-queue.jsonl
|
||
```
|
||
|
||
First one is the current poll-derived queue. Second is older webhook-derived queue but worth keeping for audit/history.
|
||
|
||
### Useful but not strictly required
|
||
|
||
```text
|
||
PROJECT-PROGRESS.md
|
||
docs/
|
||
logs/
|
||
.git/
|
||
```
|
||
|
||
I’d migrate these too unless you want a clean production-only deploy. The logs and docs are useful for future archaeology when something inevitably gets weird.
|
||
|
||
## Do not blindly migrate
|
||
|
||
I would **not** migrate the existing virtualenv folders directly:
|
||
|
||
```text
|
||
bin/
|
||
lib/
|
||
lib64/
|
||
include/
|
||
pyvenv.cfg
|
||
__pycache__/
|
||
```
|
||
|
||
They are host/Python-version sensitive. Recreate the venv on the new server.
|
||
|
||
## Python / pip packages
|
||
|
||
Current venv freeze shows:
|
||
|
||
```text
|
||
fastapi==0.136.0
|
||
uvicorn==0.45.0
|
||
requests==2.33.1
|
||
python-dotenv==1.2.2
|
||
pydantic==2.13.3
|
||
starlette==1.0.0
|
||
anyio==4.13.0
|
||
click==8.3.3
|
||
h11==0.16.0
|
||
httptools==0.7.1
|
||
uvloop==0.22.1
|
||
watchfiles==1.1.1
|
||
websockets==16.0
|
||
certifi==2026.4.22
|
||
charset-normalizer==3.4.7
|
||
idna==3.13
|
||
urllib3==2.6.3
|
||
PyYAML==6.0.3
|
||
typing_extensions==4.15.0
|
||
typing-inspection==0.4.2
|
||
annotated-types==0.7.0
|
||
annotated-doc==0.0.4
|
||
pydantic_core==2.46.3
|
||
```
|
||
|
||
Practical install line:
|
||
|
||
```bash
|
||
python3.12 -m venv /opt/webhooks
|
||
/opt/webhooks/bin/pip install --upgrade pip
|
||
/opt/webhooks/bin/pip install fastapi 'uvicorn[standard]' requests python-dotenv
|
||
```
|
||
|
||
Or create a `requirements.txt` from the freeze if you want exact locking.
|
||
|
||
System packages likely needed:
|
||
|
||
```bash
|
||
python3.12
|
||
python3.12-venv
|
||
sqlite3
|
||
bash
|
||
curl
|
||
rsync
|
||
systemd
|
||
```
|
||
|
||
Important: the wrapper uses GNU `date -d`, so Linux/GNU coreutils is assumed.
|
||
|
||
## Environment variables required
|
||
|
||
For polling/applying:
|
||
|
||
```text
|
||
SERVICEM8_ACCESS_TOKEN
|
||
# or
|
||
SERVICEM8_API_KEY
|
||
```
|
||
|
||
Optional/current defaults:
|
||
|
||
```text
|
||
SERVICEM8_BASE_URL=https://api.servicem8.com/api_1.0
|
||
SERVICEM8_TIMEOUT=30
|
||
SERVICEM8_QUOTE_TEMPLATE_FORM_UUID=3621b6be-1d19-4756-9ab4-9d5e4120f6d9
|
||
WEBHOOK_POLL_DB_PATH=/opt/webhooks/servicem8_formresponse_poll.db
|
||
WEBHOOK_RUN_LOG_DIR=/opt/webhooks/logs
|
||
```
|
||
|
||
For webhook receiver:
|
||
|
||
```text
|
||
WEBHOOK_HOST=0.0.0.0
|
||
WEBHOOK_PORT=18354
|
||
WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db
|
||
```
|
||
|
||
For inspector:
|
||
|
||
```text
|
||
WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db
|
||
WEBHOOK_STATE_DB_PATH=/opt/webhooks/servicem8_quote_materials_state.db
|
||
WEBHOOK_POLL_DB_PATH=/opt/webhooks/servicem8_formresponse_poll.db
|
||
WEBHOOK_INSPECTOR_HOST=0.0.0.0
|
||
WEBHOOK_INSPECTOR_PORT=18355
|
||
```
|
||
|
||
## Existing systemd services to recreate
|
||
|
||
Current services are:
|
||
|
||
```text
|
||
servicem8-dev-webhook.service
|
||
servicem8-dev-inspector.service
|
||
```
|
||
|
||
Current paths assume `/opt/webhooks`.
|
||
|
||
Webhook:
|
||
|
||
```ini
|
||
WorkingDirectory=/opt/webhooks
|
||
Environment="WEBHOOK_HOST=0.0.0.0"
|
||
Environment="WEBHOOK_PORT=18354"
|
||
Environment="WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db"
|
||
ExecStart=/opt/webhooks/bin/uvicorn servicem8_webhook_receiver:app --host 0.0.0.0 --port 18354
|
||
```
|
||
|
||
Inspector:
|
||
|
||
```ini
|
||
WorkingDirectory=/opt/webhooks
|
||
Environment="WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db"
|
||
Environment="WEBHOOK_INSPECTOR_HOST=0.0.0.0"
|
||
Environment="WEBHOOK_INSPECTOR_PORT=18355"
|
||
ExecStart=/opt/webhooks/bin/uvicorn servicem8_inspector:app --host 0.0.0.0 --port 18355
|
||
```
|
||
|
||
On the new host, probably change:
|
||
|
||
```ini
|
||
User=michael
|
||
Group=michael
|
||
```
|
||
|
||
to whatever the production service user is, e.g.:
|
||
|
||
```ini
|
||
User=openclaw
|
||
Group=openclaw
|
||
```
|
||
|
||
or create a dedicated `servicem8` user.
|
||
|
||
## Path/code changes for seamless migration
|
||
|
||
Best option: keep the same production path:
|
||
|
||
```text
|
||
/opt/webhooks
|
||
```
|
||
|
||
If you do that, almost no code changes are needed.
|
||
|
||
If you move it elsewhere, update:
|
||
|
||
1. systemd `WorkingDirectory`
|
||
2. systemd `ExecStart`
|
||
3. env vars:
|
||
- `WEBHOOK_DB_PATH`
|
||
- `WEBHOOK_POLL_DB_PATH`
|
||
- `WEBHOOK_STATE_DB_PATH`
|
||
- `WEBHOOK_RUN_LOG_DIR`
|
||
4. any OpenClaw cron or host cron entry that calls:
|
||
- `/opt/webhooks/poll_and_apply_quote_templates.sh`
|
||
|
||
One gotcha: `servicem8_quote_template_parser.py` currently hardcodes the state DB beside the script:
|
||
|
||
```python
|
||
STATE_DB_PATH = Path(__file__).with_name("servicem8_quote_materials_state.db")
|
||
```
|
||
|
||
So keeping DBs beside the scripts is the safest path. If you want DBs somewhere else, I’d adjust that code to respect `WEBHOOK_STATE_DB_PATH`.
|
||
|
||
## External callback / production URL
|
||
|
||
Current webhook helper scripts still reference dev callback URLs like:
|
||
|
||
```text
|
||
https://nps-dev.coast2cloud.net/...
|
||
```
|
||
|
||
For the new production host, ServiceM8 webhook subscriptions need to point at the new public URL if you want inbound webhooks working.
|
||
|
||
Polling does **not** depend on inbound webhooks, so the wrapper can operate without this, but the receiver/inspector diagnostics won’t capture new webhook events unless ServiceM8 is updated.
|
||
|
||
## Security note
|
||
|
||
There are old helper scripts with hardcoded ServiceM8 token fallbacks:
|
||
|
||
```text
|
||
servicem8-create-webhook-form-response.py
|
||
servicem8-create-webhook-job-object.py
|
||
servicem8-list-webhook-subscriptions-table.py
|
||
```
|
||
|
||
I would not treat those as production-safe as-is. Before migration to prod, remove hardcoded fallback tokens and require env vars only.
|
||
|
||
## Recommended cutover shape
|
||
|
||
1. Stop current webhook/inspector services.
|
||
2. Copy `/opt/webhooks` excluding venv/cache, including DBs/jsonl/logs.
|
||
3. Recreate venv on new host.
|
||
4. Install pip packages.
|
||
5. Add env/secrets via systemd `EnvironmentFile`, not hardcoded scripts.
|
||
6. Recreate systemd services.
|
||
7. Run:
|
||
|
||
```bash
|
||
/opt/webhooks/poll_and_apply_quote_templates.sh --dry-run --hours 48
|
||
```
|
||
|
||
8. If clean, run controlled live apply.
|
||
9. Only then schedule the wrapper.
|
||
|
||
At present I found no OpenClaw cron job already running this plumbing wrapper, so scheduling still appears to be a pending production decision rather than something to migrate from OpenClaw cron.
|