Added migration notes
This commit is contained in:
@@ -0,0 +1,295 @@
|
|||||||
|
Yep — I inspected the current setup. The “current bash wrapper” is the ServiceM8 plumbing quote-template pipeline under:
|
||||||
|
|
||||||
|
`/opt/webhooks`
|
||||||
|
|
||||||
|
In the workspace it appears as a symlink:
|
||||||
|
|
||||||
|
`/home/openclaw/.openclaw/workspace/projects/plumbing -> /opt/webhooks`
|
||||||
|
|
||||||
|
## Definitive migration set
|
||||||
|
|
||||||
|
### Required operational scripts
|
||||||
|
|
||||||
|
Copy these from `/opt/webhooks/`:
|
||||||
|
|
||||||
|
```text
|
||||||
|
poll_and_apply_quote_templates.sh
|
||||||
|
poll_form_responses_since.py
|
||||||
|
apply_polled_quote_template_jobmaterials.py
|
||||||
|
servicem8_quote_template_parser.py
|
||||||
|
servicem8_webhook_receiver.py
|
||||||
|
servicem8_inspector.py
|
||||||
|
```
|
||||||
|
|
||||||
|
These are the active pieces:
|
||||||
|
|
||||||
|
- `poll_and_apply_quote_templates.sh` — main wrapper / scheduled entry point
|
||||||
|
- `poll_form_responses_since.py` — polls ServiceM8 form responses
|
||||||
|
- `apply_polled_quote_template_jobmaterials.py` — applies parsed quote rows to ServiceM8 jobMaterials
|
||||||
|
- `servicem8_quote_template_parser.py` — parsing/state logic
|
||||||
|
- `servicem8_webhook_receiver.py` — webhook receiver/diagnostics
|
||||||
|
- `servicem8_inspector.py` — web UI inspector
|
||||||
|
|
||||||
|
### Required DB/state files
|
||||||
|
|
||||||
|
Copy these, ideally while services are stopped:
|
||||||
|
|
||||||
|
```text
|
||||||
|
servicem8_formresponse_poll.db
|
||||||
|
servicem8_quote_materials_state.db
|
||||||
|
servicem8_webhooks.db
|
||||||
|
```
|
||||||
|
|
||||||
|
Also copy any SQLite sidecar files if present at cutover:
|
||||||
|
|
||||||
|
```text
|
||||||
|
*.db-wal
|
||||||
|
*.db-shm
|
||||||
|
```
|
||||||
|
|
||||||
|
### Required JSONL queue/history files
|
||||||
|
|
||||||
|
```text
|
||||||
|
quote-template-jobmaterials-poll-queue.jsonl
|
||||||
|
quote-template-jobmaterials-queue.jsonl
|
||||||
|
```
|
||||||
|
|
||||||
|
First one is the current poll-derived queue. Second is older webhook-derived queue but worth keeping for audit/history.
|
||||||
|
|
||||||
|
### Useful but not strictly required
|
||||||
|
|
||||||
|
```text
|
||||||
|
PROJECT-PROGRESS.md
|
||||||
|
docs/
|
||||||
|
logs/
|
||||||
|
.git/
|
||||||
|
```
|
||||||
|
|
||||||
|
I’d migrate these too unless you want a clean production-only deploy. The logs and docs are useful for future archaeology when something inevitably gets weird.
|
||||||
|
|
||||||
|
## Do not blindly migrate
|
||||||
|
|
||||||
|
I would **not** migrate the existing virtualenv folders directly:
|
||||||
|
|
||||||
|
```text
|
||||||
|
bin/
|
||||||
|
lib/
|
||||||
|
lib64/
|
||||||
|
include/
|
||||||
|
pyvenv.cfg
|
||||||
|
__pycache__/
|
||||||
|
```
|
||||||
|
|
||||||
|
They are host/Python-version sensitive. Recreate the venv on the new server.
|
||||||
|
|
||||||
|
## Python / pip packages
|
||||||
|
|
||||||
|
Current venv freeze shows:
|
||||||
|
|
||||||
|
```text
|
||||||
|
fastapi==0.136.0
|
||||||
|
uvicorn==0.45.0
|
||||||
|
requests==2.33.1
|
||||||
|
python-dotenv==1.2.2
|
||||||
|
pydantic==2.13.3
|
||||||
|
starlette==1.0.0
|
||||||
|
anyio==4.13.0
|
||||||
|
click==8.3.3
|
||||||
|
h11==0.16.0
|
||||||
|
httptools==0.7.1
|
||||||
|
uvloop==0.22.1
|
||||||
|
watchfiles==1.1.1
|
||||||
|
websockets==16.0
|
||||||
|
certifi==2026.4.22
|
||||||
|
charset-normalizer==3.4.7
|
||||||
|
idna==3.13
|
||||||
|
urllib3==2.6.3
|
||||||
|
PyYAML==6.0.3
|
||||||
|
typing_extensions==4.15.0
|
||||||
|
typing-inspection==0.4.2
|
||||||
|
annotated-types==0.7.0
|
||||||
|
annotated-doc==0.0.4
|
||||||
|
pydantic_core==2.46.3
|
||||||
|
```
|
||||||
|
|
||||||
|
Practical install line:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3.12 -m venv /opt/webhooks
|
||||||
|
/opt/webhooks/bin/pip install --upgrade pip
|
||||||
|
/opt/webhooks/bin/pip install fastapi 'uvicorn[standard]' requests python-dotenv
|
||||||
|
```
|
||||||
|
|
||||||
|
Or create a `requirements.txt` from the freeze if you want exact locking.
|
||||||
|
|
||||||
|
System packages likely needed:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3.12
|
||||||
|
python3.12-venv
|
||||||
|
sqlite3
|
||||||
|
bash
|
||||||
|
curl
|
||||||
|
rsync
|
||||||
|
systemd
|
||||||
|
```
|
||||||
|
|
||||||
|
Important: the wrapper uses GNU `date -d`, so Linux/GNU coreutils is assumed.
|
||||||
|
|
||||||
|
## Environment variables required
|
||||||
|
|
||||||
|
For polling/applying:
|
||||||
|
|
||||||
|
```text
|
||||||
|
SERVICEM8_ACCESS_TOKEN
|
||||||
|
# or
|
||||||
|
SERVICEM8_API_KEY
|
||||||
|
```
|
||||||
|
|
||||||
|
Optional/current defaults:
|
||||||
|
|
||||||
|
```text
|
||||||
|
SERVICEM8_BASE_URL=https://api.servicem8.com/api_1.0
|
||||||
|
SERVICEM8_TIMEOUT=30
|
||||||
|
SERVICEM8_QUOTE_TEMPLATE_FORM_UUID=3621b6be-1d19-4756-9ab4-9d5e4120f6d9
|
||||||
|
WEBHOOK_POLL_DB_PATH=/opt/webhooks/servicem8_formresponse_poll.db
|
||||||
|
WEBHOOK_RUN_LOG_DIR=/opt/webhooks/logs
|
||||||
|
```
|
||||||
|
|
||||||
|
For webhook receiver:
|
||||||
|
|
||||||
|
```text
|
||||||
|
WEBHOOK_HOST=0.0.0.0
|
||||||
|
WEBHOOK_PORT=18354
|
||||||
|
WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db
|
||||||
|
```
|
||||||
|
|
||||||
|
For inspector:
|
||||||
|
|
||||||
|
```text
|
||||||
|
WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db
|
||||||
|
WEBHOOK_STATE_DB_PATH=/opt/webhooks/servicem8_quote_materials_state.db
|
||||||
|
WEBHOOK_POLL_DB_PATH=/opt/webhooks/servicem8_formresponse_poll.db
|
||||||
|
WEBHOOK_INSPECTOR_HOST=0.0.0.0
|
||||||
|
WEBHOOK_INSPECTOR_PORT=18355
|
||||||
|
```
|
||||||
|
|
||||||
|
## Existing systemd services to recreate
|
||||||
|
|
||||||
|
Current services are:
|
||||||
|
|
||||||
|
```text
|
||||||
|
servicem8-dev-webhook.service
|
||||||
|
servicem8-dev-inspector.service
|
||||||
|
```
|
||||||
|
|
||||||
|
Current paths assume `/opt/webhooks`.
|
||||||
|
|
||||||
|
Webhook:
|
||||||
|
|
||||||
|
```ini
|
||||||
|
WorkingDirectory=/opt/webhooks
|
||||||
|
Environment="WEBHOOK_HOST=0.0.0.0"
|
||||||
|
Environment="WEBHOOK_PORT=18354"
|
||||||
|
Environment="WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db"
|
||||||
|
ExecStart=/opt/webhooks/bin/uvicorn servicem8_webhook_receiver:app --host 0.0.0.0 --port 18354
|
||||||
|
```
|
||||||
|
|
||||||
|
Inspector:
|
||||||
|
|
||||||
|
```ini
|
||||||
|
WorkingDirectory=/opt/webhooks
|
||||||
|
Environment="WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db"
|
||||||
|
Environment="WEBHOOK_INSPECTOR_HOST=0.0.0.0"
|
||||||
|
Environment="WEBHOOK_INSPECTOR_PORT=18355"
|
||||||
|
ExecStart=/opt/webhooks/bin/uvicorn servicem8_inspector:app --host 0.0.0.0 --port 18355
|
||||||
|
```
|
||||||
|
|
||||||
|
On the new host, probably change:
|
||||||
|
|
||||||
|
```ini
|
||||||
|
User=michael
|
||||||
|
Group=michael
|
||||||
|
```
|
||||||
|
|
||||||
|
to whatever the production service user is, e.g.:
|
||||||
|
|
||||||
|
```ini
|
||||||
|
User=openclaw
|
||||||
|
Group=openclaw
|
||||||
|
```
|
||||||
|
|
||||||
|
or create a dedicated `servicem8` user.
|
||||||
|
|
||||||
|
## Path/code changes for seamless migration
|
||||||
|
|
||||||
|
Best option: keep the same production path:
|
||||||
|
|
||||||
|
```text
|
||||||
|
/opt/webhooks
|
||||||
|
```
|
||||||
|
|
||||||
|
If you do that, almost no code changes are needed.
|
||||||
|
|
||||||
|
If you move it elsewhere, update:
|
||||||
|
|
||||||
|
1. systemd `WorkingDirectory`
|
||||||
|
2. systemd `ExecStart`
|
||||||
|
3. env vars:
|
||||||
|
- `WEBHOOK_DB_PATH`
|
||||||
|
- `WEBHOOK_POLL_DB_PATH`
|
||||||
|
- `WEBHOOK_STATE_DB_PATH`
|
||||||
|
- `WEBHOOK_RUN_LOG_DIR`
|
||||||
|
4. any OpenClaw cron or host cron entry that calls:
|
||||||
|
- `/opt/webhooks/poll_and_apply_quote_templates.sh`
|
||||||
|
|
||||||
|
One gotcha: `servicem8_quote_template_parser.py` currently hardcodes the state DB beside the script:
|
||||||
|
|
||||||
|
```python
|
||||||
|
STATE_DB_PATH = Path(__file__).with_name("servicem8_quote_materials_state.db")
|
||||||
|
```
|
||||||
|
|
||||||
|
So keeping DBs beside the scripts is the safest path. If you want DBs somewhere else, I’d adjust that code to respect `WEBHOOK_STATE_DB_PATH`.
|
||||||
|
|
||||||
|
## External callback / production URL
|
||||||
|
|
||||||
|
Current webhook helper scripts still reference dev callback URLs like:
|
||||||
|
|
||||||
|
```text
|
||||||
|
https://nps-dev.coast2cloud.net/...
|
||||||
|
```
|
||||||
|
|
||||||
|
For the new production host, ServiceM8 webhook subscriptions need to point at the new public URL if you want inbound webhooks working.
|
||||||
|
|
||||||
|
Polling does **not** depend on inbound webhooks, so the wrapper can operate without this, but the receiver/inspector diagnostics won’t capture new webhook events unless ServiceM8 is updated.
|
||||||
|
|
||||||
|
## Security note
|
||||||
|
|
||||||
|
There are old helper scripts with hardcoded ServiceM8 token fallbacks:
|
||||||
|
|
||||||
|
```text
|
||||||
|
servicem8-create-webhook-form-response.py
|
||||||
|
servicem8-create-webhook-job-object.py
|
||||||
|
servicem8-list-webhook-subscriptions-table.py
|
||||||
|
```
|
||||||
|
|
||||||
|
I would not treat those as production-safe as-is. Before migration to prod, remove hardcoded fallback tokens and require env vars only.
|
||||||
|
|
||||||
|
## Recommended cutover shape
|
||||||
|
|
||||||
|
1. Stop current webhook/inspector services.
|
||||||
|
2. Copy `/opt/webhooks` excluding venv/cache, including DBs/jsonl/logs.
|
||||||
|
3. Recreate venv on new host.
|
||||||
|
4. Install pip packages.
|
||||||
|
5. Add env/secrets via systemd `EnvironmentFile`, not hardcoded scripts.
|
||||||
|
6. Recreate systemd services.
|
||||||
|
7. Run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
/opt/webhooks/poll_and_apply_quote_templates.sh --dry-run --hours 48
|
||||||
|
```
|
||||||
|
|
||||||
|
8. If clean, run controlled live apply.
|
||||||
|
9. Only then schedule the wrapper.
|
||||||
|
|
||||||
|
At present I found no OpenClaw cron job already running this plumbing wrapper, so scheduling still appears to be a pending production decision rather than something to migrate from OpenClaw cron.
|
||||||
@@ -1,5 +0,0 @@
|
|||||||
home = /usr/bin
|
|
||||||
include-system-site-packages = false
|
|
||||||
version = 3.12.3
|
|
||||||
executable = /usr/bin/python3.12
|
|
||||||
command = /usr/bin/python3 -m venv /opt/webhooks
|
|
||||||
Reference in New Issue
Block a user