Added migration notes
This commit is contained in:
@@ -0,0 +1,295 @@
|
||||
Yep — I inspected the current setup. The “current bash wrapper” is the ServiceM8 plumbing quote-template pipeline under:
|
||||
|
||||
`/opt/webhooks`
|
||||
|
||||
In the workspace it appears as a symlink:
|
||||
|
||||
`/home/openclaw/.openclaw/workspace/projects/plumbing -> /opt/webhooks`
|
||||
|
||||
## Definitive migration set
|
||||
|
||||
### Required operational scripts
|
||||
|
||||
Copy these from `/opt/webhooks/`:
|
||||
|
||||
```text
|
||||
poll_and_apply_quote_templates.sh
|
||||
poll_form_responses_since.py
|
||||
apply_polled_quote_template_jobmaterials.py
|
||||
servicem8_quote_template_parser.py
|
||||
servicem8_webhook_receiver.py
|
||||
servicem8_inspector.py
|
||||
```
|
||||
|
||||
These are the active pieces:
|
||||
|
||||
- `poll_and_apply_quote_templates.sh` — main wrapper / scheduled entry point
|
||||
- `poll_form_responses_since.py` — polls ServiceM8 form responses
|
||||
- `apply_polled_quote_template_jobmaterials.py` — applies parsed quote rows to ServiceM8 jobMaterials
|
||||
- `servicem8_quote_template_parser.py` — parsing/state logic
|
||||
- `servicem8_webhook_receiver.py` — webhook receiver/diagnostics
|
||||
- `servicem8_inspector.py` — web UI inspector
|
||||
|
||||
### Required DB/state files
|
||||
|
||||
Copy these, ideally while services are stopped:
|
||||
|
||||
```text
|
||||
servicem8_formresponse_poll.db
|
||||
servicem8_quote_materials_state.db
|
||||
servicem8_webhooks.db
|
||||
```
|
||||
|
||||
Also copy any SQLite sidecar files if present at cutover:
|
||||
|
||||
```text
|
||||
*.db-wal
|
||||
*.db-shm
|
||||
```
|
||||
|
||||
### Required JSONL queue/history files
|
||||
|
||||
```text
|
||||
quote-template-jobmaterials-poll-queue.jsonl
|
||||
quote-template-jobmaterials-queue.jsonl
|
||||
```
|
||||
|
||||
First one is the current poll-derived queue. Second is older webhook-derived queue but worth keeping for audit/history.
|
||||
|
||||
### Useful but not strictly required
|
||||
|
||||
```text
|
||||
PROJECT-PROGRESS.md
|
||||
docs/
|
||||
logs/
|
||||
.git/
|
||||
```
|
||||
|
||||
I’d migrate these too unless you want a clean production-only deploy. The logs and docs are useful for future archaeology when something inevitably gets weird.
|
||||
|
||||
## Do not blindly migrate
|
||||
|
||||
I would **not** migrate the existing virtualenv folders directly:
|
||||
|
||||
```text
|
||||
bin/
|
||||
lib/
|
||||
lib64/
|
||||
include/
|
||||
pyvenv.cfg
|
||||
__pycache__/
|
||||
```
|
||||
|
||||
They are host/Python-version sensitive. Recreate the venv on the new server.
|
||||
|
||||
## Python / pip packages
|
||||
|
||||
Current venv freeze shows:
|
||||
|
||||
```text
|
||||
fastapi==0.136.0
|
||||
uvicorn==0.45.0
|
||||
requests==2.33.1
|
||||
python-dotenv==1.2.2
|
||||
pydantic==2.13.3
|
||||
starlette==1.0.0
|
||||
anyio==4.13.0
|
||||
click==8.3.3
|
||||
h11==0.16.0
|
||||
httptools==0.7.1
|
||||
uvloop==0.22.1
|
||||
watchfiles==1.1.1
|
||||
websockets==16.0
|
||||
certifi==2026.4.22
|
||||
charset-normalizer==3.4.7
|
||||
idna==3.13
|
||||
urllib3==2.6.3
|
||||
PyYAML==6.0.3
|
||||
typing_extensions==4.15.0
|
||||
typing-inspection==0.4.2
|
||||
annotated-types==0.7.0
|
||||
annotated-doc==0.0.4
|
||||
pydantic_core==2.46.3
|
||||
```
|
||||
|
||||
Practical install line:
|
||||
|
||||
```bash
|
||||
python3.12 -m venv /opt/webhooks
|
||||
/opt/webhooks/bin/pip install --upgrade pip
|
||||
/opt/webhooks/bin/pip install fastapi 'uvicorn[standard]' requests python-dotenv
|
||||
```
|
||||
|
||||
Or create a `requirements.txt` from the freeze if you want exact locking.
|
||||
|
||||
System packages likely needed:
|
||||
|
||||
```bash
|
||||
python3.12
|
||||
python3.12-venv
|
||||
sqlite3
|
||||
bash
|
||||
curl
|
||||
rsync
|
||||
systemd
|
||||
```
|
||||
|
||||
Important: the wrapper uses GNU `date -d`, so Linux/GNU coreutils is assumed.
|
||||
|
||||
## Environment variables required
|
||||
|
||||
For polling/applying:
|
||||
|
||||
```text
|
||||
SERVICEM8_ACCESS_TOKEN
|
||||
# or
|
||||
SERVICEM8_API_KEY
|
||||
```
|
||||
|
||||
Optional/current defaults:
|
||||
|
||||
```text
|
||||
SERVICEM8_BASE_URL=https://api.servicem8.com/api_1.0
|
||||
SERVICEM8_TIMEOUT=30
|
||||
SERVICEM8_QUOTE_TEMPLATE_FORM_UUID=3621b6be-1d19-4756-9ab4-9d5e4120f6d9
|
||||
WEBHOOK_POLL_DB_PATH=/opt/webhooks/servicem8_formresponse_poll.db
|
||||
WEBHOOK_RUN_LOG_DIR=/opt/webhooks/logs
|
||||
```
|
||||
|
||||
For webhook receiver:
|
||||
|
||||
```text
|
||||
WEBHOOK_HOST=0.0.0.0
|
||||
WEBHOOK_PORT=18354
|
||||
WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db
|
||||
```
|
||||
|
||||
For inspector:
|
||||
|
||||
```text
|
||||
WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db
|
||||
WEBHOOK_STATE_DB_PATH=/opt/webhooks/servicem8_quote_materials_state.db
|
||||
WEBHOOK_POLL_DB_PATH=/opt/webhooks/servicem8_formresponse_poll.db
|
||||
WEBHOOK_INSPECTOR_HOST=0.0.0.0
|
||||
WEBHOOK_INSPECTOR_PORT=18355
|
||||
```
|
||||
|
||||
## Existing systemd services to recreate
|
||||
|
||||
Current services are:
|
||||
|
||||
```text
|
||||
servicem8-dev-webhook.service
|
||||
servicem8-dev-inspector.service
|
||||
```
|
||||
|
||||
Current paths assume `/opt/webhooks`.
|
||||
|
||||
Webhook:
|
||||
|
||||
```ini
|
||||
WorkingDirectory=/opt/webhooks
|
||||
Environment="WEBHOOK_HOST=0.0.0.0"
|
||||
Environment="WEBHOOK_PORT=18354"
|
||||
Environment="WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db"
|
||||
ExecStart=/opt/webhooks/bin/uvicorn servicem8_webhook_receiver:app --host 0.0.0.0 --port 18354
|
||||
```
|
||||
|
||||
Inspector:
|
||||
|
||||
```ini
|
||||
WorkingDirectory=/opt/webhooks
|
||||
Environment="WEBHOOK_DB_PATH=/opt/webhooks/servicem8_webhooks.db"
|
||||
Environment="WEBHOOK_INSPECTOR_HOST=0.0.0.0"
|
||||
Environment="WEBHOOK_INSPECTOR_PORT=18355"
|
||||
ExecStart=/opt/webhooks/bin/uvicorn servicem8_inspector:app --host 0.0.0.0 --port 18355
|
||||
```
|
||||
|
||||
On the new host, probably change:
|
||||
|
||||
```ini
|
||||
User=michael
|
||||
Group=michael
|
||||
```
|
||||
|
||||
to whatever the production service user is, e.g.:
|
||||
|
||||
```ini
|
||||
User=openclaw
|
||||
Group=openclaw
|
||||
```
|
||||
|
||||
or create a dedicated `servicem8` user.
|
||||
|
||||
## Path/code changes for seamless migration
|
||||
|
||||
Best option: keep the same production path:
|
||||
|
||||
```text
|
||||
/opt/webhooks
|
||||
```
|
||||
|
||||
If you do that, almost no code changes are needed.
|
||||
|
||||
If you move it elsewhere, update:
|
||||
|
||||
1. systemd `WorkingDirectory`
|
||||
2. systemd `ExecStart`
|
||||
3. env vars:
|
||||
- `WEBHOOK_DB_PATH`
|
||||
- `WEBHOOK_POLL_DB_PATH`
|
||||
- `WEBHOOK_STATE_DB_PATH`
|
||||
- `WEBHOOK_RUN_LOG_DIR`
|
||||
4. any OpenClaw cron or host cron entry that calls:
|
||||
- `/opt/webhooks/poll_and_apply_quote_templates.sh`
|
||||
|
||||
One gotcha: `servicem8_quote_template_parser.py` currently hardcodes the state DB beside the script:
|
||||
|
||||
```python
|
||||
STATE_DB_PATH = Path(__file__).with_name("servicem8_quote_materials_state.db")
|
||||
```
|
||||
|
||||
So keeping DBs beside the scripts is the safest path. If you want DBs somewhere else, I’d adjust that code to respect `WEBHOOK_STATE_DB_PATH`.
|
||||
|
||||
## External callback / production URL
|
||||
|
||||
Current webhook helper scripts still reference dev callback URLs like:
|
||||
|
||||
```text
|
||||
https://nps-dev.coast2cloud.net/...
|
||||
```
|
||||
|
||||
For the new production host, ServiceM8 webhook subscriptions need to point at the new public URL if you want inbound webhooks working.
|
||||
|
||||
Polling does **not** depend on inbound webhooks, so the wrapper can operate without this, but the receiver/inspector diagnostics won’t capture new webhook events unless ServiceM8 is updated.
|
||||
|
||||
## Security note
|
||||
|
||||
There are old helper scripts with hardcoded ServiceM8 token fallbacks:
|
||||
|
||||
```text
|
||||
servicem8-create-webhook-form-response.py
|
||||
servicem8-create-webhook-job-object.py
|
||||
servicem8-list-webhook-subscriptions-table.py
|
||||
```
|
||||
|
||||
I would not treat those as production-safe as-is. Before migration to prod, remove hardcoded fallback tokens and require env vars only.
|
||||
|
||||
## Recommended cutover shape
|
||||
|
||||
1. Stop current webhook/inspector services.
|
||||
2. Copy `/opt/webhooks` excluding venv/cache, including DBs/jsonl/logs.
|
||||
3. Recreate venv on new host.
|
||||
4. Install pip packages.
|
||||
5. Add env/secrets via systemd `EnvironmentFile`, not hardcoded scripts.
|
||||
6. Recreate systemd services.
|
||||
7. Run:
|
||||
|
||||
```bash
|
||||
/opt/webhooks/poll_and_apply_quote_templates.sh --dry-run --hours 48
|
||||
```
|
||||
|
||||
8. If clean, run controlled live apply.
|
||||
9. Only then schedule the wrapper.
|
||||
|
||||
At present I found no OpenClaw cron job already running this plumbing wrapper, so scheduling still appears to be a pending production decision rather than something to migrate from OpenClaw cron.
|
||||
Reference in New Issue
Block a user