Added all of chat-topics, persona;-history, projects, receipts and vehicle. This is the knowledge base section after all....

This commit is contained in:
2026-05-26 16:53:50 +10:00
parent 85f78332d1
commit 6ada86fe73
91 changed files with 3983 additions and 26 deletions
+55
View File
@@ -0,0 +1,55 @@
# Receipts Archive
Local receipt/warranty archive for Michael.
## Structure
- `receipts.sqlite` — searchable SQLite database.
- `images/YY-MM-images/` — original receipt/warranty images, grouped by month.
## Current database tables
- `receipts` — one row per receipt/document image, including receipt-level `notes`.
- `receipt_items` — line items linked to `receipts.id`.
## Suggested conventions
- `category` is broad and filterable, e.g. `Home purchases`, `Business expenses`, `Vehicle`, `Tools`, `Medical`, `Travel`.
- `doc_type` distinguishes `receipt`, `invoice`, `warranty_certificate`, `quote`, etc.
- Keep images immutable; fix OCR/database fields rather than replacing the source image.
- Use `image_sha256` to avoid duplicate imports.
## Local import/OCR wrapper
```bash
# Dry-run OCR/parse without touching the DB
knowledge/receipts/scripts/receipt_import.py /path/to/receipt.jpg --dry-run --json
# Import a new receipt image/PDF into the archive and DB
knowledge/receipts/scripts/receipt_import.py /path/to/receipt.jpg --category "Home purchases" --notes "Warranty for office"
# Re-OCR/update an existing DB row
knowledge/receipts/scripts/receipt_import.py knowledge/receipts/images/26-05-images/example.jpg --receipt-id 2 --category "Personal care"
```
The wrapper keeps originals immutable, writes OCR derivatives under `knowledge/receipts/ocr/<sha-prefix>/`, uses local Tesseract/OCRmyPDF only, and stores parsed fields plus raw OCR text in `receipts.sqlite`.
## Useful queries
```sql
-- Recent records
SELECT id, category, doc_type, merchant_name, receipt_date, total, image_path
FROM receipts
ORDER BY COALESCE(receipt_date, captured_at) DESC;
-- Filter by category
SELECT id, merchant_name, receipt_date, total, image_path
FROM receipts
WHERE category = 'Home purchases';
-- Search items/text
SELECT r.id, r.category, r.merchant_name, i.description, i.sku, r.image_path
FROM receipts r
JOIN receipt_items i ON i.receipt_id = r.id
WHERE i.description LIKE '%powerbank%' OR r.ocr_raw_text LIKE '%powerbank%';
```
Binary file not shown.

After

Width:  |  Height:  |  Size: 140 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 87 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 78 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 80 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 86 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 165 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 100 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 86 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 85 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 79 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 342 KiB

+123
View File
@@ -0,0 +1,123 @@
--- OCR original.png psm=6 ---
é AliExpress Qgare#
Orders ve
Viewall Topay(0) Toship(2) Shipped(0) Process
To ship May 23, 2026 >
vChoice| SPEEDRACING Official Store >
Ag PC USB Sequential Shifter Gear Shifter For Logitech ...
Clamp
| Bingen
AUS1 coupon if delayed
citty
Free returns
Total:AU$ 25.80
PC USB Sequential Shifter Gear Shifter For Logitech ...
black
Hit AUS 83.69 xi
+2 AUS1 coupon if delayed
“ Free returns
Total:AU$ 92.06
--- OCR original.png psm=11 ---
AliExpress
Orders
Qa
View all
To pay (0)
To ship (2)
Shipped (0)
Process
—=
May 23, 2026 >
To ship
e
AUS 23.32 x1
aaae
Total:AUS$ 25.80
AUS 83.69 x1
<
Total:AUS 92.06
--- OCR gray_contrast_2x.png psm=6 ---
< AliExpress Q Q oy)
Orders oe
Viewall Topay(0) Toship(2) Shipped(0O) Process
V¥Choice| SPEEDRACING Official Store >
oss,
sore AUS 83.69 x1
--- OCR gray_contrast_2x.png psm=11 ---
Q 8
°°
=
Total:AUS 25.80
——~
boee
--- OCR threshold_2x.png psm=6 ---
¢ AliExpress agar
Orders oo
VChoice | SPEEDRACING Official Store >
8068,
ors AUS 83.69 x1
Total:AUS$ 92.06
--- OCR threshold_2x.png psm=11 ---
Q &
R
qn»
vVChoice| SPEEDRACING Official Store >
6000
ttt
--- OCR crop_1_2x.png psm=6 ---
< Orders Q ran R
--- OCR crop_1_2x.png psm=11 ---
RF
[ss
--- OCR crop_2_2x.png psm=6 ---
————- ll i
--- OCR crop_2_2x.png psm=11 ---
--- OCR crop_3_2x.png psm=6 ---
| PC USB Sequential Shifter Gear Shifter For Logitech ...
--- OCR crop_3_2x.png psm=11 ---
|
--- OCR crop_4_2x.png psm=6 ---
erat AUS 83.69 x1
Edit address
--- OCR crop_4_2x.png psm=11 ---
seoe
@@ -0,0 +1,86 @@
[
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/4eba4bf8f3eaf083/original.png",
"psm": 6,
"score": 715,
"chars": 465
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/4eba4bf8f3eaf083/original.png",
"psm": 11,
"score": 694,
"chars": 494
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/4eba4bf8f3eaf083/gray_contrast_2x.png",
"psm": 6,
"score": 665,
"chars": 465
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/4eba4bf8f3eaf083/gray_contrast_2x.png",
"psm": 11,
"score": 649,
"chars": 499
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/4eba4bf8f3eaf083/threshold_2x.png",
"psm": 6,
"score": 712,
"chars": 462
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/4eba4bf8f3eaf083/threshold_2x.png",
"psm": 11,
"score": 703,
"chars": 503
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/4eba4bf8f3eaf083/crop_1_2x.png",
"psm": 6,
"score": 184,
"chars": 134
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/4eba4bf8f3eaf083/crop_1_2x.png",
"psm": 11,
"score": 203,
"chars": 153
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/4eba4bf8f3eaf083/crop_2_2x.png",
"psm": 6,
"score": 242,
"chars": 142
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/4eba4bf8f3eaf083/crop_2_2x.png",
"psm": 11,
"score": 187,
"chars": 137
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/4eba4bf8f3eaf083/crop_3_2x.png",
"psm": 6,
"score": 124,
"chars": 124
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/4eba4bf8f3eaf083/crop_3_2x.png",
"psm": 11,
"score": 129,
"chars": 129
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/4eba4bf8f3eaf083/crop_4_2x.png",
"psm": 6,
"score": 132,
"chars": 82
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/4eba4bf8f3eaf083/crop_4_2x.png",
"psm": 11,
"score": 126,
"chars": 76
}
]
Binary file not shown.

After

Width:  |  Height:  |  Size: 355 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 48 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 88 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 39 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 65 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 89 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 262 KiB

+96
View File
@@ -0,0 +1,96 @@
--- OCR original.png psm=6 ---
Merchant AIAU MARKETS PTY LTD
Transaction date 23 May 2026
Order ID 202605238503130000436825
8230069
Deliver to Michael Manning
Forsters Bay Road
7
Narooma NSW 2546
Australia
Track Package
PC USB Sequential Sh... $25.80
Qty: 1
Product ID: 8211773665403987
PC USB Sequential Sh... $92.06
Product ID: 8211773665383987
Subtotal $117.86
Total $117.86 AUD
Paid AIAU MARKETS PTY LTD with
fase svIsA $117.86 AUD
Ss Credit --4942
This charge will appear on your credit card
statement as "PAYPAL *AIAUMARKETS".
Transaction ID: 4EA50033E33881234
--- OCR original.png psm=11 ---
Merchant
AIAU MARKETS PTY LTD
Transaction date
23 May 2026
Order ID
202605238503130000436825
Deliver to
Michael Manning
$25.80
PC USB Sequential Sh...
$92.06
Subtotal
$117.86
Total
$117.86 AUD
VISA
Credit +-4942
--- OCR gray_contrast_2x.png psm=6 ---
VISA $117.86 AUD
--- OCR gray_contrast_2x.png psm=11 ---
=
Credit +4942
--- OCR threshold_2x.png psm=6 ---
4 VISA $117.86 AUD
--- OCR threshold_2x.png psm=11 ---
Credit --4942
--- OCR crop_1_2x.png psm=6 ---
--- OCR crop_1_2x.png psm=11 ---
--- OCR crop_2_2x.png psm=6 ---
Dradiict IN- 2911772RASANZOR7
--- OCR crop_2_2x.png psm=11 ---
Drodiict IN: 28911772ARA5SANZORT
--- OCR crop_3_2x.png psm=6 ---
--- OCR crop_3_2x.png psm=11 ---
ee er el
eet ete ee Se ee es Oa
--- OCR crop_4_2x.png psm=6 ---
--- OCR crop_4_2x.png psm=11 ---
Credit +*-4942
@@ -0,0 +1,86 @@
[
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/92677db8f57247de/original.png",
"psm": 6,
"score": 992,
"chars": 542
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/92677db8f57247de/original.png",
"psm": 11,
"score": 1015,
"chars": 565
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/92677db8f57247de/gray_contrast_2x.png",
"psm": 6,
"score": 990,
"chars": 540
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/92677db8f57247de/gray_contrast_2x.png",
"psm": 11,
"score": 1016,
"chars": 566
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/92677db8f57247de/threshold_2x.png",
"psm": 6,
"score": 984,
"chars": 534
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/92677db8f57247de/threshold_2x.png",
"psm": 11,
"score": 1014,
"chars": 564
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/92677db8f57247de/crop_1_2x.png",
"psm": 6,
"score": 214,
"chars": 164
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/92677db8f57247de/crop_1_2x.png",
"psm": 11,
"score": 224,
"chars": 174
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/92677db8f57247de/crop_2_2x.png",
"psm": 6,
"score": 141,
"chars": 91
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/92677db8f57247de/crop_2_2x.png",
"psm": 11,
"score": 149,
"chars": 99
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/92677db8f57247de/crop_3_2x.png",
"psm": 6,
"score": 403,
"chars": 103
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/92677db8f57247de/crop_3_2x.png",
"psm": 11,
"score": 445,
"chars": 145
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/92677db8f57247de/crop_4_2x.png",
"psm": 6,
"score": 225,
"chars": 175
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/92677db8f57247de/crop_4_2x.png",
"psm": 11,
"score": 235,
"chars": 185
}
]
Binary file not shown.

After

Width:  |  Height:  |  Size: 251 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 76 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 45 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 70 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 219 KiB

+194
View File
@@ -0,0 +1,194 @@
--- OCR original.png psm=6 ---
1:00 O©4ZQ4H° WY 8: 529 He at 66%
3 CE |
i vy Receipt for Your Payment to GoDaddy 0...
"PayPal" <service@paypal.com.au> 7 - =
GoDaddy Online
°
Servi...
Merchant GoDaddy Online Servi...
+61 0280428920
Transaction date 14 May 2026
Order ID 4071702979_AUD_2415
View Payment Details
{ts ]
-AU (.NET.AU) Domain... $21.95
Qty: 1
Subtotal $21.95
Tax $2.20
Total $24.15 AUD
See
~ 00OOGOE®
© https://sm.mmanningau.net/# g A (&) :
--- OCR original.png psm=11 ---
5.00
1:00 ©FQASK
B-
8
vas
66%
1 + Receipt for Your Payment to ame fo)
"PayPal" <service@paypal.com.au> 7
Merchant
GoDaddy Online Servi
Transaction date
14 May 2026
Order ID
4071702979_AUD_2415
lets
$21.95
AU (.NET.AU) Domain.
Subtotal
Tax
$2.20
Total
$24.15 AUD
26
© https://sm.mmanningau.net/# g A (&)
--- OCR gray_contrast_2x.png psm=6 ---
1:00 O4ZQ4BH * AY B= 3:99 SZ te at! 66%
3 CE [a+ |
1 YY Receipt for Your Payment to GoDaddy 0... x
"PayPal" < service@paypal.com.au> # -« =
e
| [ents |
aA se oe Ge -©
@ hitps://sm.mmanningau.net/# 7 A (23) :
--- OCR gray_contrast_2x.png psm=11 ---
1:00 © 4Q4B °
WY B= $99 ZS te atl! 66%
1 Yy Receipt for Your Payment to soe 0
x
"PayPal" < service@paypal.com.au> #7
<5 A
(tests
AU (.NET.AU) Domain
aN
“@
@ https://sm.mmanningau.net/# 7 A (23)
--- OCR threshold_2x.png psm=6 ---
1:00 ©4Q4B « A B= 5:29 S: te al 66%
3 CSE Ex
1 v Receipt for Your Payment to GoDaddy 0... x
“PayPal” < service@paypal.com.au> 7 eo =
Llets
~ se oe Ye -&
@ fites://sm.mmanningau.net/# g A ®& :
--- OCR threshold_2x.png psm=11 ---
1:00 ©4@Q4BH °
& %:
5.00 =
tt ail 66%
Kas oe
1 Yy Receipt for Your Payment to Goaddy 0 Oo
“PayPal” < service@paypal.com.au> 7
“py.
Lite
a
¢
© bttps://sm.mmanningau.net/# g A &
--- OCR crop_1_2x.png psm=6 ---
1:00 O©4Q4B » RY B= $99 ZR Me at 66%
3 CE
i YY Receipt for Your Payment to GoDaddy 0... x
"PayPal" < service@paypal.com.au> ¥ * =
--- OCR crop_1_2x.png psm=11 ---
1:00 © 4Q4BH
WY 8: 5:99 ZS te at 66%
xo} otat<|>
1 YY Receipt for Your Payment to omees* O...
"PayPal" < service@paypal.com.au> %
--- OCR crop_2_2x.png psm=6 ---
--- OCR crop_2_2x.png psm=11 ---
GoDaddy Online Servi...
--- OCR crop_3_2x.png psm=6 ---
--- OCR crop_3_2x.png psm=11 ---
pis
-AU (.NET.AU) Domain...
§21.95
=
es
--- OCR crop_4_2x.png psm=6 ---
1ax 92.20
~ 0 oe Ge -@
@ hitps://sm.mmanningau.net/# g A (23) :
--- OCR crop_4_2x.png psm=11 ---
1ax
92.20
NN
- «© 0 » Ge -@
@ https://sm.mmanningau.net/# g A &)
@@ -0,0 +1,86 @@
[
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/ae6c6d9927838a7c/original.png",
"psm": 6,
"score": 978,
"chars": 428
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/ae6c6d9927838a7c/original.png",
"psm": 11,
"score": 886,
"chars": 386
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/ae6c6d9927838a7c/gray_contrast_2x.png",
"psm": 6,
"score": 995,
"chars": 445
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/ae6c6d9927838a7c/gray_contrast_2x.png",
"psm": 11,
"score": 1037,
"chars": 437
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/ae6c6d9927838a7c/threshold_2x.png",
"psm": 6,
"score": 970,
"chars": 420
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/ae6c6d9927838a7c/threshold_2x.png",
"psm": 11,
"score": 927,
"chars": 427
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/ae6c6d9927838a7c/crop_1_2x.png",
"psm": 6,
"score": 303,
"chars": 153
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/ae6c6d9927838a7c/crop_1_2x.png",
"psm": 11,
"score": 259,
"chars": 159
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/ae6c6d9927838a7c/crop_2_2x.png",
"psm": 6,
"score": 226,
"chars": 126
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/ae6c6d9927838a7c/crop_2_2x.png",
"psm": 11,
"score": 160,
"chars": 110
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/ae6c6d9927838a7c/crop_3_2x.png",
"psm": 6,
"score": 253,
"chars": 53
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/ae6c6d9927838a7c/crop_3_2x.png",
"psm": 11,
"score": 222,
"chars": 72
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/ae6c6d9927838a7c/crop_4_2x.png",
"psm": 6,
"score": 180,
"chars": 80
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/ae6c6d9927838a7c/crop_4_2x.png",
"psm": 11,
"score": 187,
"chars": 87
}
]
Binary file not shown.

After

Width:  |  Height:  |  Size: 251 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 122 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 64 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 100 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 108 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 389 KiB

+270
View File
@@ -0,0 +1,270 @@
--- OCR original.png psm=6 ---
ai7am 39 9s ee @
thes | fmf trist: | @ Aust | © Code | MWe | Ri tran: | GD (ec x) @NoP | @nop | +
@ < > GB nopong.comau/check-out/orderreceivea/siesize%7 QO A WH }
= NO PONG oe
CHECKOUT
© THANK YOU! YOUR ORDER HAS BEEN RECEIVED.
ORDER NUMBER: DATE:
3465186 May 14, 2026
EMAIL: TOTAL:
michael@mmanningau.net $99.98
PAYMENT METHOD:
PayPal
ORDER DETAILS
PRODUCT TOTAL
LUSCIOUS LIME BICARB FREE 85G x 2 $29.90
BICARB FREE, LOW FRAGRANCE 85 x 4 $59.80
Subtotal: $89.70
Points Redeemed: ~$3.14. 314 points
Shipping: $13.42 via Parcel Post
Payment method: PayPal
Total: $99.98 (includes $9.09 GST)
Points: You've earned 433 points
a) ka)
BILLING ADDRESS SHIPPING ADDRESS
Michael Manning Michael Manning
7 Forsters Bay Road 7 Forsters Bay Rood
Narooma New South Wales 2546 Narooma New South Wales 2546
& +erareat2207
BH weno @+1iry @
--- OCR original.png psm=11 ---
si7am 39 9 a:
ee @ @
= the
fi Tris:
@ Aus:
| © coos
| wet
| Biter
@NoF
op |
co
@
<
>
e@
oS
nopong.com.au/check-out/orderreceived/saesie6/7 1] Q@ A &
cd
NO PONG
@ THANK YOU! YOUR ORDER HAS BEEN RECEIVED.
ORDER NUMBER:
DATE:
3465186
May 14, 2026
EMAIL:
TOTAL:
michael(@mmanningau.net
$99.98
PRODUCT
TOTAL
LUSCIOUS LIME BICARB FREE 85G x 2
$29.90
BICARB FREE, LOW FRAGRANCE 856 x 4
$59.80
Subtotal:
$89.70
Points Redeemed:
~$3.4 314 points
Shipping:
$13.42 via Parcel Post
$99.98 (includes $9.09 GsT)
Points:
You've earned 433 points
i)
BILLING ADDRESS
SHIPPING ADDRESS
Michael Manning
7 Forsters Bay Road
Narooma New South Wales 2546
& +61418412227
anor
® +
--- OCR gray_contrast_2x.png psm=6 ---
817am 3B GF 9 Ge On| = OD Gwe)
Wi thes | fm|trist | @ aust | © Code | Bwelc | KB tran EDlec x| @Nor | Q@Nop | +
@ << > © _ & nopong.com.au/check-out/order-received/3465186/% [] g A ® :
iv} THANK YOU! YOUR ORDER HAS BEEN RECEIVED.
BICARB FREE, LOW FRAGRANCE 85G x 4 $59.80
Points Redeemed: -$3.14 314 points
ey w)
7 Forsters Bay Road 7 Forsters Bay Road
Q, +67418412227
i Wwnes @+)y @
--- OCR gray_contrast_2x.png psm=11 ---
8:17am 3B GO 9 Gs
{= © te
®ci x
@NoP | QNoP |
+
Mi The | fl tis | @ Aust | © Code | MWele | Ki tren: | QD
@ <
©
o
=o
nopong.com.au/check-out/order-received/3465186/% g A &
NO PONG
eo
i ceckour
michael@mmanningau.net
BICARB FREE, LOW FRAGRANCE 85G x 4
-$3.14 314 points
2
4)
ber
wwe @ +) 5
--- OCR threshold_2x.png psm=6 ---
si7am 3B) 9 9 Bs on = @ oD
Mi the: | fr trist: | @ aust | © code | wel | BB tran: | Gp | Qc x| @Nop | QNoP | +
@ < > © & nopong.com.au/check-out/order-received/3465186/% [J g A ® :
= @ NO PONG o
iv) THANK YOU! YOUR ORDER HAS BEEN RECEIVED.
PROOUCT TOTAL
a) ~)
ii wnuwosop® @+iy @
--- OCR threshold_2x.png psm=11 ---
8:17am 3B F 9 Es
ee © aD
Mi The | fmf trist: | @ Aust | © Code | Ml wei | BB Tran: | ap @c x) Gnor | QNor | +
Cc
~o
nopong.com.au/check-out/order-received/3465186/% [J g A &
@® NO PONG
~$3.14 314 points
a
.
weno @+¢! .
]
--- OCR crop_1_2x.png psm=6 ---
817am 3H OF 9 ae on = OO iw)
Wi The* | fj Trist | @Aust) | © Code | MP Wele | FATran: | 3 obs Me x] @wor | @nop | +
@ < > © & nopong.com.au/check-out/order-received/3465186/% [) g A ® :
= NO PONG 9
--- OCR crop_1_2x.png psm=11 ---
8:17am 3B 9 9 Gs
on = OO Ge
Wi Thes | fl tists | @ aus | © Code | HWele | Ki tren | ED
@®ci x
oO
nopong.com.au/check-out/order-received/3465186/% [] @ A
--- OCR crop_2_2x.png psm=6 ---
--- OCR crop_2_2x.png psm=11 ---
--- OCR crop_3_2x.png psm=6 ---
Points Redeemed: -$3.14 374 points
--- OCR crop_3_2x.png psm=11 ---
Points Redeemed
-$3.14 374 points
Payment method
--- OCR crop_4_2x.png psm=6 ---
2) w)
Hi Wwne™ @ +)» g
--- OCR crop_4_2x.png psm=11 ---
Ka)
:
@o2
Ww “M fF
Q® +
@@ -0,0 +1,86 @@
[
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/b0c012b61bd94d08/original.png",
"psm": 6,
"score": 1825,
"chars": 825
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/b0c012b61bd94d08/original.png",
"psm": 11,
"score": 1853,
"chars": 853
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/b0c012b61bd94d08/gray_contrast_2x.png",
"psm": 6,
"score": 1824,
"chars": 824
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/b0c012b61bd94d08/gray_contrast_2x.png",
"psm": 11,
"score": 1897,
"chars": 897
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/b0c012b61bd94d08/threshold_2x.png",
"psm": 6,
"score": 1848,
"chars": 848
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/b0c012b61bd94d08/threshold_2x.png",
"psm": 11,
"score": 1902,
"chars": 902
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/b0c012b61bd94d08/crop_1_2x.png",
"psm": 6,
"score": 343,
"chars": 243
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/b0c012b61bd94d08/crop_1_2x.png",
"psm": 11,
"score": 360,
"chars": 260
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/b0c012b61bd94d08/crop_2_2x.png",
"psm": 6,
"score": 371,
"chars": 121
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/b0c012b61bd94d08/crop_2_2x.png",
"psm": 11,
"score": 381,
"chars": 131
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/b0c012b61bd94d08/crop_3_2x.png",
"psm": 6,
"score": 889,
"chars": 239
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/b0c012b61bd94d08/crop_3_2x.png",
"psm": 11,
"score": 902,
"chars": 252
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/b0c012b61bd94d08/crop_4_2x.png",
"psm": 6,
"score": 236,
"chars": 236
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/b0c012b61bd94d08/crop_4_2x.png",
"psm": 11,
"score": 259,
"chars": 259
}
]
Binary file not shown.

After

Width:  |  Height:  |  Size: 380 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 377 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 416 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 450 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 388 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 MiB

+423
View File
@@ -0,0 +1,423 @@
--- OCR original.png psm=6 ---
| 2UNIN INS ;
a . a
(Beg MANSe myiK
ABN 26 008 672 179 ae
; Ph: (02) 4412-9600 =
. Fri BS 02:33:44 PH ;
| RETURNS/TNFO DESK RO? oe
Sale eS a a
ae TAX INVOTCE 4 5
9311644072980 LIGHT DESK LAMP VERVE we
sree SRO ic == UI
9311644106489 PONE Be ANDAR CLICK mz.00 =
B OUTLET WHT CLRPBD™ ae o
3 @ SubTotal: $88.90 BX
of Fe! IN THE TOTAL $56 a ee ay
cap i 405221-942 ies :
me) cheb
-. | @hanse , sa.og fag
: itr it Sey f
a $7349 ROT P956 Cy 4221 #007-B6289-7 4-2 6-05-22 3
TNT TAT TT
i. Ue aaa ee |
--- OCR original.png psm=11 ---
Kvarchguse
BATEMANSBAY
asi Hage, ar
Ph (ba) Hit 800
7
Fri 2
2/0
6/2026 02:33:44 PA
RETURK
:
/IN DESK i)
eat eee
aR TAX” TNVOICE. RE
9311644072980
4
H
TGH
50
td
ESK LAMP U
LED AREN
RUE
$71.40
9311644139799 4 OUT!
WR BRD AR'
tec
$12.00
9311644106463 POWE
6 OUTLET Wi
i
TC
BOA
KPB
DS
TeAtaHD CLICK
3d Subtotal:
$88.90
CLUDED IN THE TOTAL
GST i
405221 -942
$88 *
00
Hg ea nee
$0.00
*" ig
Flybuy
4a)
956 C|
22
TTT
Hii
"
--- OCR gray_contrast_2x.png psm=6 ---
a: 2 warehouse (a
all aIBATEMANSH AY Wie :
ABN 26 0B 672 173 |
b, Ph: (02) 4412 3500 |
aoe Fri fo 1 ie
“4 ~—RETURNS/INEU DESK RO?
~~ 9311644072980 LIGHT DESK LAMP UERUE =
2 1 a ee nh o
“| —-9311644106463 rE | i ANDARD CLICK a
| B OUTLET GHY CL RPBG |
| 3 SubTotal: $88.90 |
4 7 oY Fea aw rHe room $a “$8.08
% ms 405221-942 is '
--- OCR gray_contrast_2x.png psm=11 ---
(5
warehou € |
ats Lani
Ph! (02) Mas 9400
Fri 22/0
0/2026 02:33:44 Ph
mt M
|
yt Ina iT
=e
EE TAX, "ft NVOICE
931164407
SCH oh B01
ES
L
LA
A
Rev'f
C
RVE
9311644139
99 4 OUT
R BRD AR
9311644106
6 OUTLET
463 POW
ae
oe
pppanoaio CLICK
3 @ Subsotal:
$5
Af i fa, IN THE TOTAL
CRED
<
405221-942
ye
oii
“Ta
--- OCR threshold_2x.png psm=6 ---
is ion os ey
4 Zi warehouse =~"
So TBATEMARS RAY Mi
a7 ABN 26 00B 672 179
a Ph: (02) 4412 3600
See of . '
4 Fri Ce |
et] RETR TIL-OESK ROT
ee ne TAX ENVOT CE}
7 9311644072980 LTOHT DESK LAMP UERUE
PR Me ki tulades
| " 4p 93tteaat des ie i “non CLICK
wae | f .3@ SubTotal: i
é «| “ie. IN THE TOYA sq |
3 KARR £ 405221-942 Save
Bai
AA
--- OCR threshold_2x.png psm=11 ---
%
ze
yt Fi
L warehouse
saat wane Aicie
2 179
Say
Pr t2) Mae 3500
be
Sos at
a?
Fri 72/05/2006 02:33:44
AS? 7
FET al DEK WO
><
_ EE Tax” ; NGOTCE
a reef em
fin ob Bop
ESK LA
Rev fe
°
93116441
9799 4 OUT
!
ALEC
6 0
LE
vi
cw
a CLICK
3 @ SubTotal:
;
SE
UDED IN THE rot
cH
405221-942_
eyany
vy
Change SS
Fini
"Til iin
--- OCR crop_1_2x.png psm=6 ---
oat.
Bs y
os i ~~
BURNING
- BATEMANSBAYWH
--- OCR crop_1_2x.png psm=11 ---
ee
2Un NINGS
/BATEMANSBAYWH
BUNNINGS GROUP LTMITED
--- OCR crop_2_2x.png psm=6 ---
=) rr NBG O84 POSCS:«*d
d HANES a
we TaOPNVoTCE see
--- OCR crop_2_2x.png psm=11 ---
Ph: (02) 4412 3500
ABN 26 U0G 672 179
Fri 22/04
0/2026 02:33:44.
RET if Me UE A
Ke TAX ? NVOTCE
--- OCR crop_3_2x.png psm=6 ---
~§ JITTO9HS0 72300 LIGHT UEOR LAMP VERVE in
>. an RL ata Zl
at | 931164410 is Pui i a a
98 Subota: ; se
r 7 od IN THE TOTAL $& ,
| canna: 405221-942 $08.90
--- OCR crop_3_2x.png psm=11 ---
JTTO44U7
Ah 9 5001
AREN TC
AVE
931164413
REC
9311644106463 POW
'
STaNOARD CLICK
3 @ SubTotal
@ }
GST
LUDED IN THE TOTAL
CARD
--- OCR crop_4_2x.png psm=6 ---
Be PN ty,
-. | Change $0.00" |
ME, A wou +.” il my Ste . le) - . A i
= . FA yous s card: oH Set she
57348 07555 oat anions 1549-26-05 -22 oe
TIE TMT
--- OCR crop_4_2x.png psm=11 ---
Ro
1ng
ange
$0.
ec,
Bra
*" Tndicé
bh
Flybuys
an
A@)
» Tiiii
@@ -0,0 +1,86 @@
[
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/f1ef054a94e5632c/original.png",
"psm": 6,
"score": 793,
"chars": 493
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/f1ef054a94e5632c/original.png",
"psm": 11,
"score": 928,
"chars": 478
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/f1ef054a94e5632c/gray_contrast_2x.png",
"psm": 6,
"score": 584,
"chars": 334
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/f1ef054a94e5632c/gray_contrast_2x.png",
"psm": 11,
"score": 453,
"chars": 353
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/f1ef054a94e5632c/threshold_2x.png",
"psm": 6,
"score": 438,
"chars": 338
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/f1ef054a94e5632c/threshold_2x.png",
"psm": 11,
"score": 564,
"chars": 414
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/f1ef054a94e5632c/crop_1_2x.png",
"psm": 6,
"score": 47,
"chars": 47
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/f1ef054a94e5632c/crop_1_2x.png",
"psm": 11,
"score": 57,
"chars": 57
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/f1ef054a94e5632c/crop_2_2x.png",
"psm": 6,
"score": 51,
"chars": 51
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/f1ef054a94e5632c/crop_2_2x.png",
"psm": 11,
"score": 100,
"chars": 100
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/f1ef054a94e5632c/crop_3_2x.png",
"psm": 6,
"score": 305,
"chars": 155
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/f1ef054a94e5632c/crop_3_2x.png",
"psm": 11,
"score": 395,
"chars": 195
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/f1ef054a94e5632c/crop_4_2x.png",
"psm": 6,
"score": 200,
"chars": 150
},
{
"file": "/home/openclaw/.openclaw/workspace/knowledge/receipts/ocr/f1ef054a94e5632c/crop_4_2x.png",
"psm": 11,
"score": 131,
"chars": 81
}
]
Binary file not shown.

After

Width:  |  Height:  |  Size: 946 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 96 KiB

Binary file not shown.
+491
View File
@@ -0,0 +1,491 @@
#!/usr/bin/env python3
"""Local-first receipt/document importer for knowledge/receipts.
Pipeline:
1. Keep the original image/PDF immutable in knowledge/receipts/images/YY-MM-images/.
2. Generate local OCR derivatives in knowledge/receipts/ocr/<sha-prefix>/.
3. OCR locally with Tesseract/OCRmyPDF/Poppler.
4. Parse likely receipt fields with conservative heuristics.
5. Insert a searchable row into receipts.sqlite, or update an existing receipt id.
No cloud APIs. No deletion. Designed as a wrapper Soren can call quickly from chat.
"""
from __future__ import annotations
import argparse
import datetime as dt
import difflib
import hashlib
import json
import os
import re
import shutil
import sqlite3
import subprocess
import sys
import tempfile
from pathlib import Path
from typing import Any
try:
from PIL import Image, ImageEnhance, ImageOps
except Exception as e: # pragma: no cover
print(f"Pillow/PIL required: {e}", file=sys.stderr)
raise SystemExit(2)
ROOT = Path(__file__).resolve().parents[1]
DB = ROOT / "receipts.sqlite"
IMAGES = ROOT / "images"
OCR_ROOT = ROOT / "ocr"
TEXT_EXTS = {".txt"}
IMAGE_EXTS = {".jpg", ".jpeg", ".png", ".webp", ".tif", ".tiff", ".bmp"}
PDF_EXTS = {".pdf"}
def run(cmd: list[str], *, capture: bool = False, check: bool = True, timeout: int = 120) -> subprocess.CompletedProcess[str]:
try:
return subprocess.run(cmd, text=True, capture_output=capture, check=check, timeout=timeout)
except subprocess.CalledProcessError as e:
if e.stdout:
print(e.stdout, end="")
if e.stderr:
print(e.stderr, end="", file=sys.stderr)
raise SystemExit(e.returncode)
def have(name: str) -> bool:
return shutil.which(name) is not None
def sha256_file(path: Path) -> str:
h = hashlib.sha256()
with path.open("rb") as f:
for chunk in iter(lambda: f.read(1024 * 1024), b""):
h.update(chunk)
return h.hexdigest()
def now_local() -> dt.datetime:
return dt.datetime.now().astimezone()
def month_dir(ts: dt.datetime) -> Path:
return IMAGES / f"{ts:%y-%m}-images"
def slug(s: str) -> str:
s = re.sub(r"[^A-Za-z0-9._-]+", "-", s.strip()).strip("-_.")
return s[:80] or "receipt"
def copy_original(src: Path, *, category: str, captured: dt.datetime, sha: str) -> Path:
ext = src.suffix.lower() or ".jpg"
dest_dir = month_dir(captured)
dest_dir.mkdir(parents=True, exist_ok=True)
cat = slug(category.lower())
dest = dest_dir / f"{captured:%Y-%m-%d_%H%M}_{cat}_{sha[:8]}{ext}"
if not dest.exists():
shutil.copy2(src, dest)
return dest
def image_derivatives(src: Path, ocr_dir: Path) -> list[Path]:
ocr_dir.mkdir(parents=True, exist_ok=True)
im = Image.open(src).convert("RGB")
outputs: list[Path] = []
# 1. Original as PNG, useful for consistent tesseract input.
original = ocr_dir / "original.png"
im.save(original)
outputs.append(original)
# 2. Grayscale + contrast + upscale. Good for screenshots and receipts.
g = ImageOps.grayscale(im)
g = ImageEnhance.Contrast(g).enhance(2.2)
up = g.resize((g.width * 2, g.height * 2))
gray = ocr_dir / "gray_contrast_2x.png"
up.save(gray)
outputs.append(gray)
# 3. Binary-ish threshold, helps faint receipt paper.
bw = up.point(lambda p: 255 if p > 175 else 0)
bw_path = ocr_dir / "threshold_2x.png"
bw.save(bw_path)
outputs.append(bw_path)
# 4. Vertical crops, useful for long phone screenshots/receipts.
if im.height > im.width * 1.2:
for i in range(4):
y0 = i * im.height // 4
y1 = (i + 1) * im.height // 4
crop = im.crop((0, y0, im.width, y1))
crop = ImageEnhance.Contrast(ImageOps.grayscale(crop)).enhance(2.0)
crop = crop.resize((crop.width * 2, crop.height * 2))
p = ocr_dir / f"crop_{i+1}_2x.png"
crop.save(p)
outputs.append(p)
return outputs
def tesseract_text(image: Path, *, psm: int) -> str:
if not have("tesseract"):
raise RuntimeError("tesseract not installed")
cp = run(["tesseract", str(image), "stdout", "-l", "eng", "--psm", str(psm)], capture=True, check=False)
return cp.stdout or ""
def ocr_image(src: Path, ocr_dir: Path) -> dict[str, Any]:
variants = image_derivatives(src, ocr_dir)
runs = []
combined_parts: list[str] = []
for img in variants:
for psm in (6, 11):
text = tesseract_text(img, psm=psm)
cleaned = clean_text(text)
score = score_text(cleaned)
runs.append({"file": str(img), "psm": psm, "score": score, "chars": len(cleaned)})
if cleaned:
combined_parts.append(f"--- OCR {img.name} psm={psm} ---\n{cleaned}")
combined = dedupe_lines("\n".join(combined_parts))
(ocr_dir / "ocr_raw.txt").write_text(combined)
(ocr_dir / "ocr_runs.json").write_text(json.dumps(runs, indent=2))
return {"text": combined, "runs": runs, "dir": str(ocr_dir)}
def ocr_pdf(src: Path, ocr_dir: Path) -> dict[str, Any]:
ocr_dir.mkdir(parents=True, exist_ok=True)
raw_text = ""
if have("pdftotext"):
cp = run(["pdftotext", "-layout", str(src), "-"], capture=True, check=False)
raw_text = clean_text(cp.stdout or "")
ocr_pdf_path = None
if len(raw_text.strip()) < 40 and have("ocrmypdf"):
ocr_pdf_path = ocr_dir / "ocr.pdf"
run(["ocrmypdf", "--skip-text", "--deskew", "--rotate-pages", str(src), str(ocr_pdf_path)], check=False, timeout=300)
if ocr_pdf_path.exists() and have("pdftotext"):
cp = run(["pdftotext", "-layout", str(ocr_pdf_path), "-"], capture=True, check=False)
raw_text = clean_text(cp.stdout or raw_text)
(ocr_dir / "ocr_raw.txt").write_text(raw_text)
return {"text": raw_text, "runs": [{"pdf_ocr": bool(ocr_pdf_path and ocr_pdf_path.exists())}], "dir": str(ocr_dir)}
def clean_text(text: str) -> str:
text = text.replace("\r", "\n")
text = re.sub(r"[ \t]+", " ", text)
text = re.sub(r"\n{3,}", "\n\n", text)
return text.strip()
def dedupe_lines(text: str) -> str:
seen = set()
out = []
for line in text.splitlines():
key = re.sub(r"\s+", " ", line.strip()).lower()
if not key:
if out and out[-1] != "":
out.append("")
continue
# Preserve OCR section headers even when similar.
if key.startswith("--- ocr") or key not in seen:
out.append(line.rstrip())
seen.add(key)
return "\n".join(out).strip()
def score_text(text: str) -> int:
score = len(text)
for word in ("total", "subtotal", "gst", "receipt", "invoice", "order", "payment", "$"):
score += text.lower().count(word) * 50
return score
def money_to_float(s: str | None, *, require_dollars: bool = False, last: bool = False) -> float | None:
if not s:
return None
s = s.replace(",", "")
pat = r"-?\$\s*([0-9]+(?:\.[0-9]{2})?)" if require_dollars else r"-?\$?\s*([0-9]+(?:\.[0-9]{2})?)"
matches = list(re.finditer(pat, s))
if not matches:
return None
m = matches[-1] if last else matches[0]
return float(m.group(1))
def parse_date(text: str) -> str | None:
patterns = [
(r"\b(20\d{2})[-/](\d{1,2})[-/](\d{1,2})\b", "%Y-%m-%d"),
(r"\b(\d{1,2})[-/](\d{1,2})[-/](20\d{2})\b", "%d-%m-%Y"),
(r"\b([A-Z][a-z]+)\s+(\d{1,2}),\s*(20\d{2})\b", "%B %d %Y"),
(r"\b(\d{1,2})\s+([A-Z][a-z]+)\s+(20\d{2})\b", "%d %B %Y"),
]
for pat, fmt in patterns:
for m in re.finditer(pat, text):
val = " ".join(m.groups()) if "%B" in fmt else "-".join(m.groups())
try:
return dt.datetime.strptime(val, fmt).date().isoformat()
except ValueError:
continue
return None
def likely_merchant(lines: list[str], text: str = "") -> str | None:
# Specific useful hint: many web receipts contain the domain even when the logo OCR is noisy.
domain = re.search(r"\b(?:www\.)?([a-z0-9-]+)\.com\.au\b", text, re.I)
if domain:
known = {"nopong": "No Pong"}
key = domain.group(1).lower().replace("-", "")
if key in known:
return known[key]
return domain.group(1).replace("-", " ").title()
bad = re.compile(r"^(--- ocr|receipt|tax invoice|invoice|order|checkout|date|total|subtotal|payment|product|qty|amount|thank you)", re.I)
for line in lines[:30]:
l = line.strip(" :-|•")
if len(l) < 3 or bad.search(l) or re.search(r"@|https?://|www\.|\$|\d{4,}", l):
continue
# Prefer line with letters, not mostly symbols.
if sum(ch.isalpha() for ch in l) >= 3:
return l[:120]
return None
def parse_fields(text: str) -> dict[str, Any]:
lines = [l.strip() for l in text.splitlines() if l.strip()]
lower = text.lower()
fields: dict[str, Any] = {
"merchant_name": likely_merchant(lines, text),
"receipt_date": parse_date(text),
"currency": "AUD" if "$" in text else None,
}
# Order / invoice / transaction refs. Prefer same-line matches, then label-next-line matches.
ref_patterns = {
"docket_number": r"(?:order\s*(?:number|no\.?|#)|receipt\s*(?:number|no\.?|#)|docket\s*(?:number|no\.?|#))\s*[:#]?\s*([A-Z0-9-]{4,})",
"invoice_number": r"invoice\s*(?:number|no\.?|#)?\s*[:#]?\s*([A-Z0-9-]{4,})",
"transaction_reference": r"(?:transaction|txn|reference|ref)\s*(?:number|no\.?|#)?\s*[:#]?\s*([A-Z0-9-]{4,})",
}
for key, pat in ref_patterns.items():
m = re.search(pat, text, re.I)
if m and m.group(1).lower() not in {"date", "total", "email", "payment"}:
fields[key] = m.group(1)
for i, line in enumerate(lines[:-1]):
if "order number" in line.lower() and not fields.get("docket_number"):
# Common receipt layout: ORDER NUMBER: DATE: / 3465186 May 14, 2026
nxt = lines[i + 1]
m = re.search(r"\b([0-9]{4,}|[A-Z0-9-]{6,})\b", nxt)
if m:
fields["docket_number"] = m.group(1)
pm = re.search(r"payment\s*method\s*:?\s*([A-Za-z0-9 /_-]{3,40})", text, re.I)
if pm:
fields["payment_method"] = pm.group(1).strip()
else:
for method in ("PayPal", "Visa", "Mastercard", "EFTPOS", "Cash", "Amex", "Afterpay", "Apple Pay", "Google Pay"):
if method.lower() in lower:
fields["payment_method"] = method
break
# Amounts. Line-based parsing avoids confusing PRODUCT TOTAL headers and 85G sizes.
for line in lines:
low = line.lower()
amount_last = money_to_float(line, require_dollars=True, last=True)
amount_first = money_to_float(line, require_dollars=True, last=False)
if amount_last is None:
continue
if "subtotal" in low:
fields["subtotal"] = amount_last
if re.search(r"\bgst\b|\btax\b", low):
fields["gst"] = amount_last
if re.search(r"^(grand\s+total|amount\s+paid|total)\b", low):
# For "Total: $99.98 (includes $9.09 GST)", the total is the first amount.
fields["total"] = amount_first
# Embedded form: includes $9.09 GST
gst_m = re.search(r"includes\s+\$\s*([0-9,]+\.[0-9]{2})\s+GST", text, re.I)
if gst_m:
fields["gst"] = float(gst_m.group(1).replace(",", ""))
# Fallback total = largest money amount, conservative.
if not fields.get("total"):
monies = [money_to_float(m.group(0), require_dollars=True) for m in re.finditer(r"\$\s*[0-9,]+\.[0-9]{2}", text)]
monies = [m for m in monies if m is not None]
if monies:
fields["total"] = max(monies)
# Simple item extraction: lines with money, excluding summary/payment lines.
item_lines = []
seen_items = set()
skip = re.compile(r"subtotal|total|gst|tax|shipping|payment|points|redeemed|balance|change|cash|visa|mastercard|email|@", re.I)
for line in lines:
if "$" not in line or skip.search(line):
continue
amount = money_to_float(line, require_dollars=True, last=True)
desc = re.sub(r"\s*-?\$\s*[0-9,]+\.[0-9]{2}.*$", "", line).strip(" :-|•")
if amount is not None and len(desc) >= 3:
qty = None
qm = re.search(r"(?:x|×)\s*(\d+(?:\.\d+)?)\b", desc, re.I)
if qm:
qty = float(qm.group(1))
desc = re.sub(r"\s*(?:x|×)\s*\d+(?:\.\d+)?\b", "", desc, flags=re.I).strip()
norm_desc = re.sub(r"[^a-z0-9]+", " ", desc.lower()).strip()
key = (norm_desc, qty, amount)
if key in seen_items:
continue
# OCR variants can repeat the same item from different preprocessing runs.
duplicate_idx = None
for idx, existing in enumerate(item_lines):
if existing.get("quantity") == qty and existing.get("line_total") == amount:
ex_norm = re.sub(r"[^a-z0-9]+", " ", existing["description"].lower()).strip()
if difflib.SequenceMatcher(None, norm_desc, ex_norm).ratio() >= 0.92:
duplicate_idx = idx
break
if duplicate_idx is not None:
if len(desc) > len(item_lines[duplicate_idx]["description"]):
item_lines[duplicate_idx]["description"] = desc[:200]
continue
seen_items.add(key)
item_lines.append({"description": desc[:200], "quantity": qty, "line_total": amount})
fields["items"] = item_lines[:50]
return fields
def connect() -> sqlite3.Connection:
con = sqlite3.connect(DB)
con.row_factory = sqlite3.Row
return con
def existing_by_sha(con: sqlite3.Connection, sha: str) -> sqlite3.Row | None:
return con.execute("SELECT * FROM receipts WHERE image_sha256=?", (sha,)).fetchone()
def insert_or_update(con: sqlite3.Connection, *, source: Path, stored: Path, sha: str, parsed: dict[str, Any], text: str, ocr_info: dict[str, Any], args: argparse.Namespace) -> int:
captured = args.captured_at or now_local().isoformat(timespec="seconds")
confidence = "Local OCR via Tesseract. Parsed fields are heuristic; verify important warranty/tax data against original image."
ocr_json = json.dumps({"parsed": parsed, "ocr": ocr_info}, ensure_ascii=False)
values = {
"category": args.category,
"doc_type": args.doc_type,
"merchant_name": parsed.get("merchant_name"),
"receipt_date": parsed.get("receipt_date"),
"docket_number": parsed.get("docket_number"),
"invoice_number": parsed.get("invoice_number"),
"transaction_reference": parsed.get("transaction_reference"),
"payment_method": parsed.get("payment_method"),
"subtotal": parsed.get("subtotal"),
"gst": parsed.get("gst"),
"total": parsed.get("total"),
"currency": parsed.get("currency") or "AUD",
"image_path": str(stored),
"source_media_path": str(source),
"image_sha256": sha,
"ocr_raw_text": text,
"ocr_json": ocr_json,
"confidence_notes": confidence,
"captured_at": captured,
"notes": args.notes,
}
if args.receipt_id:
rid = args.receipt_id
assignments = ", ".join(f"{k}=?" for k in values.keys()) + ", updated_at=CURRENT_TIMESTAMP"
con.execute(f"UPDATE receipts SET {assignments} WHERE id=?", [*values.values(), rid])
else:
row = existing_by_sha(con, sha)
if row:
rid = int(row["id"])
if not args.update_existing:
return rid
assignments = ", ".join(f"{k}=?" for k in values.keys()) + ", updated_at=CURRENT_TIMESTAMP"
con.execute(f"UPDATE receipts SET {assignments} WHERE id=?", [*values.values(), rid])
else:
cols = ", ".join(values.keys())
qs = ", ".join("?" for _ in values)
cur = con.execute(f"INSERT INTO receipts ({cols}) VALUES ({qs})", list(values.values()))
rid = int(cur.lastrowid)
if args.replace_items:
con.execute("DELETE FROM receipt_items WHERE receipt_id=?", (rid,))
for item in parsed.get("items", []):
con.execute(
"INSERT INTO receipt_items (receipt_id, description, quantity, unit_price, line_total, notes) VALUES (?, ?, ?, ?, ?, ?)",
(rid, item.get("description"), item.get("quantity"), item.get("unit_price"), item.get("line_total"), "auto-parsed from OCR"),
)
con.commit()
return rid
def main() -> None:
ap = argparse.ArgumentParser(description="Import/OCR a receipt image/PDF into knowledge/receipts")
ap.add_argument("source", help="Path to receipt image/PDF")
ap.add_argument("--category", default="Uncategorised")
ap.add_argument("--doc-type", default="receipt")
ap.add_argument("--captured-at", help="ISO timestamp; default now")
ap.add_argument("--receipt-id", type=int, help="Update an existing receipts.id instead of inserting")
ap.add_argument("--notes", help="Receipt-level note from Michael/Soren, separate from OCR confidence notes")
ap.add_argument("--update-existing", action="store_true", help="If SHA already exists, update that row")
ap.add_argument("--replace-items", action=argparse.BooleanOptionalAction, default=True)
ap.add_argument("--dry-run", action="store_true", help="OCR/parse only; no DB write/copy")
ap.add_argument("--json", action="store_true", help="Print machine-readable result")
args = ap.parse_args()
src = Path(args.source).expanduser().resolve()
if not src.exists():
raise SystemExit(f"source not found: {src}")
if not DB.exists():
raise SystemExit(f"database not found: {DB}")
ext = src.suffix.lower()
sha = sha256_file(src)
captured = dt.datetime.fromisoformat(args.captured_at) if args.captured_at else now_local()
ocr_dir = OCR_ROOT / sha[:16]
if ext in IMAGE_EXTS:
ocr_info = ocr_image(src, ocr_dir)
elif ext in PDF_EXTS:
ocr_info = ocr_pdf(src, ocr_dir)
elif ext in TEXT_EXTS:
ocr_dir.mkdir(parents=True, exist_ok=True)
ocr_info = {"text": clean_text(src.read_text(errors="replace")), "runs": [], "dir": str(ocr_dir)}
(ocr_dir / "ocr_raw.txt").write_text(ocr_info["text"])
else:
raise SystemExit(f"unsupported file type: {ext}")
text = ocr_info["text"]
parsed = parse_fields(text)
result: dict[str, Any] = {"source": str(src), "sha256": sha, "ocr_dir": str(ocr_dir), "parsed": parsed}
if args.dry_run:
result["dry_run"] = True
else:
try:
# If the source is already inside the receipts image archive, don't duplicate it.
src.relative_to(IMAGES.resolve())
stored = src.relative_to(Path.cwd()) if src.is_relative_to(Path.cwd()) else src
except Exception:
stored = copy_original(src, category=args.category, captured=captured, sha=sha)
with connect() as con:
rid = insert_or_update(con, source=src, stored=stored, sha=sha, parsed=parsed, text=text, ocr_info=ocr_info, args=args)
result.update({"receipt_id": rid, "image_path": str(stored)})
if args.json:
print(json.dumps(result, indent=2, ensure_ascii=False))
else:
print(f"OCR dir: {result['ocr_dir']}")
if "receipt_id" in result:
print(f"Receipt ID: {result['receipt_id']}")
print(f"Image: {result['image_path']}")
print("Parsed:")
for k, v in parsed.items():
if k == "items":
print(f" items: {len(v)}")
for item in v[:10]:
print(f" - {item}")
else:
print(f" {k}: {v}")
if __name__ == "__main__":
main()