{"success": true, "mode": "discover", "query": "Karen Tan 6597630553 G3 prayer cycling blessing", "results": [{"session_id": "20260604_110036_c095e9e1", "when": "June 04, 2026 at 11:00 AM", "source": "webhook", "model": "glm-5.1", "title": "Naomi School Timetable Update", "matched_role": "user", "match_message_id": 101312, "snippet": "...draft_text\": \"Thanks >>>Karen<<<, will miss you both β€” praying for a good time too πŸ™\",\n \"extras\": {\n \"group_jid\": \"120363351787895353@g.us\",\n \"sender_name\": \">>>Karen<<< >>>Tan<<<\",\n \"sender_jid\": \">>>6597630553<<<@s.whatsapp.net\",\n \"category\": \">>>prayer<<>>blessing<<<\",\n \"original_message\": \"Happy >>>cycling<<<, to those who...", "bookend_start": [], "messages": [{"id": 101312, "role": "user", "content": "[IMPORTANT: The user has invoked the \"whatsapp-evolution\" skill, indicating they want you to follow its instructions. The full skill content is loaded below.]\n\n---\nname: whatsapp-evolution\ndescription: WhatsApp messaging via Evolution API β€” send, receive, analyze, and auto-reply. Use when processing an incoming WhatsApp message delivered via the webhook, replying on Vincent's behalf, liaising with contacts, sending messages, or handling WhatsApp-related tasks. Covers intent analysis, auto-reply rules, escalation (VIP β†’ phone call), conversation context fetching, and draft approvals via the jimmy-wa-drafts sidecar.\n---\n\n# WhatsApp via Evolution API\n\n## Credentials\nStored in `/opt/data/secrets/.whatsapp-evolution-credentials`:\n- `EVOLUTION_API_URL`, `EVOLUTION_API_KEY`, `EVOLUTION_INSTANCE`\n\nLoad with: `export $(cat .whatsapp-evolution-credentials | xargs)`\n\n## Sending Messages (direct, no approval)\n\n```bash\nexport $(cat .whatsapp-evolution-credentials | xargs)\ncurl -s -X POST \"${EVOLUTION_API_URL}/message/sendText/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"number\": \"6591234567\", \"text\": \"Hello!\"}'\n```\n\n- Phone format: country code, no `+` (e.g. `6590054438`)\n- Groups: use JID `{groupid}@g.us`\n- For media/location/contacts β€” see [references/api.md](references/api.md)\n\nUse direct send only when Vincent has explicitly commanded/approved the exact send, or for pre-approved auto-replies/follow-ups. For any reply that needs Vincent's review, use the sidecar (next section).\n\n### Direct replies when Vincent names an existing WhatsApp thread\n\nIf Vincent says to β€œreply to X” or β€œconfirm with X” and has clearly approved the action, but the contact JID/number is not in the current prompt:\n\n1. Search recent session history first (`session_search`) for the contact name, clinic/business name, or appointment details. Prior webhook summaries often contain the exact `remoteJid`, alternate phone JID, sender name, and context.\n2. If session history identifies the WhatsApp phone JID, send directly only if Vincent’s instruction is explicit enough to act. Keep the text concise and natural.\n3. If only a LID JID is found, look for `remoteJidAlt`/alternate phone JID in the session summary or fetch contacts/chats before sending.\n4. Do not invent phone numbers from web search when replying to an existing WhatsApp conversation. Use web search only as supporting context, not as the destination, unless Vincent explicitly asks to contact a public number.\n\n\n### Sending trip/camp details from retrieved emails\n\nWhen Vincent explicitly asks to send extracted school trip/camp details to a child or family member, send directly. Keep it practical and grounded:\n- Include dates, reporting/pick-up times, flights, venue/itinerary highlights, deadlines, forms, and safety/medication notes when found.\n- If an official packing list is missing, say that clearly. Do not invent one. You may add a short β€œbased on itinerary/weather, prepare…” list, clearly labeled as practical guidance rather than official school instructions.\n- Avoid long explanations about the search process in WhatsApp. Send the useful details.\n\n### Sending a local image/file as media\n\nFor user-approved media sends, base64-encode the local file and call `sendMedia`:\n\n```bash\nexport $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\npython3 - <<'PY'\nimport base64, json\npath = '/tmp/file.png'\npayload = {\n 'number': '6591234567',\n 'mediatype': 'image',\n 'mimetype': 'image/png',\n 'caption': 'Caption here',\n 'media': base64.b64encode(open(path, 'rb').read()).decode(),\n 'fileName': 'file.png',\n}\nopen('/tmp/wa_media_payload.json', 'w').write(json.dumps(payload))\nPY\ncurl -s --max-time 60 -X POST \"${EVOLUTION_API_URL}/message/sendMedia/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_media_payload.json\n```\n\nWhen sending QR tickets or passes, verify the crop/image contains the full QR code and the intended attendee name before sending; the QR screenshot is effectively the redeemable ticket.\n\n## Fetching Conversation Context\n\nBefore drafting or proposing any reply, always fetch recent chat history for context:\n\n```bash\npython3 /opt/data/scripts/wa_recent_messages.py \"6591234567@s.whatsapp.net\" 30\n```\n\nUse `limit=30` by default. For active group chats or ambiguous situations, fetch up to 50 messages. Read enough context to understand who is being discussed, current tone, previous commitments, and whether Vincent has already replied. The script returns newest messages first (descending order).\n\n## Draft Approvals β€” `jimmy-wa-drafts` sidecar\n\nWhen you need Vincent to approve a WhatsApp reply, POST the draft to the sidecar. It owns the entire approval flow: Telegram message with 4 inline buttons, Telegram WebApp for Edit/Context, sending to WhatsApp via Evolution API on Approve.\n\n### Posting a draft\n\nAlways include an `extras` object with the original WA context. The sidecar passes `extras` through verbatim if Vincent later clicks \"Context\" β€” without it, your redraft will be flying blind.\n\n```bash\ncat > /tmp/wa_draft.json <<'JSON'\n{\n \"contact\": \"120363351787895353@g.us\",\n \"contact_name\": \"Cycling group\",\n \"summary\": \"Karen sent a blessing over the bike ride; she's not joining\",\n \"draft_text\": \"Thanks Karen, will miss you both β€” praying for a good time too πŸ™\",\n \"extras\": {\n \"group_jid\": \"120363351787895353@g.us\",\n \"sender_name\": \"Karen Tan\",\n \"sender_jid\": \"6597630553@s.whatsapp.net\",\n \"category\": \"prayer/blessing\",\n \"original_message\": \"Happy cycling, to those who can make it… Dn Jeff and I had made plans for this PM so we won't be able to join y'all. Praying the Lord's favor, protection and good-weather blessings be upon y'all and your route!\",\n \"extracted\": {\n \"what\": \"Cycling from East Coast Park to Changi Village, then dinner\",\n \"when\": \"Today, 4:00 PM\",\n \"where\": \"East Coast Park to Changi Village\",\n \"who\": \"Group members who voted yes for cycling/dinner or dinner\"\n },\n \"confidence\": \"high\",\n \"why_appropriate\": \"Multiple group messages are blessings/wishes; Vincent has not replied; tone matches grace-based norm.\",\n \"wa_message_id\": \"ABC123XYZ\"\n }\n}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_draft.json\n```\n\nResponse: `{\"draft_id\":\"abc123def\",\"telegram_message_id\":42}`\n\nWhat to put in `extras` (use whatever makes sense for the message type):\n\n- **Always**: `original_message` (verbatim WA text), `sender_name`, `category` (e.g. `prayer`, `birthday`, `event`, `dm-reply`, `enquiry`)\n- **Group messages**: `group_jid`, `sender_jid`\n- **DMs**: `sender_jid`\n- **Strongly recommended (drives the rich Telegram render)**: `extracted` (dict of who/what/when/where), `confidence` (`high`/`medium`/`low`), `why_appropriate` (one-line reasoning shown to Vincent so he can decide)\n- **Optional**: `wa_message_id`, `tone_hint` (e.g. `warm-grace-based`), `sender_phone` (auto-derived from `sender_jid` if omitted)\n\nVincent receives a Telegram message in the `@jimmy_WA_bot` chat with 4 inline buttons. The sidecar handles every interaction; **you do not track replies, callbacks, or message IDs**. Just POST and move on.\n\n| Button | Behavior |\n|--------|----------|\n| **Approve** | Sidecar sends draft to WhatsApp |\n| **Edit** | Opens a Telegram WebApp with draft pre-filled; Vincent edits + sends; sidecar pushes to WhatsApp |\n| **Context** | Opens a Telegram WebApp with empty textarea + the original WA message visible; Vincent provides context. (When `HERMES_REDRAFT_WEBHOOK` is wired in the sidecar's env, this will POST back to Jimmy for a redraft. Until then, the context is just stored on the draft.) |\n| **Pass** | Sidecar deletes the draft |\n\n### JSON escaping when `draft_text` contains quotes\n\nSingle quotes (`It's`, `God's`) break shell interpolation. Write the JSON to a temp file first:\n\n```bash\ncat > /tmp/wa_draft.json <<'JSON'\n{\"contact\":\"6591234567@s.whatsapp.net\",\"contact_name\":\"John\",\"summary\":\"…\",\"draft_text\":\"It's nice to hear from you!\"}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_draft.json\n```\n\n### Networking\n\n`https://jimmy-draft.huper.asia` is the public HTTPS endpoint of the sidecar (TLS-terminated by Traefik on ubuntu-server). HTTPS is required because Hermes blocks plain-HTTP commands via tirith. The internal Docker DNS hostname http://jimmy-wa-v64mqe:8080 also works for non-shell calls but the agent is forbidden from using it.\n\n### Sidecar status & logs\n\n- Source: (auto-deploys on push)\n- Public URL: (`/health` for ping)\n- Container: `jimmy-wa-v64mqe.1.` on ubuntu-server\n- Bot: `@jimmy_WA_bot`\n\n## Handling a Redraft Trigger (`wa-redraft` webhook)\n\nWhen Vincent clicks the **Context** button on a draft, the sidecar:\n1. Edits the original Telegram message to \"πŸ”„ Adding context for X… agent is redrafting.\" (removes the old buttons so Vincent can't approve the now-stale text)\n2. POSTs to your `wa-redraft` webhook (`http://localhost:8644/webhooks/wa-redraft`) with the original draft fields plus Vincent's `context` instructions and the `extras` you stored when you first created the draft.\n\nWhen you receive a `wa-redraft` activation, the prompt template will lay out the fields. Your job:\n\n1. Read Vincent's `context` carefully β€” that's the steering signal. The `previous_draft` is just history, not necessarily a starting point.\n2. Re-fetch recent WA history if the original context isn't enough (`python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`). Use `extras.group_jid` for groups or `contact` for DMs.\n3. Write a NEW draft that addresses Vincent's instructions while staying faithful to the original WA context (in `extras.original_message`, `extras.extracted`).\n4. POST the new draft back to the sidecar **with the same `draft_id`** so the existing draft is updated, not duplicated:\n\n```bash\ncat > /tmp/wa_redraft.json <\",\n \"contact\": \"\",\n \"contact_name\": \"\",\n \"summary\": \"\",\n \"draft_text\": \"\",\n \"extras\": \n}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_redraft.json\n```\n\nThe sidecar will send a NEW Telegram message with 4 fresh buttons. Vincent can Approve, Edit, Context-again (loop), or Pass.\n\nDo **not** send to WhatsApp directly during a redraft β€” the approval flow is the sidecar's job.\n\n## Incoming Message Processing\n\n### Delivery Pipeline (current)\n1. Evolution API POSTs each `MESSAGES_UPSERT` webhook directly to Hermes at `http://localhost:8644/webhooks/whatsapp-incoming` (proxied via `jimmy.huper.asia/webhooks/whatsapp-incoming`)\n2. Hermes webhook route (`whatsapp-incoming`, dynamic subscription) validates and spawns an agent\n3. Agent loads `whatsapp-evolution` skill, follows the rules below\n4. Webhook subscription uses `deliver: log` because webhooks do not support `deliver: local`; the agent explicitly calls `send_message(target=\"telegram\", ...)` only for important/actionable/VIP/event/birthday/prayer/error items, OR posts a draft to the sidecar for approval items.\n\n**No more OpenClaw, no Python webhook server, no port 5001, no keepalive cron.** If messages aren't being processed, check: nginx proxy to port 8644, `hermes webhook list` for the subscription, Evolution API webhook URL setting.\n\n## Contact Rules & Escalation\n\nConfig: `memory/whatsapp-vip-contacts.json`\n\n### Priority Levels\n\n| Priority | Action | Example |\n|----------|--------|---------|\n| **VIP (call-always)** | **CALL Vincent immediately via Twilio without asking.** Do NOT ask for approval first. During quiet hours (1am-10am SGT) suppress call and use Telegram notification only, unless life-and-death emergency. Match VIPs by both phone JID (`659...@s.whatsapp.net`) and LID JID (`...@lid`) when present, because recent chats may arrive with `key.remoteJid` as an LID and the phone number only in `key.remoteJidAlt`. | Dn Jeff (+6597630553) |\n| **Family** | Always notify Vincent on Telegram | ps112 group, family members |\n| **Auto-reply** | Reply based on context file, notify Vincent | Repair shops |\n| **Normal** | Analyze intent, reply if appropriate, notify if important | Everyone else |\n| **Broadcast/Spam** | Ignore silently | Forwarded job posts, chain messages |\n\n### Quiet Hours\n- 1am-10am SGT (17:00-02:00 UTC): avoid phone calls, including for Dn Jeff / VIP call-always contacts\n- **Exception:** life-and-death emergencies always call regardless of quiet hours\n- During quiet hours, use Telegram notification only and mention that a call was suppressed due to quiet hours\n\n## Making a Call via Twilio\n\nCredentials: `/opt/data/secrets/.twilio-credentials-new` (`TWILIO_ACCOUNT_SID`, `TWILIO_AUTH_TOKEN`, `TWILIO_PHONE_NUMBER`). From: `+16614809287`, Vincent: `+6590054438`.\n\nPreferred script: `/opt/data/scripts/call_vincent.py`, now using the Twilio Python SDK baked into the container.\n\n```bash\npython3 /opt/data/scripts/call_vincent.py \"Hello Vincent, this is Jimmy. Urgent summary here.\"\n```\n\n## Action Rules\n\n### Default: DO NOT auto-reply directly, and do not spam main Telegram\nCollect intent and stay local unless something truly needs Vincent's immediate attention.\n\n**When you need Vincent to approve a reply, POST a draft to the sidecar (see \"Draft Approvals\" above).** The sidecar handles the Telegram notification with buttons, the WebApp for editing, and sending to WhatsApp on approve. Do **not** duplicate that draft or summary in the main Telegram chat.\n\nMain Telegram WhatsApp summaries are muted by default. Only use `send_message(target=\"telegram\", ...)` for urgent/important/actionable items that need Vincent now: VIP actionable messages/calls, missed/ringing family calls, emergencies, safety/medical/school-pickup issues, money/legal/account/security matters, explicit call-back/decision-now requests, or pipeline errors. Routine logistics, casual family pings, generic group events, links, standalone codes, normal confirmations, and suggested replies should be local-only or sidecar-only.\n\nDo NOT send WhatsApp replies directly unless Vincent explicitly approves them or the category is pre-approved.\n\n### Tone (when proposing or sending replies)\n- Casual, direct, no sign-offs (no \"Thanks & Regards\", no \"- Vincent\")\n- Ask questions instead of giving instructions\n- Keep it natural β€” like texting a friend\n- Christian and grace-based: warm, humble, encouraging, non-judgmental, peace-making\n- Never use em dashes in drafted replies or approval messages. Use commas, periods, colons, or parentheses instead.\n- For prayer/birthday contexts, use gentle faith-filled language; avoid sounding preachy, performative, or overly formal\n\n### What to Do With Each Message\n\n**DMs (personal chats):**\n- Analyze intent\n- Notify Vincent on Telegram (via `send_message(target=\"telegram\", ...)`) only if important/actionable/time-sensitive, requires a decision, comes from a VIP contact\n- If you have a useful suggested reply, POST it as a draft to the sidecar instead of a Telegram notification β€” Vincent gets the buttons inline\n- Casual/social/non-actionable DMs β†’ local log only, no Telegram notification, no draft\n- Do NOT reply unless Vincent says to\n- VIP contacts β†’ **call Vincent immediately without asking**. Do NOT post a draft. Just call. During quiet hours (1am-10am SGT), suppress the call and send Telegram notification only, unless life-and-death emergency.\n\n**Group chats β€” events/meetups:**\n- If someone posts about an event, gathering, meetup, dinner, etc:\n 1. Extract: what, when, where, who\n 2. Notify Vincent on Telegram with details (no draft needed unless you also want to suggest an RSVP reply)\n 3. Ask: \"Want me to add this to Google Calendar and set a reminder?\"\n\n**Group chats β€” prayer requests / praying for others:**\n- If someone asks for prayer, shares a burden, illness, grief, struggle, thanksgiving, or asks the group to pray:\n 1. Fetch recent group context first: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n 2. Identify who/what to pray for and any sensitive details\n 3. Draft a short, grace-based Christian reply/prayer\n 4. POST it to the sidecar with `summary` describing the request and `draft_text` being the prayer\n 5. Do NOT post to WhatsApp unless Vincent approves\n\n**Group chats β€” birthday greetings:**\n- If you see birthday wishes for someone in a group:\n 1. Fetch recent group context first: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n 2. Note who the birthday person is and relationship/context if inferable\n 3. Draft a warm, grace-based Christian birthday greeting\n 4. POST it to the sidecar\n 5. Do NOT post to WhatsApp unless Vincent approves\n\n**Group chats β€” general:**\n- Casual banter, memes, forwards, broadcast spam β†’ ignore silently\n- Religious content shares β†’ ignore unless specifically addressed to Vincent\n\n### When to Notify Vincent (Telegram) vs Post a Draft\n\n**Use `send_message(target=\"telegram\", ...)` sparingly** for plain notifications only when the message needs Vincent's immediate attention: VIP actionable DMs/calls, emergencies, school pickup/child logistics requiring action now, money/legal/account/security matters, explicit call-back/decision-now requests, or pipeline errors.\n\n**Use the sidecar (POST `/drafts`)** whenever you have a suggested reply for Vincent to approve: prayer responses, birthday greetings, DM replies, group RSVPs. Do not also send the same summary to the main Telegram chat.\n\nDo **not** notify or draft for casual/social/non-actionable DMs, routine logistics, read receipts/status updates, religious shares not addressed to Vincent, memes, forwarded/broadcast/spam, generic group events, standalone codes, normal confirmations, or general group chatter. Family messages are not automatically main-chat alerts unless urgent, safety-related, school/pickup critical, or explicitly requiring Vincent now.\n\n## Intent Categories\n\nWhen analyzing incoming messages, classify as:\n\n1. **Urgent** β€” time-sensitive, emergencies β†’ call Vincent outside quiet hours; during quiet hours use Telegram only unless life-and-death\n2. **Important** β€” decisions, money, appointments, anything consequential β†’ call Vincent outside quiet hours; during quiet hours notify on Telegram only unless life-threatening\n3. **Request** β€” asking Vincent to do something β†’ **notify Vincent** (Telegram) or post a draft if you have a safe reply\n4. **Enquiry** β€” asking for info, pricing, availability β†’ auto-reply if context available, otherwise post a draft\n5. **Confirmation** β€” confirming plans, bookings β†’ acknowledge and notify Vincent\n6. **Casual/Social** β€” greetings, memes, forwards β†’ usually ignore or react\n7. **Broadcast/Spam** β€” forwarded job posts, chain messages β†’ ignore\n\n## Finding Individual Contacts and JIDs\n\nWhen Vincent asks to monitor or prioritize a specific WhatsApp contact and only gives a name:\n\n1. Query contacts and chats, then search both display names and serialized message metadata:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n python3 - <<'PY'\n import os, requests, json\n base=os.environ['EVOLUTION_API_URL']; inst=os.environ['EVOLUTION_INSTANCE']; key=os.environ['EVOLUTION_API_KEY']\n headers={'apikey':key,'Content-Type':'application/json'}\n terms=['whye','keong'] # replace with lowercase search terms\n for ep in ['findContacts','findChats']:\n r=requests.post(f'{base}/chat/{ep}/{inst}', headers=headers, json={}, timeout=30)\n print('endpoint', ep, 'status', r.status_code)\n for obj in (r.json() if r.ok else []):\n if any(t in json.dumps(obj, ensure_ascii=False).lower() for t in terms):\n print(json.dumps(obj, ensure_ascii=False)[:1000])\n PY\n ```\n2. If `findChats` shows `remoteJid` ending in `@lid`, inspect `lastMessage.key.remoteJidAlt` for the phone JID. Store both in `memory/whatsapp-vip-contacts.json` when configuring VIP/call rules.\n3. Verify the number exists with `/chat/whatsappNumbers/{instance}` before finalizing.\n4. Validate the JSON config after editing: `python3 -m json.tool /opt/data/agent-state/memory/whatsapp-vip-contacts.json >/tmp/wa_vip_validated.json`.\n\n## Finding Group JIDs\n\nThe `group/fetchGroups` endpoint returns 404 on some Evolution API instances (including ours). To find a group JID when you don't have it:\n\n1. **Use `chat/findMessages` with an empty `where` filter** to pull recent messages across all chats, then extract unique `remoteJid` values containing `@g.us`.\n2. **Match by content**: Fetch messages from each group JID using `where.key.remoteJid` and match sender names or message text to identify the target group.\n3. **Known group JIDs** (store in `whatsapp-vip-contacts.json` as you discover them):\n - ps112 family group: `6591887488-1464061170@g.us`\n - Overcomers group: `6591444911-1614821287@g.us`\n - η₯žηš„εŽηΎŽ group: `6596756578-1405873840@g.us` (content-sharing, casual/broadcast, local log only)\n - Overcomers/TCians group: `120363230236878544@g.us` (church fellowship, PJ sends weekly attendance reports, routine attendance = local log only)\n\nExample search:\n```bash\ncurl -s -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"remoteJid\": \"6591444911-1614821287@g.us\"}}, \"limit\": 10, \"order\": \"desc\"}'\n```\n\n## Processing Image/Media Messages\n\nWhen an incoming WhatsApp message is an image (imageMessage), video, or other media with no text or only a caption:\n\n1. **Fetch conversation context** as usual with `wa_recent_messages.py`\n2. **Download the media** using the Evolution API:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"message\": {\"key\": {\"id\": \"MESSAGE_ID\", \"fromMe\": false, \"remoteJid\": \"GROUP_OR_CHAT_JID\"}}, \"convertToMp4\": false}' \\\n -o /tmp/wa_media_response.json\n ```\n Then decode: `python3 -c \"import json,base64; d=json.load(open('/tmp/wa_media_response.json')); open('/tmp/wa_media.jpg','wb').write(base64.b64decode(d['base64']))\"`\n3. **Analyze the image** using `vision_analyze` tool with `/tmp/wa_media.jpg` path and a context-aware question\n4. **Fallback if vision is unavailable**: Use `tesseract /tmp/wa_media.jpg stdout` for OCR, combined with pixel analysis (PIL/numpy) and conversation context to infer what the image contains\n5. **Album messages** (`albumMessage`): These bundle multiple images. Each image arrives as a separate `imageMessage` with a `messageAssociation.parentMessageKey` linking them to the album. Download each image separately. The `expectedImageCount` field tells how many images to expect\n6. **messageAssociation**: When images have `parentMessageKey` with `fromMe: true`, they are replying to a message Vincent sent. Use this for context\n\n### Audio / Voice Note Messages\nWhen an incoming WhatsApp message is `audioMessage` or a voice note (`ptt: true`), treat it like other media, but transcribe before intent classification when possible:\n\n1. **Fetch conversation context first**, especially in groups:\n ```bash\n python3 /opt/data/scripts/wa_recent_messages.py \"\" 50\n ```\n2. **Download the audio** with Evolution API, saving the response to a file:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n cat > /tmp/wa_audio_req.json <<'JSON'\n {\"message\":{\"key\":{\"id\":\"MESSAGE_ID\",\"fromMe\":false,\"remoteJid\":\"GROUP_OR_CHAT_JID\"}},\"convertToMp4\":false}\n JSON\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_audio_req.json -o /tmp/wa_audio_response.json\n python3 - <<'PY'\n import json, base64\n d=json.load(open('/tmp/wa_audio_response.json'))\n b=d.get('base64') or d.get('data',{}).get('base64')\n if ',' in b[:100]: b=b.split(',',1)[1]\n open('/tmp/wa_audio.ogg','wb').write(base64.b64decode(b))\n PY\n ```\n3. **Convert OGG/Opus to WAV**:\n ```bash\n ffmpeg -y -i /tmp/wa_audio.ogg -ar 16000 -ac 1 /tmp/wa_audio.wav\n ```\n4. **Transcribe fallback when Whisper is unavailable**: the container may not have `whisper`, `faster-whisper`, `torch`, or `speech_recognition` installed globally. A reliable lightweight fallback is a temp venv with SpeechRecognition and Google web speech:\n ```bash\n python3 -m venv /tmp/srvenv\n /tmp/srvenv/bin/pip install --quiet SpeechRecognition\n /tmp/srvenv/bin/python - <<'PY'\n import speech_recognition as sr\n r=sr.Recognizer()\n with sr.AudioFile('/tmp/wa_audio.wav') as source:\n audio=r.record(source)\n print(r.recognize_google(audio, language='en-SG'))\n PY\n ```\n5. **Check adjacent voice notes from the same sender**. WhatsApp voice notes are often sent in quick sequential chunks. Download and transcribe nearby `audioMessage` records from the same sender before deciding whether to notify Vincent.\n6. Use the transcript plus recent context to classify the message. For group meetup planning, extract what/when/where/who and notify Vincent if actionable, including transcript confidence.\n\n### Video Messages\n\nWhen an incoming WhatsApp message is `videoMessage` (with or without a caption):\n\n1. **Fetch conversation context** first, as with any media message: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n2. **Download the video** using Evolution API (same pattern as images/audio β€” save response to file, then decode):\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n cat > /tmp/wa_video_req.json <<'JSON'\n {\"message\":{\"key\":{\"id\":\"MESSAGE_ID\",\"fromMe\":false,\"remoteJid\":\"CONTACT_JID\"}},\"convertToMp4\":false}\n JSON\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_video_req.json -o /tmp/wa_video_response.json\n python3 - <<'PY'\n import json, base64\n d = json.load(open('/tmp/wa_video_response.json'))\n b = d.get('base64') or d.get('data', {}).get('base64', '')\n if ',' in b[:100]: b = b.split(',', 1)[1]\n open('/tmp/wa_video.mp4', 'wb').write(base64.b64decode(b))\n print(f\"Video saved: {len(base64.b64decode(b))} bytes\")\n PY\n ```\n3. **Extract representative frames** with ffmpeg (do NOT try to analyze video directly β€” vision tools work on images):\n ```bash\n # Frame at start\n ffmpeg -y -ss 0 -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame0.jpg\n # Frame at ~1/3 of the way (good for short videos)\n ffmpeg -y -ss 3 -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame3s.jpg\n # If video is longer (>15s), also extract at midpoint\n # ffprobe -v quiet -print_format json -show_format /tmp/wa_video.mp4 # get duration\n # ffmpeg -y -ss $HALF_DURATION -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame_mid.jpg\n ```\n4. **Analyze the frames** with `vision_analyze`. Ask specific questions about content: what objects/people/text are visible, what type of video this appears to be (screen recording, personal clip, forwarded meme, etc.), and any text or identifying details.\n5. **Use the caption** from `videoMessage.caption` if present (can be empty/null). Combine frame analysis + caption + conversation context to classify the message intent.\n6. **Transcription of audio in video** is generally not feasible with available tools. If the video likely contains important spoken content (e.g. voice memo style), note this limitation and flag it to Vincent.\n\n**Key points:**\n- Always use `-o` flag and file-based decode; videos can be 1-10MB and base64 responses will be even larger\n- `--max-time 25` on curl is essential for video downloads\n- Short casual clips (showing a key fob, a parking spot, quick view of surroundings) are usually casual/social β€” classify accordingly and don't over-escalate\n- If `vision_analyze` fails (400 on models that don't support images, or 429 rate limit), fall back to conversation context inference alone\n\n### Poll Messages\nWhen `pollUpdateMessage` appears in chat history (someone voted), the message contains a `pollCreationMessageKey` with the original poll's message `id`. To get the poll question and options, query:\n```bash\ncurl -s -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"id\": \"POLL_MESSAGE_ID\", \"remoteJid\": \"GROUP_JID\"}}, \"limit\": 1}'\n```\nThe response will have `pollCreationMessageV3` with `name` (the poll question) and `options` array. Always look up the poll when votes or poll references appear, since the poll text contains the event details.\n\n### `associatedChildMessage` (Motion Photos / Companion Media)\nWhatsApp motion photos and companion media arrive as `associatedChildMessage` type. These are short videos or supplementary images attached to a parent message via `messageAssociation.parentMessageKey`. The structure is:\n\n```\nmessage: {\n messageContextInfo: {\n messageAssociation: {\n associationType: 12,\n parentMessageKey: { id, fromMe, remoteJid }\n }\n },\n associatedChildMessage: {\n message: {\n videoMessage: { ... } // or imageMessage: { ... }\n }\n }\n}\n```\n\nKey points:\n- The `parentMessageKey.id` links back to the main message (the image/video they accompany)\n- When `parentMessageKey.fromMe: true`, the companion media is replying to a message Vincent sent\n- Captions may appear on the child's `videoMessage.caption` or `imageMessage.caption` inside the nested message, NOT at the top level\n- These often contain motion-photo short videos (3 seconds) or album companion images\n- Treat them as media attachments to the parent; don't process them as standalone messages requiring independent intent analysis\n\n### Sequential Image Sets (Non-Album)\nMultiple images sent rapidly by the same sender at similar timestamps may NOT be wrapped in an `albumMessage`. They appear as separate `imageMessage` entries. Always check for adjacent images from the same sender and download+OCR all of them together, as they often form a coherent multi-page set (e.g., route info, comparison tables, maps). Missing one can lose important context.\n\n### Key pitfalls\n- The Evolution API `getBase64FromMediaMessage` endpoint returns the full base64 payload (can be 100KB+ for images). Use `-o` flag to save to file, then parse with Python. Do NOT try to pipe it directly\n- `curl` with `--max-time 25` is essential; without it, large media responses can hang\n- The `albumMessage` type itself has no `fromMe` field; check each individual `imageMessage` instead\n- Image thumbnails in the webhook payload are truncated; always download the full image for analysis\n- **WhatsApp media URLs expire**: CDN download links (`mmg.whatsapp.net`) expire after roughly 24-48 hours. When `getBase64FromMediaMessage` returns a 400/404 error like \"Failed to fetch stream from https://mmg.whatsapp.net/...\", the media is no longer retrievable via the API. This means you CANNOT download adjacent/older voice notes, images, or documents for context if they're more than a day or two old. Fall back to conversation text context only. Fresh/incoming messages are almost always still downloadable.\n- **Voice note transcription reliability**: Short voice notes (<15s), especially in Singlish/Chinese-English mix common in Singapore contacts, often produce unreliable transcriptions via SpeechRecognition+Google. Always flag low-confidence transcriptions to Vincent rather than acting on potentially garbled text.\n- **Vision API failures**: `vision_analyze` can fail with 429 (rate limit) **or 400** (model doesn't support image input, e.g. glm-5.1 via ollama-cloud). Always have a fallback plan. When vision is unavailable, try tesseract OCR, then pixel analysis (PIL/numpy for brightness zones, text region detection via row-level dark-pixel density)\n- **OCR can fail on decorative/complex images**: Obituaries, memorial notices, religious flyers with ornate fonts, or dark-background images often defeat tesseract even with aggressive preprocessing (upscaling, thresholding, inverting, PSM mode variations). When OCR returns gibberish or empty, fall back to conversation context inference: use preceding/surrounding messages to determine what the image likely contains, then draft an appropriate response based on that context\n- **OCR on devotional/dark-background images**: These often defeat tesseract on full-image scans. Use a multi-strategy approach: (1) run tesseract on the full image first, (2) if text is sparse, crop into regions (top 15%, middle, bottom 15%), upscale 2-3x, enhance contrast (2-3x), and try both normal and inverted (`ImageOps.invert`) versions with `--psm 6`, (3) combine all partial OCR results. For Exodus/verse images, the title (e.g. \"WHEN MOSES' HANDS WERE\") is often in decorative font that OCR misses, while the verse body text is usually readable.\n- **Prioritize conversation context**: When processing group images, always fetch recent context FIRST (before attempting vision/OCR). If image analysis fails, the conversation history often makes the image's purpose clear enough to draft a meaningful reply\n- **DM conversation context is one-sided (fromMe only)**: The Evolution API `findMessages` endpoint often returns only Vincent's sent messages (`fromMe: true`) for DM chats, not the contact's incoming messages. This means `wa_recent_messages.py` can show only one side of the conversation. When processing a DM webhook, do NOT rely solely on fetched history for the contact's side. The webhook payload already contains the incoming message text. Use what the webhook gives you plus Vincent's side from the API. If you need the contact's recent messages, check if `fromMe: false` records exist; if none are returned, the API simply doesn't have them and you must work with what you have (webhook message + Vincent's replies for context).\n- **`findMessages` ignores `key.fromMe` filter**: Passing `\"key.fromMe\": false` in the `where` clause still returns all messages (including `fromMe: true`). To find messages from a specific sender, fetch results and filter client-side by checking `fromMe` in Python.\n- **`findMessages` caps at ~50 records per request**: Even with `limit: 200` or `limit: 500`, the API returns at most 50 records. When one party sends many consecutive messages, the other party's messages may not appear in the first 50 results. Use pagination or fetch multiple pages if you need older messages.\n- **Video frame extraction**: Use `ffmpeg -y -ss N -i video.mp4 -frames:v 1 -update 1 output.jpg` (the `-update 1` flag is needed; without it ffmpeg treats the output as an image sequence and fails). Extract frames at multiple timestamps (start, ~1/3, midpoint for longer videos) since a single frame may miss the key content.\n- **ffmpeg \"Invalid color space\" failure on WhatsApp videos**: Some WhatsApp H.264 videos have `color_space=reserved` and `color_primaries=reserved` metadata that ffmpeg cannot decode at all. Every codec/format/filter combination fails with \"Invalid color space\" / return code 234. This is NOT fixable with `-vf setparams`, `-color_trc`, `-colorspace`, `-color_primaries`, or `-vf format=` flags because the error occurs at the decoder level before any filter runs. **Do not waste time trying ffmpeg workarounds.** When this happens, fall back to: (1) conversation context inference (who sent it, what thread it's in, adjacent messages), (2) the jpegThumbnail embedded in the videoMessage (small but gives a rough idea; base64-encoded JPEG, typically 600-900 bytes, use `vision_analyze` on it if available), or (3) if all analysis fails, just classify based on surrounding text context.\n- **`wa_recent_messages.py` JSON output breaks `json.loads()`**: The script's stdout contains WhatsApp message content with control characters (emoji, CJK, deviceListMetadata binary blobs) that cause `json.loads()` to fail with `Invalid control character` or `Expecting ',' delimiter` errors. Even `strict=False` does not reliably fix it. **Reliable approach**: use `curl -o /tmp/wa_msgs.json` to save the Evolution API response directly to a file (the file write handles binary safely), then parse with `json.load(open('/tmp/wa_msgs.json'))` in Python. Example:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n curl -s --max-time 15 -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"remoteJid\": \"CONTACT_JID\"}}, \"limit\": 20}' -o /tmp/wa_msgs.json\n ```\n Then in Python: `data = json.load(open('/tmp/wa_msgs.json')); records = data['messages']['records']`\n\n## API Reference\n\nFor full API endpoints (media, contacts, location, buttons) β€” see [references/api.md](references/api.md).\n\n[Skill directory: /opt/data/skills/personal/whatsapp-evolution]\nResolve any relative paths in this skill (e.g. `scripts/foo.js`, `templates/config.yaml`) against that directory, then run them with the terminal tool using the absolute path.\n\n[This skill has supporting files:]\n- references/api.md -> /opt/data/skills/personal/whatsapp-evolution/references/api.md\n- scripts/process_queue.py -> /opt/data/skills/personal/whatsapp-evolution/scripts/process_queue.py\n- scripts/send_text.sh -> /opt/data/skills/personal/whatsapp-evolution/scripts/send_text.sh\n\nLoad any of these with skill_view(name=\"personal/whatsapp-evolution\", file_path=\"\"), or run scripts directly by absolute path (e.g. `node /opt/data/skills/personal/whatsapp-evolution/scripts/foo.js`).\n\nThe user has provided the following instruction alongside the skill invocation: Incoming WhatsApp webhook received.\n\nEvent: messages.upsert\nInstance: n8n-WA\nSender: Shalom Evonne\nSender JID: 120363418526182134@g.us\nFrom me: False\nMessage type: conversation\nMessage: {\n \"event\": \"messages.upsert\",\n \"instance\": \"n8n-WA\",\n \"data\": {\n \"key\": {\n \"remoteJid\": \"120363418526182134@g.us\",\n \"fromMe\": false,\n \"id\": \"3EB0B35E196CCC497A358F\",\n \"participant\": \"131503308673050@lid\",\n \"participantAlt\": \"6591887488@s.whatsapp.net\",\n \"addressingMode\": \"lid\"\n },\n \"pushName\": \"Shalom Evonne\",\n \"status\": \"DELIVERY_ACK\",\n \"message\": {\n \"conversation\": \"Good evening Shalom Family!\\n\\nTomorrow is Friday, Week Odd.\\n\\nNaomi schedule (G2/G3):\\n8:00am - Morning Assembly\\n8:25am - CCE\\n9:00am - English Language\\n9:35am - Recess\\n10:10am - R recess\\n10:45am - Music\\n11:20am - Needlework\\n11:55am - Geography\\n\\nDismissal: 12:30pm\\n\\nNo PE tomorrow, so no need to pack PE gear tonight.\\n\\nHave a restful evening and blessed night! Trust the Lord to refresh you all as you rest in Him. \\ud83d\\ude4f\\n\\nWith love,\\nShalom Evonne\"\n },\n \"contextInfo\": null,\n \"messageType\": \"conversation\",\n \"messageTimestamp\": 1780570836,\n \"instanceId\": \"b2bfaa30-d5f1-4b22-b979-791892619414\",\n \"source\": \"web\"\n },\n \"destination\": \"https://jimmy.huper.asia/webhooks/whatsapp-incoming\",\n \"date_time\": \"2026-06-04T08:00:36.440Z\",\n \"sender\": \"6590054438@s.whatsapp.net\",\n \"server_url\": \"https://evolution.huper.asia\",\n \"apikey\": \"5E4C532223A0-460B-A749-16C584F56E74\"\n}\n\nPrimary goal: REDUCE NOISE. Stay silent in the main Telegram chat unless the message truly needs Vincent's immediate attention. Approval-worthy reply drafts should go to the jimmy-wa-drafts sidecar, NOT to the main Telegram chat.\n\nFollow the whatsapp-evolution skill and /opt/data/agent-state/memory/whatsapp-vip-contacts.json, with these stricter notification rules:\n\nABSOLUTE SILENCE RULES:\n- If fromMe is true: ignore. Do NOT call send_message. Final response local log only.\n- Ignore status/update/read-receipt events or payloads with no real message content.\n- Do NOT send main Telegram summaries for ordinary DMs, routine logistics, casual family messages, generic group events, group chatter, forwards, memes, devotional/religious shares, links, standalone codes, normal confirmations, or suggested replies.\n- Do NOT ask \"Want me to add this to Calendar?\" for WhatsApp group events unless Vincent is explicitly invited, named, responsible, or the event is time-sensitive and clearly relevant to him today.\n- Do NOT surface family messages just because they are family. Only surface if urgent, safety-related, school/pickup critical, money/legal/medical, or explicitly requires Vincent now.\n\nSIDEcar RULES:\n- If a reply may be useful but it is not urgent, POST it to https://jimmy-draft.huper.asia/drafts using the whatsapp-evolution Draft Approvals flow. Do NOT duplicate that draft/summary in the main Telegram chat.\n- Prayer and birthday replies should go to the sidecar when appropriate, not the main Telegram chat, unless the underlying situation is urgent/sensitive and Vincent needs immediate awareness.\n- Before drafting any reply or approval suggestion, fetch recent context: python3 /opt/data/scripts/wa_recent_messages.py \"120363418526182134@g.us\" 30. For active/ambiguous groups use 50.\n\nMAIN TELEGRAM ALERTS ARE ALLOWED ONLY FOR:\n1. VIP contacts marked always_call or always_notify when the message is important/actionable, or any VIP call event.\n2. Calls from family/VIP that appear missed/ringing and likely need immediate attention. Avoid repeated alerts for accepted/terminated duplicate call state updates.\n3. Emergencies, safety issues, medical issues, school pickup/child logistics requiring action now, legal/money/account/security matters, or time-sensitive commitments in the next few hours.\n4. Messages explicitly asking Vincent to call back, decide now, approve payment, provide OTP/code for an active task, or respond urgently.\n5. Pipeline errors or webhook processing failures.\n\nCALL RULES:\n- VIP contacts (e.g. Dn Jeff, Ng Whye Keong): call Vincent immediately outside quiet hours when actionably important or on ringing/missed call. During 1am-10am SGT, Telegram only unless life-and-death emergency.\n- Do not call for casual VIP confirmations or non-actionable courtesy messages.\n\nTONE/RULES FOR DRAFTS:\n- Proposed replies must be Christian and grace-based where appropriate: warm, humble, encouraging, non-judgmental, peace-making, not preachy.\n- Never use em dashes in drafted replies or approval messages. Use commas, periods, colons, or parentheses instead.\n- NEVER auto-reply directly on WhatsApp unless Vincent explicitly approves or future config marks that exact category pre-approved.\n\nDelivery rule:\n- For allowed main alerts only: call send_message(target=\"telegram\", message=...). Keep it concise.\n- For non-urgent reply suggestions: use the sidecar only.\n- For everything else: DO NOT call send_message. Keep final response as local-only log.\n", "timestamp": 1780570848.0694947, "anchor": true}, {"id": 101313, "role": "assistant", "content": "**Local log only. No action needed.**\n\nClassification: **Routine logistics** β€” school timetable for Naomi in a group chat. Not urgent, not pickup-critical, not time-sensitive, and no reply expected.\n\n- Group: `120363418526182134@g.us`\n- Sender: Shalom Evonne (`6591887488@s.whatsapp.net`)\n- Content: Naomi's Friday schedule (G2/G3), dismissal 12:30pm, no PE tomorrow\n- Action: None. No Telegram notification, no sidecar draft.", "timestamp": 1780570848.0895429}], "bookend_end": [], "messages_before": 0, "messages_after": 2}, {"session_id": "20260603_110828_3d39ab70", "when": "June 03, 2026 at 11:08 AM", "source": "webhook", "model": "glm-5.1", "title": "WhatsApp School Schedule Classification", "matched_role": "user", "match_message_id": 99974, "snippet": "...draft_text\": \"Thanks >>>Karen<<<, will miss you both β€” praying for a good time too πŸ™\",\n \"extras\": {\n \"group_jid\": \"120363351787895353@g.us\",\n \"sender_name\": \">>>Karen<<< >>>Tan<<<\",\n \"sender_jid\": \">>>6597630553<<<@s.whatsapp.net\",\n \"category\": \">>>prayer<<>>blessing<<<\",\n \"original_message\": \"Happy >>>cycling<<<, to those who...", "bookend_start": [], "messages": [{"id": 99974, "role": "user", "content": "[IMPORTANT: The user has invoked the \"whatsapp-evolution\" skill, indicating they want you to follow its instructions. The full skill content is loaded below.]\n\n---\nname: whatsapp-evolution\ndescription: WhatsApp messaging via Evolution API β€” send, receive, analyze, and auto-reply. Use when processing an incoming WhatsApp message delivered via the webhook, replying on Vincent's behalf, liaising with contacts, sending messages, or handling WhatsApp-related tasks. Covers intent analysis, auto-reply rules, escalation (VIP β†’ phone call), conversation context fetching, and draft approvals via the jimmy-wa-drafts sidecar.\n---\n\n# WhatsApp via Evolution API\n\n## Credentials\nStored in `/opt/data/secrets/.whatsapp-evolution-credentials`:\n- `EVOLUTION_API_URL`, `EVOLUTION_API_KEY`, `EVOLUTION_INSTANCE`\n\nLoad with: `export $(cat .whatsapp-evolution-credentials | xargs)`\n\n## Sending Messages (direct, no approval)\n\n```bash\nexport $(cat .whatsapp-evolution-credentials | xargs)\ncurl -s -X POST \"${EVOLUTION_API_URL}/message/sendText/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"number\": \"6591234567\", \"text\": \"Hello!\"}'\n```\n\n- Phone format: country code, no `+` (e.g. `6590054438`)\n- Groups: use JID `{groupid}@g.us`\n- For media/location/contacts β€” see [references/api.md](references/api.md)\n\nUse direct send only when Vincent has explicitly commanded/approved the exact send, or for pre-approved auto-replies/follow-ups. For any reply that needs Vincent's review, use the sidecar (next section).\n\n### Direct replies when Vincent names an existing WhatsApp thread\n\nIf Vincent says to β€œreply to X” or β€œconfirm with X” and has clearly approved the action, but the contact JID/number is not in the current prompt:\n\n1. Search recent session history first (`session_search`) for the contact name, clinic/business name, or appointment details. Prior webhook summaries often contain the exact `remoteJid`, alternate phone JID, sender name, and context.\n2. If session history identifies the WhatsApp phone JID, send directly only if Vincent’s instruction is explicit enough to act. Keep the text concise and natural.\n3. If only a LID JID is found, look for `remoteJidAlt`/alternate phone JID in the session summary or fetch contacts/chats before sending.\n4. Do not invent phone numbers from web search when replying to an existing WhatsApp conversation. Use web search only as supporting context, not as the destination, unless Vincent explicitly asks to contact a public number.\n\n\n### Sending trip/camp details from retrieved emails\n\nWhen Vincent explicitly asks to send extracted school trip/camp details to a child or family member, send directly. Keep it practical and grounded:\n- Include dates, reporting/pick-up times, flights, venue/itinerary highlights, deadlines, forms, and safety/medication notes when found.\n- If an official packing list is missing, say that clearly. Do not invent one. You may add a short β€œbased on itinerary/weather, prepare…” list, clearly labeled as practical guidance rather than official school instructions.\n- Avoid long explanations about the search process in WhatsApp. Send the useful details.\n\n### Sending a local image/file as media\n\nFor user-approved media sends, base64-encode the local file and call `sendMedia`:\n\n```bash\nexport $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\npython3 - <<'PY'\nimport base64, json\npath = '/tmp/file.png'\npayload = {\n 'number': '6591234567',\n 'mediatype': 'image',\n 'mimetype': 'image/png',\n 'caption': 'Caption here',\n 'media': base64.b64encode(open(path, 'rb').read()).decode(),\n 'fileName': 'file.png',\n}\nopen('/tmp/wa_media_payload.json', 'w').write(json.dumps(payload))\nPY\ncurl -s --max-time 60 -X POST \"${EVOLUTION_API_URL}/message/sendMedia/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_media_payload.json\n```\n\nWhen sending QR tickets or passes, verify the crop/image contains the full QR code and the intended attendee name before sending; the QR screenshot is effectively the redeemable ticket.\n\n## Fetching Conversation Context\n\nBefore drafting or proposing any reply, always fetch recent chat history for context:\n\n```bash\npython3 /opt/data/scripts/wa_recent_messages.py \"6591234567@s.whatsapp.net\" 30\n```\n\nUse `limit=30` by default. For active group chats or ambiguous situations, fetch up to 50 messages. Read enough context to understand who is being discussed, current tone, previous commitments, and whether Vincent has already replied. The script returns newest messages first (descending order).\n\n## Draft Approvals β€” `jimmy-wa-drafts` sidecar\n\nWhen you need Vincent to approve a WhatsApp reply, POST the draft to the sidecar. It owns the entire approval flow: Telegram message with 4 inline buttons, Telegram WebApp for Edit/Context, sending to WhatsApp via Evolution API on Approve.\n\n### Posting a draft\n\nAlways include an `extras` object with the original WA context. The sidecar passes `extras` through verbatim if Vincent later clicks \"Context\" β€” without it, your redraft will be flying blind.\n\n```bash\ncat > /tmp/wa_draft.json <<'JSON'\n{\n \"contact\": \"120363351787895353@g.us\",\n \"contact_name\": \"Cycling group\",\n \"summary\": \"Karen sent a blessing over the bike ride; she's not joining\",\n \"draft_text\": \"Thanks Karen, will miss you both β€” praying for a good time too πŸ™\",\n \"extras\": {\n \"group_jid\": \"120363351787895353@g.us\",\n \"sender_name\": \"Karen Tan\",\n \"sender_jid\": \"6597630553@s.whatsapp.net\",\n \"category\": \"prayer/blessing\",\n \"original_message\": \"Happy cycling, to those who can make it… Dn Jeff and I had made plans for this PM so we won't be able to join y'all. Praying the Lord's favor, protection and good-weather blessings be upon y'all and your route!\",\n \"extracted\": {\n \"what\": \"Cycling from East Coast Park to Changi Village, then dinner\",\n \"when\": \"Today, 4:00 PM\",\n \"where\": \"East Coast Park to Changi Village\",\n \"who\": \"Group members who voted yes for cycling/dinner or dinner\"\n },\n \"confidence\": \"high\",\n \"why_appropriate\": \"Multiple group messages are blessings/wishes; Vincent has not replied; tone matches grace-based norm.\",\n \"wa_message_id\": \"ABC123XYZ\"\n }\n}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_draft.json\n```\n\nResponse: `{\"draft_id\":\"abc123def\",\"telegram_message_id\":42}`\n\nWhat to put in `extras` (use whatever makes sense for the message type):\n\n- **Always**: `original_message` (verbatim WA text), `sender_name`, `category` (e.g. `prayer`, `birthday`, `event`, `dm-reply`, `enquiry`)\n- **Group messages**: `group_jid`, `sender_jid`\n- **DMs**: `sender_jid`\n- **Strongly recommended (drives the rich Telegram render)**: `extracted` (dict of who/what/when/where), `confidence` (`high`/`medium`/`low`), `why_appropriate` (one-line reasoning shown to Vincent so he can decide)\n- **Optional**: `wa_message_id`, `tone_hint` (e.g. `warm-grace-based`), `sender_phone` (auto-derived from `sender_jid` if omitted)\n\nVincent receives a Telegram message in the `@jimmy_WA_bot` chat with 4 inline buttons. The sidecar handles every interaction; **you do not track replies, callbacks, or message IDs**. Just POST and move on.\n\n| Button | Behavior |\n|--------|----------|\n| **Approve** | Sidecar sends draft to WhatsApp |\n| **Edit** | Opens a Telegram WebApp with draft pre-filled; Vincent edits + sends; sidecar pushes to WhatsApp |\n| **Context** | Opens a Telegram WebApp with empty textarea + the original WA message visible; Vincent provides context. (When `HERMES_REDRAFT_WEBHOOK` is wired in the sidecar's env, this will POST back to Jimmy for a redraft. Until then, the context is just stored on the draft.) |\n| **Pass** | Sidecar deletes the draft |\n\n### JSON escaping when `draft_text` contains quotes\n\nSingle quotes (`It's`, `God's`) break shell interpolation. Write the JSON to a temp file first:\n\n```bash\ncat > /tmp/wa_draft.json <<'JSON'\n{\"contact\":\"6591234567@s.whatsapp.net\",\"contact_name\":\"John\",\"summary\":\"…\",\"draft_text\":\"It's nice to hear from you!\"}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_draft.json\n```\n\n### Networking\n\n`https://jimmy-draft.huper.asia` is the public HTTPS endpoint of the sidecar (TLS-terminated by Traefik on ubuntu-server). HTTPS is required because Hermes blocks plain-HTTP commands via tirith. The internal Docker DNS hostname http://jimmy-wa-v64mqe:8080 also works for non-shell calls but the agent is forbidden from using it.\n\n### Sidecar status & logs\n\n- Source: (auto-deploys on push)\n- Public URL: (`/health` for ping)\n- Container: `jimmy-wa-v64mqe.1.` on ubuntu-server\n- Bot: `@jimmy_WA_bot`\n\n## Handling a Redraft Trigger (`wa-redraft` webhook)\n\nWhen Vincent clicks the **Context** button on a draft, the sidecar:\n1. Edits the original Telegram message to \"πŸ”„ Adding context for X… agent is redrafting.\" (removes the old buttons so Vincent can't approve the now-stale text)\n2. POSTs to your `wa-redraft` webhook (`http://localhost:8644/webhooks/wa-redraft`) with the original draft fields plus Vincent's `context` instructions and the `extras` you stored when you first created the draft.\n\nWhen you receive a `wa-redraft` activation, the prompt template will lay out the fields. Your job:\n\n1. Read Vincent's `context` carefully β€” that's the steering signal. The `previous_draft` is just history, not necessarily a starting point.\n2. Re-fetch recent WA history if the original context isn't enough (`python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`). Use `extras.group_jid` for groups or `contact` for DMs.\n3. Write a NEW draft that addresses Vincent's instructions while staying faithful to the original WA context (in `extras.original_message`, `extras.extracted`).\n4. POST the new draft back to the sidecar **with the same `draft_id`** so the existing draft is updated, not duplicated:\n\n```bash\ncat > /tmp/wa_redraft.json <\",\n \"contact\": \"\",\n \"contact_name\": \"\",\n \"summary\": \"\",\n \"draft_text\": \"\",\n \"extras\": \n}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_redraft.json\n```\n\nThe sidecar will send a NEW Telegram message with 4 fresh buttons. Vincent can Approve, Edit, Context-again (loop), or Pass.\n\nDo **not** send to WhatsApp directly during a redraft β€” the approval flow is the sidecar's job.\n\n## Incoming Message Processing\n\n### Delivery Pipeline (current)\n1. Evolution API POSTs each `MESSAGES_UPSERT` webhook directly to Hermes at `http://localhost:8644/webhooks/whatsapp-incoming` (proxied via `jimmy.huper.asia/webhooks/whatsapp-incoming`)\n2. Hermes webhook route (`whatsapp-incoming`, dynamic subscription) validates and spawns an agent\n3. Agent loads `whatsapp-evolution` skill, follows the rules below\n4. Webhook subscription uses `deliver: log` because webhooks do not support `deliver: local`; the agent explicitly calls `send_message(target=\"telegram\", ...)` only for important/actionable/VIP/event/birthday/prayer/error items, OR posts a draft to the sidecar for approval items.\n\n**No more OpenClaw, no Python webhook server, no port 5001, no keepalive cron.** If messages aren't being processed, check: nginx proxy to port 8644, `hermes webhook list` for the subscription, Evolution API webhook URL setting.\n\n## Contact Rules & Escalation\n\nConfig: `memory/whatsapp-vip-contacts.json`\n\n### Priority Levels\n\n| Priority | Action | Example |\n|----------|--------|---------|\n| **VIP (call-always)** | **CALL Vincent immediately via Twilio without asking.** Do NOT ask for approval first. During quiet hours (1am-10am SGT) suppress call and use Telegram notification only, unless life-and-death emergency. Match VIPs by both phone JID (`659...@s.whatsapp.net`) and LID JID (`...@lid`) when present, because recent chats may arrive with `key.remoteJid` as an LID and the phone number only in `key.remoteJidAlt`. | Dn Jeff (+6597630553) |\n| **Family** | Always notify Vincent on Telegram | ps112 group, family members |\n| **Auto-reply** | Reply based on context file, notify Vincent | Repair shops |\n| **Normal** | Analyze intent, reply if appropriate, notify if important | Everyone else |\n| **Broadcast/Spam** | Ignore silently | Forwarded job posts, chain messages |\n\n### Quiet Hours\n- 1am-10am SGT (17:00-02:00 UTC): avoid phone calls, including for Dn Jeff / VIP call-always contacts\n- **Exception:** life-and-death emergencies always call regardless of quiet hours\n- During quiet hours, use Telegram notification only and mention that a call was suppressed due to quiet hours\n\n## Making a Call via Twilio\n\nCredentials: `/opt/data/secrets/.twilio-credentials-new` (`TWILIO_ACCOUNT_SID`, `TWILIO_AUTH_TOKEN`, `TWILIO_PHONE_NUMBER`). From: `+16614809287`, Vincent: `+6590054438`.\n\nPreferred script: `/opt/data/scripts/call_vincent.py`, now using the Twilio Python SDK baked into the container.\n\n```bash\npython3 /opt/data/scripts/call_vincent.py \"Hello Vincent, this is Jimmy. Urgent summary here.\"\n```\n\n## Action Rules\n\n### Default: DO NOT auto-reply directly, and do not spam main Telegram\nCollect intent and stay local unless something truly needs Vincent's immediate attention.\n\n**When you need Vincent to approve a reply, POST a draft to the sidecar (see \"Draft Approvals\" above).** The sidecar handles the Telegram notification with buttons, the WebApp for editing, and sending to WhatsApp on approve. Do **not** duplicate that draft or summary in the main Telegram chat.\n\nMain Telegram WhatsApp summaries are muted by default. Only use `send_message(target=\"telegram\", ...)` for urgent/important/actionable items that need Vincent now: VIP actionable messages/calls, missed/ringing family calls, emergencies, safety/medical/school-pickup issues, money/legal/account/security matters, explicit call-back/decision-now requests, or pipeline errors. Routine logistics, casual family pings, generic group events, links, standalone codes, normal confirmations, and suggested replies should be local-only or sidecar-only.\n\nDo NOT send WhatsApp replies directly unless Vincent explicitly approves them or the category is pre-approved.\n\n### Tone (when proposing or sending replies)\n- Casual, direct, no sign-offs (no \"Thanks & Regards\", no \"- Vincent\")\n- Ask questions instead of giving instructions\n- Keep it natural β€” like texting a friend\n- Christian and grace-based: warm, humble, encouraging, non-judgmental, peace-making\n- Never use em dashes in drafted replies or approval messages. Use commas, periods, colons, or parentheses instead.\n- For prayer/birthday contexts, use gentle faith-filled language; avoid sounding preachy, performative, or overly formal\n\n### What to Do With Each Message\n\n**DMs (personal chats):**\n- Analyze intent\n- Notify Vincent on Telegram (via `send_message(target=\"telegram\", ...)`) only if important/actionable/time-sensitive, requires a decision, comes from a VIP contact\n- If you have a useful suggested reply, POST it as a draft to the sidecar instead of a Telegram notification β€” Vincent gets the buttons inline\n- Casual/social/non-actionable DMs β†’ local log only, no Telegram notification, no draft\n- Do NOT reply unless Vincent says to\n- VIP contacts β†’ **call Vincent immediately without asking**. Do NOT post a draft. Just call. During quiet hours (1am-10am SGT), suppress the call and send Telegram notification only, unless life-and-death emergency.\n\n**Group chats β€” events/meetups:**\n- If someone posts about an event, gathering, meetup, dinner, etc:\n 1. Extract: what, when, where, who\n 2. Notify Vincent on Telegram with details (no draft needed unless you also want to suggest an RSVP reply)\n 3. Ask: \"Want me to add this to Google Calendar and set a reminder?\"\n\n**Group chats β€” prayer requests / praying for others:**\n- If someone asks for prayer, shares a burden, illness, grief, struggle, thanksgiving, or asks the group to pray:\n 1. Fetch recent group context first: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n 2. Identify who/what to pray for and any sensitive details\n 3. Draft a short, grace-based Christian reply/prayer\n 4. POST it to the sidecar with `summary` describing the request and `draft_text` being the prayer\n 5. Do NOT post to WhatsApp unless Vincent approves\n\n**Group chats β€” birthday greetings:**\n- If you see birthday wishes for someone in a group:\n 1. Fetch recent group context first: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n 2. Note who the birthday person is and relationship/context if inferable\n 3. Draft a warm, grace-based Christian birthday greeting\n 4. POST it to the sidecar\n 5. Do NOT post to WhatsApp unless Vincent approves\n\n**Group chats β€” general:**\n- Casual banter, memes, forwards, broadcast spam β†’ ignore silently\n- Religious content shares β†’ ignore unless specifically addressed to Vincent\n\n### When to Notify Vincent (Telegram) vs Post a Draft\n\n**Use `send_message(target=\"telegram\", ...)` sparingly** for plain notifications only when the message needs Vincent's immediate attention: VIP actionable DMs/calls, emergencies, school pickup/child logistics requiring action now, money/legal/account/security matters, explicit call-back/decision-now requests, or pipeline errors.\n\n**Use the sidecar (POST `/drafts`)** whenever you have a suggested reply for Vincent to approve: prayer responses, birthday greetings, DM replies, group RSVPs. Do not also send the same summary to the main Telegram chat.\n\nDo **not** notify or draft for casual/social/non-actionable DMs, routine logistics, read receipts/status updates, religious shares not addressed to Vincent, memes, forwarded/broadcast/spam, generic group events, standalone codes, normal confirmations, or general group chatter. Family messages are not automatically main-chat alerts unless urgent, safety-related, school/pickup critical, or explicitly requiring Vincent now.\n\n## Intent Categories\n\nWhen analyzing incoming messages, classify as:\n\n1. **Urgent** β€” time-sensitive, emergencies β†’ call Vincent outside quiet hours; during quiet hours use Telegram only unless life-and-death\n2. **Important** β€” decisions, money, appointments, anything consequential β†’ call Vincent outside quiet hours; during quiet hours notify on Telegram only unless life-threatening\n3. **Request** β€” asking Vincent to do something β†’ **notify Vincent** (Telegram) or post a draft if you have a safe reply\n4. **Enquiry** β€” asking for info, pricing, availability β†’ auto-reply if context available, otherwise post a draft\n5. **Confirmation** β€” confirming plans, bookings β†’ acknowledge and notify Vincent\n6. **Casual/Social** β€” greetings, memes, forwards β†’ usually ignore or react\n7. **Broadcast/Spam** β€” forwarded job posts, chain messages β†’ ignore\n\n## Finding Individual Contacts and JIDs\n\nWhen Vincent asks to monitor or prioritize a specific WhatsApp contact and only gives a name:\n\n1. Query contacts and chats, then search both display names and serialized message metadata:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n python3 - <<'PY'\n import os, requests, json\n base=os.environ['EVOLUTION_API_URL']; inst=os.environ['EVOLUTION_INSTANCE']; key=os.environ['EVOLUTION_API_KEY']\n headers={'apikey':key,'Content-Type':'application/json'}\n terms=['whye','keong'] # replace with lowercase search terms\n for ep in ['findContacts','findChats']:\n r=requests.post(f'{base}/chat/{ep}/{inst}', headers=headers, json={}, timeout=30)\n print('endpoint', ep, 'status', r.status_code)\n for obj in (r.json() if r.ok else []):\n if any(t in json.dumps(obj, ensure_ascii=False).lower() for t in terms):\n print(json.dumps(obj, ensure_ascii=False)[:1000])\n PY\n ```\n2. If `findChats` shows `remoteJid` ending in `@lid`, inspect `lastMessage.key.remoteJidAlt` for the phone JID. Store both in `memory/whatsapp-vip-contacts.json` when configuring VIP/call rules.\n3. Verify the number exists with `/chat/whatsappNumbers/{instance}` before finalizing.\n4. Validate the JSON config after editing: `python3 -m json.tool /opt/data/agent-state/memory/whatsapp-vip-contacts.json >/tmp/wa_vip_validated.json`.\n\n## Finding Group JIDs\n\nThe `group/fetchGroups` endpoint returns 404 on some Evolution API instances (including ours). To find a group JID when you don't have it:\n\n1. **Use `chat/findMessages` with an empty `where` filter** to pull recent messages across all chats, then extract unique `remoteJid` values containing `@g.us`.\n2. **Match by content**: Fetch messages from each group JID using `where.key.remoteJid` and match sender names or message text to identify the target group.\n3. **Known group JIDs** (store in `whatsapp-vip-contacts.json` as you discover them):\n - ps112 family group: `6591887488-1464061170@g.us`\n - Overcomers group: `6591444911-1614821287@g.us`\n - η₯žηš„εŽηΎŽ group: `6596756578-1405873840@g.us` (content-sharing, casual/broadcast, local log only)\n - Overcomers/TCians group: `120363230236878544@g.us` (church fellowship, PJ sends weekly attendance reports, routine attendance = local log only)\n\nExample search:\n```bash\ncurl -s -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"remoteJid\": \"6591444911-1614821287@g.us\"}}, \"limit\": 10, \"order\": \"desc\"}'\n```\n\n## Processing Image/Media Messages\n\nWhen an incoming WhatsApp message is an image (imageMessage), video, or other media with no text or only a caption:\n\n1. **Fetch conversation context** as usual with `wa_recent_messages.py`\n2. **Download the media** using the Evolution API:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"message\": {\"key\": {\"id\": \"MESSAGE_ID\", \"fromMe\": false, \"remoteJid\": \"GROUP_OR_CHAT_JID\"}}, \"convertToMp4\": false}' \\\n -o /tmp/wa_media_response.json\n ```\n Then decode: `python3 -c \"import json,base64; d=json.load(open('/tmp/wa_media_response.json')); open('/tmp/wa_media.jpg','wb').write(base64.b64decode(d['base64']))\"`\n3. **Analyze the image** using `vision_analyze` tool with `/tmp/wa_media.jpg` path and a context-aware question\n4. **Fallback if vision is unavailable**: Use `tesseract /tmp/wa_media.jpg stdout` for OCR, combined with pixel analysis (PIL/numpy) and conversation context to infer what the image contains\n5. **Album messages** (`albumMessage`): These bundle multiple images. Each image arrives as a separate `imageMessage` with a `messageAssociation.parentMessageKey` linking them to the album. Download each image separately. The `expectedImageCount` field tells how many images to expect\n6. **messageAssociation**: When images have `parentMessageKey` with `fromMe: true`, they are replying to a message Vincent sent. Use this for context\n\n### Audio / Voice Note Messages\nWhen an incoming WhatsApp message is `audioMessage` or a voice note (`ptt: true`), treat it like other media, but transcribe before intent classification when possible:\n\n1. **Fetch conversation context first**, especially in groups:\n ```bash\n python3 /opt/data/scripts/wa_recent_messages.py \"\" 50\n ```\n2. **Download the audio** with Evolution API, saving the response to a file:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n cat > /tmp/wa_audio_req.json <<'JSON'\n {\"message\":{\"key\":{\"id\":\"MESSAGE_ID\",\"fromMe\":false,\"remoteJid\":\"GROUP_OR_CHAT_JID\"}},\"convertToMp4\":false}\n JSON\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_audio_req.json -o /tmp/wa_audio_response.json\n python3 - <<'PY'\n import json, base64\n d=json.load(open('/tmp/wa_audio_response.json'))\n b=d.get('base64') or d.get('data',{}).get('base64')\n if ',' in b[:100]: b=b.split(',',1)[1]\n open('/tmp/wa_audio.ogg','wb').write(base64.b64decode(b))\n PY\n ```\n3. **Convert OGG/Opus to WAV**:\n ```bash\n ffmpeg -y -i /tmp/wa_audio.ogg -ar 16000 -ac 1 /tmp/wa_audio.wav\n ```\n4. **Transcribe fallback when Whisper is unavailable**: the container may not have `whisper`, `faster-whisper`, `torch`, or `speech_recognition` installed globally. A reliable lightweight fallback is a temp venv with SpeechRecognition and Google web speech:\n ```bash\n python3 -m venv /tmp/srvenv\n /tmp/srvenv/bin/pip install --quiet SpeechRecognition\n /tmp/srvenv/bin/python - <<'PY'\n import speech_recognition as sr\n r=sr.Recognizer()\n with sr.AudioFile('/tmp/wa_audio.wav') as source:\n audio=r.record(source)\n print(r.recognize_google(audio, language='en-SG'))\n PY\n ```\n5. **Check adjacent voice notes from the same sender**. WhatsApp voice notes are often sent in quick sequential chunks. Download and transcribe nearby `audioMessage` records from the same sender before deciding whether to notify Vincent.\n6. Use the transcript plus recent context to classify the message. For group meetup planning, extract what/when/where/who and notify Vincent if actionable, including transcript confidence.\n\n### Video Messages\n\nWhen an incoming WhatsApp message is `videoMessage` (with or without a caption):\n\n1. **Fetch conversation context** first, as with any media message: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n2. **Download the video** using Evolution API (same pattern as images/audio β€” save response to file, then decode):\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n cat > /tmp/wa_video_req.json <<'JSON'\n {\"message\":{\"key\":{\"id\":\"MESSAGE_ID\",\"fromMe\":false,\"remoteJid\":\"CONTACT_JID\"}},\"convertToMp4\":false}\n JSON\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_video_req.json -o /tmp/wa_video_response.json\n python3 - <<'PY'\n import json, base64\n d = json.load(open('/tmp/wa_video_response.json'))\n b = d.get('base64') or d.get('data', {}).get('base64', '')\n if ',' in b[:100]: b = b.split(',', 1)[1]\n open('/tmp/wa_video.mp4', 'wb').write(base64.b64decode(b))\n print(f\"Video saved: {len(base64.b64decode(b))} bytes\")\n PY\n ```\n3. **Extract representative frames** with ffmpeg (do NOT try to analyze video directly β€” vision tools work on images):\n ```bash\n # Frame at start\n ffmpeg -y -ss 0 -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame0.jpg\n # Frame at ~1/3 of the way (good for short videos)\n ffmpeg -y -ss 3 -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame3s.jpg\n # If video is longer (>15s), also extract at midpoint\n # ffprobe -v quiet -print_format json -show_format /tmp/wa_video.mp4 # get duration\n # ffmpeg -y -ss $HALF_DURATION -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame_mid.jpg\n ```\n4. **Analyze the frames** with `vision_analyze`. Ask specific questions about content: what objects/people/text are visible, what type of video this appears to be (screen recording, personal clip, forwarded meme, etc.), and any text or identifying details.\n5. **Use the caption** from `videoMessage.caption` if present (can be empty/null). Combine frame analysis + caption + conversation context to classify the message intent.\n6. **Transcription of audio in video** is generally not feasible with available tools. If the video likely contains important spoken content (e.g. voice memo style), note this limitation and flag it to Vincent.\n\n**Key points:**\n- Always use `-o` flag and file-based decode; videos can be 1-10MB and base64 responses will be even larger\n- `--max-time 25` on curl is essential for video downloads\n- Short casual clips (showing a key fob, a parking spot, quick view of surroundings) are usually casual/social β€” classify accordingly and don't over-escalate\n- If `vision_analyze` fails (400 on models that don't support images, or 429 rate limit), fall back to conversation context inference alone\n\n### Poll Messages\nWhen `pollUpdateMessage` appears in chat history (someone voted), the message contains a `pollCreationMessageKey` with the original poll's message `id`. To get the poll question and options, query:\n```bash\ncurl -s -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"id\": \"POLL_MESSAGE_ID\", \"remoteJid\": \"GROUP_JID\"}}, \"limit\": 1}'\n```\nThe response will have `pollCreationMessageV3` with `name` (the poll question) and `options` array. Always look up the poll when votes or poll references appear, since the poll text contains the event details.\n\n### `associatedChildMessage` (Motion Photos / Companion Media)\nWhatsApp motion photos and companion media arrive as `associatedChildMessage` type. These are short videos or supplementary images attached to a parent message via `messageAssociation.parentMessageKey`. The structure is:\n\n```\nmessage: {\n messageContextInfo: {\n messageAssociation: {\n associationType: 12,\n parentMessageKey: { id, fromMe, remoteJid }\n }\n },\n associatedChildMessage: {\n message: {\n videoMessage: { ... } // or imageMessage: { ... }\n }\n }\n}\n```\n\nKey points:\n- The `parentMessageKey.id` links back to the main message (the image/video they accompany)\n- When `parentMessageKey.fromMe: true`, the companion media is replying to a message Vincent sent\n- Captions may appear on the child's `videoMessage.caption` or `imageMessage.caption` inside the nested message, NOT at the top level\n- These often contain motion-photo short videos (3 seconds) or album companion images\n- Treat them as media attachments to the parent; don't process them as standalone messages requiring independent intent analysis\n\n### Sequential Image Sets (Non-Album)\nMultiple images sent rapidly by the same sender at similar timestamps may NOT be wrapped in an `albumMessage`. They appear as separate `imageMessage` entries. Always check for adjacent images from the same sender and download+OCR all of them together, as they often form a coherent multi-page set (e.g., route info, comparison tables, maps). Missing one can lose important context.\n\n### Key pitfalls\n- The Evolution API `getBase64FromMediaMessage` endpoint returns the full base64 payload (can be 100KB+ for images). Use `-o` flag to save to file, then parse with Python. Do NOT try to pipe it directly\n- `curl` with `--max-time 25` is essential; without it, large media responses can hang\n- The `albumMessage` type itself has no `fromMe` field; check each individual `imageMessage` instead\n- Image thumbnails in the webhook payload are truncated; always download the full image for analysis\n- **WhatsApp media URLs expire**: CDN download links (`mmg.whatsapp.net`) expire after roughly 24-48 hours. When `getBase64FromMediaMessage` returns a 400/404 error like \"Failed to fetch stream from https://mmg.whatsapp.net/...\", the media is no longer retrievable via the API. This means you CANNOT download adjacent/older voice notes, images, or documents for context if they're more than a day or two old. Fall back to conversation text context only. Fresh/incoming messages are almost always still downloadable.\n- **Voice note transcription reliability**: Short voice notes (<15s), especially in Singlish/Chinese-English mix common in Singapore contacts, often produce unreliable transcriptions via SpeechRecognition+Google. Always flag low-confidence transcriptions to Vincent rather than acting on potentially garbled text.\n- **Vision API failures**: `vision_analyze` can fail with 429 (rate limit) **or 400** (model doesn't support image input, e.g. glm-5.1 via ollama-cloud). Always have a fallback plan. When vision is unavailable, try tesseract OCR, then pixel analysis (PIL/numpy for brightness zones, text region detection via row-level dark-pixel density)\n- **OCR can fail on decorative/complex images**: Obituaries, memorial notices, religious flyers with ornate fonts, or dark-background images often defeat tesseract even with aggressive preprocessing (upscaling, thresholding, inverting, PSM mode variations). When OCR returns gibberish or empty, fall back to conversation context inference: use preceding/surrounding messages to determine what the image likely contains, then draft an appropriate response based on that context\n- **OCR on devotional/dark-background images**: These often defeat tesseract on full-image scans. Use a multi-strategy approach: (1) run tesseract on the full image first, (2) if text is sparse, crop into regions (top 15%, middle, bottom 15%), upscale 2-3x, enhance contrast (2-3x), and try both normal and inverted (`ImageOps.invert`) versions with `--psm 6`, (3) combine all partial OCR results. For Exodus/verse images, the title (e.g. \"WHEN MOSES' HANDS WERE\") is often in decorative font that OCR misses, while the verse body text is usually readable.\n- **Prioritize conversation context**: When processing group images, always fetch recent context FIRST (before attempting vision/OCR). If image analysis fails, the conversation history often makes the image's purpose clear enough to draft a meaningful reply\n- **DM conversation context is one-sided (fromMe only)**: The Evolution API `findMessages` endpoint often returns only Vincent's sent messages (`fromMe: true`) for DM chats, not the contact's incoming messages. This means `wa_recent_messages.py` can show only one side of the conversation. When processing a DM webhook, do NOT rely solely on fetched history for the contact's side. The webhook payload already contains the incoming message text. Use what the webhook gives you plus Vincent's side from the API. If you need the contact's recent messages, check if `fromMe: false` records exist; if none are returned, the API simply doesn't have them and you must work with what you have (webhook message + Vincent's replies for context).\n- **`findMessages` ignores `key.fromMe` filter**: Passing `\"key.fromMe\": false` in the `where` clause still returns all messages (including `fromMe: true`). To find messages from a specific sender, fetch results and filter client-side by checking `fromMe` in Python.\n- **`findMessages` caps at ~50 records per request**: Even with `limit: 200` or `limit: 500`, the API returns at most 50 records. When one party sends many consecutive messages, the other party's messages may not appear in the first 50 results. Use pagination or fetch multiple pages if you need older messages.\n- **Video frame extraction**: Use `ffmpeg -y -ss N -i video.mp4 -frames:v 1 -update 1 output.jpg` (the `-update 1` flag is needed; without it ffmpeg treats the output as an image sequence and fails). Extract frames at multiple timestamps (start, ~1/3, midpoint for longer videos) since a single frame may miss the key content.\n- **ffmpeg \"Invalid color space\" failure on WhatsApp videos**: Some WhatsApp H.264 videos have `color_space=reserved` and `color_primaries=reserved` metadata that ffmpeg cannot decode at all. Every codec/format/filter combination fails with \"Invalid color space\" / return code 234. This is NOT fixable with `-vf setparams`, `-color_trc`, `-colorspace`, `-color_primaries`, or `-vf format=` flags because the error occurs at the decoder level before any filter runs. **Do not waste time trying ffmpeg workarounds.** When this happens, fall back to: (1) conversation context inference (who sent it, what thread it's in, adjacent messages), (2) the jpegThumbnail embedded in the videoMessage (small but gives a rough idea; base64-encoded JPEG, typically 600-900 bytes, use `vision_analyze` on it if available), or (3) if all analysis fails, just classify based on surrounding text context.\n- **`wa_recent_messages.py` JSON output breaks `json.loads()`**: The script's stdout contains WhatsApp message content with control characters (emoji, CJK, deviceListMetadata binary blobs) that cause `json.loads()` to fail with `Invalid control character` or `Expecting ',' delimiter` errors. Even `strict=False` does not reliably fix it. **Reliable approach**: use `curl -o /tmp/wa_msgs.json` to save the Evolution API response directly to a file (the file write handles binary safely), then parse with `json.load(open('/tmp/wa_msgs.json'))` in Python. Example:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n curl -s --max-time 15 -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"remoteJid\": \"CONTACT_JID\"}}, \"limit\": 20}' -o /tmp/wa_msgs.json\n ```\n Then in Python: `data = json.load(open('/tmp/wa_msgs.json')); records = data['messages']['records']`\n\n## API Reference\n\nFor full API endpoints (media, contacts, location, buttons) β€” see [references/api.md](references/api.md).\n\n[Skill directory: /opt/data/skills/personal/whatsapp-evolution]\nResolve any relative paths in this skill (e.g. `scripts/foo.js`, `templates/config.yaml`) against that directory, then run them with the terminal tool using the absolute path.\n\n[This skill has supporting files:]\n- references/api.md -> /opt/data/skills/personal/whatsapp-evolution/references/api.md\n- scripts/process_queue.py -> /opt/data/skills/personal/whatsapp-evolution/scripts/process_queue.py\n- scripts/send_text.sh -> /opt/data/skills/personal/whatsapp-evolution/scripts/send_text.sh\n\nLoad any of these with skill_view(name=\"personal/whatsapp-evolution\", file_path=\"\"), or run scripts directly by absolute path (e.g. `node /opt/data/skills/personal/whatsapp-evolution/scripts/foo.js`).\n\nThe user has provided the following instruction alongside the skill invocation: Incoming WhatsApp webhook received.\n\nEvent: messages.upsert\nInstance: n8n-WA\nSender: Shalom Evonne\nSender JID: 120363418526182134@g.us\nFrom me: False\nMessage type: conversation\nMessage: {\n \"event\": \"messages.upsert\",\n \"instance\": \"n8n-WA\",\n \"data\": {\n \"key\": {\n \"remoteJid\": \"120363418526182134@g.us\",\n \"fromMe\": false,\n \"id\": \"3EB028D1937CA9887912D8\",\n \"participant\": \"131503308673050@lid\",\n \"participantAlt\": \"6591887488@s.whatsapp.net\",\n \"addressingMode\": \"lid\"\n },\n \"pushName\": \"Shalom Evonne\",\n \"status\": \"DELIVERY_ACK\",\n \"message\": {\n \"conversation\": \"Good evening Shalom Family!\\n\\nTomorrow is Thursday, Week Odd.\\n\\nNaomi schedule:\\n8:00-8:25 AM - Morning Assembly\\n8:25-9:00 AM - CCE and Assembly\\n9:00-9:35 AM - D&T\\n9:35-10:10 AM - Recess\\n10:10-10:45 AM - D&T\\n10:45-11:20 AM - PE (Mrs Ku)\\n11:20-11:55 AM - Literature\\n11:55 AM-12:30 PM - Lunch\\n12:30-1:05 PM - G2/G3 Class\\n1:05-1:40 PM - MTL\\n1:40-2:15 PM - MTL\\n\\nDismissal: 2:15 PM\\n\\nPE Reminder: Naomi has PE tomorrow. Please pack PE gear tonight!\\n\\nWishing you a wonderful evening and a great day tomorrow! God bless!\"\n },\n \"contextInfo\": null,\n \"messageType\": \"conversation\",\n \"messageTimestamp\": 1780484908,\n \"instanceId\": \"b2bfaa30-d5f1-4b22-b979-791892619414\",\n \"source\": \"web\"\n },\n \"destination\": \"https://jimmy.huper.asia/webhooks/whatsapp-incoming\",\n \"date_time\": \"2026-06-03T08:08:28.410Z\",\n \"sender\": \"6590054438@s.whatsapp.net\",\n \"server_url\": \"https://evolution.huper.asia\",\n \"apikey\": \"5E4C532223A0-460B-A749-16C584F56E74\"\n}\n\nPrimary goal: REDUCE NOISE. Stay silent in the main Telegram chat unless the message truly needs Vincent's immediate attention. Approval-worthy reply drafts should go to the jimmy-wa-drafts sidecar, NOT to the main Telegram chat.\n\nFollow the whatsapp-evolution skill and /opt/data/agent-state/memory/whatsapp-vip-contacts.json, with these stricter notification rules:\n\nABSOLUTE SILENCE RULES:\n- If fromMe is true: ignore. Do NOT call send_message. Final response local log only.\n- Ignore status/update/read-receipt events or payloads with no real message content.\n- Do NOT send main Telegram summaries for ordinary DMs, routine logistics, casual family messages, generic group events, group chatter, forwards, memes, devotional/religious shares, links, standalone codes, normal confirmations, or suggested replies.\n- Do NOT ask \"Want me to add this to Calendar?\" for WhatsApp group events unless Vincent is explicitly invited, named, responsible, or the event is time-sensitive and clearly relevant to him today.\n- Do NOT surface family messages just because they are family. Only surface if urgent, safety-related, school/pickup critical, money/legal/medical, or explicitly requires Vincent now.\n\nSIDEcar RULES:\n- If a reply may be useful but it is not urgent, POST it to https://jimmy-draft.huper.asia/drafts using the whatsapp-evolution Draft Approvals flow. Do NOT duplicate that draft/summary in the main Telegram chat.\n- Prayer and birthday replies should go to the sidecar when appropriate, not the main Telegram chat, unless the underlying situation is urgent/sensitive and Vincent needs immediate awareness.\n- Before drafting any reply or approval suggestion, fetch recent context: python3 /opt/data/scripts/wa_recent_messages.py \"120363418526182134@g.us\" 30. For active/ambiguous groups use 50.\n\nMAIN TELEGRAM ALERTS ARE ALLOWED ONLY FOR:\n1. VIP contacts marked always_call or always_notify when the message is important/actionable, or any VIP call event.\n2. Calls from family/VIP that appear missed/ringing and likely need immediate attention. Avoid repeated alerts for accepted/terminated duplicate call state updates.\n3. Emergencies, safety issues, medical issues, school pickup/child logistics requiring action now, legal/money/account/security matters, or time-sensitive commitments in the next few hours.\n4. Messages explicitly asking Vincent to call back, decide now, approve payment, provide OTP/code for an active task, or respond urgently.\n5. Pipeline errors or webhook processing failures.\n\nCALL RULES:\n- VIP contacts (e.g. Dn Jeff, Ng Whye Keong): call Vincent immediately outside quiet hours when actionably important or on ringing/missed call. During 1am-10am SGT, Telegram only unless life-and-death emergency.\n- Do not call for casual VIP confirmations or non-actionable courtesy messages.\n\nTONE/RULES FOR DRAFTS:\n- Proposed replies must be Christian and grace-based where appropriate: warm, humble, encouraging, non-judgmental, peace-making, not preachy.\n- Never use em dashes in drafted replies or approval messages. Use commas, periods, colons, or parentheses instead.\n- NEVER auto-reply directly on WhatsApp unless Vincent explicitly approves or future config marks that exact category pre-approved.\n\nDelivery rule:\n- For allowed main alerts only: call send_message(target=\"telegram\", message=...). Keep it concise.\n- For non-urgent reply suggestions: use the sidecar only.\n- For everything else: DO NOT call send_message. Keep final response as local-only log.\n", "timestamp": 1780484924.672017, "anchor": true}, {"id": 99975, "role": "assistant", "content": "This is a routine school schedule announcement for Naomi (timetable + PE reminder) sent to a group chat. It's not urgent, not time-critical requiring Vincent's action now, and falls under routine logistics/broadcast content.\n\n**Classification**: Routine/schedule broadcast β†’ local log only.\n\nNo Telegram notification, no sidecar draft needed. Staying silent.", "timestamp": 1780484924.8081207}], "bookend_end": [], "messages_before": 0, "messages_after": 2}, {"session_id": "20260524_110046_052cebc7", "when": "May 24, 2026 at 11:00 AM", "source": "webhook", "model": "gpt-5.5", "title": null, "matched_role": "user", "match_message_id": 81566, "snippet": "...draft_text\": \"Thanks >>>Karen<<<, will miss you both β€” praying for a good time too πŸ™\",\n \"extras\": {\n \"group_jid\": \"120363351787895353@g.us\",\n \"sender_name\": \">>>Karen<<< >>>Tan<<<\",\n \"sender_jid\": \">>>6597630553<<<@s.whatsapp.net\",\n \"category\": \">>>prayer<<>>blessing<<<\",\n \"original_message\": \"Happy >>>cycling<<<, to those who...", "bookend_start": [], "messages": [{"id": 81566, "role": "user", "content": "[IMPORTANT: The user has invoked the \"whatsapp-evolution\" skill, indicating they want you to follow its instructions. The full skill content is loaded below.]\n\n---\nname: whatsapp-evolution\ndescription: WhatsApp messaging via Evolution API β€” send, receive, analyze, and auto-reply. Use when processing an incoming WhatsApp message delivered via the webhook, replying on Vincent's behalf, liaising with contacts, sending messages, or handling WhatsApp-related tasks. Covers intent analysis, auto-reply rules, escalation (VIP β†’ phone call), conversation context fetching, and draft approvals via the jimmy-wa-drafts sidecar.\n---\n\n# WhatsApp via Evolution API\n\n## Credentials\nStored in `/opt/data/secrets/.whatsapp-evolution-credentials`:\n- `EVOLUTION_API_URL`, `EVOLUTION_API_KEY`, `EVOLUTION_INSTANCE`\n\nLoad with: `export $(cat .whatsapp-evolution-credentials | xargs)`\n\n## Sending Messages (direct, no approval)\n\n```bash\nexport $(cat .whatsapp-evolution-credentials | xargs)\ncurl -s -X POST \"${EVOLUTION_API_URL}/message/sendText/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"number\": \"6591234567\", \"text\": \"Hello!\"}'\n```\n\n- Phone format: country code, no `+` (e.g. `6590054438`)\n- Groups: use JID `{groupid}@g.us`\n- For media/location/contacts β€” see [references/api.md](references/api.md)\n\nUse direct send only when Vincent has explicitly commanded/approved the exact send, or for pre-approved auto-replies/follow-ups. For any reply that needs Vincent's review, use the sidecar (next section).\n\n### Sending trip/camp details from retrieved emails\n\nWhen Vincent explicitly asks to send extracted school trip/camp details to a child or family member, send directly. Keep it practical and grounded:\n- Include dates, reporting/pick-up times, flights, venue/itinerary highlights, deadlines, forms, and safety/medication notes when found.\n- If an official packing list is missing, say that clearly. Do not invent one. You may add a short β€œbased on itinerary/weather, prepare…” list, clearly labeled as practical guidance rather than official school instructions.\n- Avoid long explanations about the search process in WhatsApp. Send the useful details.\n\n### Sending a local image/file as media\n\nFor user-approved media sends, base64-encode the local file and call `sendMedia`:\n\n```bash\nexport $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\npython3 - <<'PY'\nimport base64, json\npath = '/tmp/file.png'\npayload = {\n 'number': '6591234567',\n 'mediatype': 'image',\n 'mimetype': 'image/png',\n 'caption': 'Caption here',\n 'media': base64.b64encode(open(path, 'rb').read()).decode(),\n 'fileName': 'file.png',\n}\nopen('/tmp/wa_media_payload.json', 'w').write(json.dumps(payload))\nPY\ncurl -s --max-time 60 -X POST \"${EVOLUTION_API_URL}/message/sendMedia/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_media_payload.json\n```\n\nWhen sending QR tickets or passes, verify the crop/image contains the full QR code and the intended attendee name before sending; the QR screenshot is effectively the redeemable ticket.\n\n## Fetching Conversation Context\n\nBefore drafting or proposing any reply, always fetch recent chat history for context:\n\n```bash\npython3 /opt/data/scripts/wa_recent_messages.py \"6591234567@s.whatsapp.net\" 30\n```\n\nUse `limit=30` by default. For active group chats or ambiguous situations, fetch up to 50 messages. Read enough context to understand who is being discussed, current tone, previous commitments, and whether Vincent has already replied. The script returns newest messages first (descending order).\n\n## Draft Approvals β€” `jimmy-wa-drafts` sidecar\n\nWhen you need Vincent to approve a WhatsApp reply, POST the draft to the sidecar. It owns the entire approval flow: Telegram message with 4 inline buttons, Telegram WebApp for Edit/Context, sending to WhatsApp via Evolution API on Approve.\n\n### Posting a draft\n\nAlways include an `extras` object with the original WA context. The sidecar passes `extras` through verbatim if Vincent later clicks \"Context\" β€” without it, your redraft will be flying blind.\n\n```bash\ncat > /tmp/wa_draft.json <<'JSON'\n{\n \"contact\": \"120363351787895353@g.us\",\n \"contact_name\": \"Cycling group\",\n \"summary\": \"Karen sent a blessing over the bike ride; she's not joining\",\n \"draft_text\": \"Thanks Karen, will miss you both β€” praying for a good time too πŸ™\",\n \"extras\": {\n \"group_jid\": \"120363351787895353@g.us\",\n \"sender_name\": \"Karen Tan\",\n \"sender_jid\": \"6597630553@s.whatsapp.net\",\n \"category\": \"prayer/blessing\",\n \"original_message\": \"Happy cycling, to those who can make it… Dn Jeff and I had made plans for this PM so we won't be able to join y'all. Praying the Lord's favor, protection and good-weather blessings be upon y'all and your route!\",\n \"extracted\": {\n \"what\": \"Cycling from East Coast Park to Changi Village, then dinner\",\n \"when\": \"Today, 4:00 PM\",\n \"where\": \"East Coast Park to Changi Village\",\n \"who\": \"Group members who voted yes for cycling/dinner or dinner\"\n },\n \"confidence\": \"high\",\n \"why_appropriate\": \"Multiple group messages are blessings/wishes; Vincent has not replied; tone matches grace-based norm.\",\n \"wa_message_id\": \"ABC123XYZ\"\n }\n}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_draft.json\n```\n\nResponse: `{\"draft_id\":\"abc123def\",\"telegram_message_id\":42}`\n\nWhat to put in `extras` (use whatever makes sense for the message type):\n\n- **Always**: `original_message` (verbatim WA text), `sender_name`, `category` (e.g. `prayer`, `birthday`, `event`, `dm-reply`, `enquiry`)\n- **Group messages**: `group_jid`, `sender_jid`\n- **DMs**: `sender_jid`\n- **Strongly recommended (drives the rich Telegram render)**: `extracted` (dict of who/what/when/where), `confidence` (`high`/`medium`/`low`), `why_appropriate` (one-line reasoning shown to Vincent so he can decide)\n- **Optional**: `wa_message_id`, `tone_hint` (e.g. `warm-grace-based`), `sender_phone` (auto-derived from `sender_jid` if omitted)\n\nVincent receives a Telegram message in the `@jimmy_WA_bot` chat with 4 inline buttons. The sidecar handles every interaction; **you do not track replies, callbacks, or message IDs**. Just POST and move on.\n\n| Button | Behavior |\n|--------|----------|\n| **Approve** | Sidecar sends draft to WhatsApp |\n| **Edit** | Opens a Telegram WebApp with draft pre-filled; Vincent edits + sends; sidecar pushes to WhatsApp |\n| **Context** | Opens a Telegram WebApp with empty textarea + the original WA message visible; Vincent provides context. (When `HERMES_REDRAFT_WEBHOOK` is wired in the sidecar's env, this will POST back to Jimmy for a redraft. Until then, the context is just stored on the draft.) |\n| **Pass** | Sidecar deletes the draft |\n\n### JSON escaping when `draft_text` contains quotes\n\nSingle quotes (`It's`, `God's`) break shell interpolation. Write the JSON to a temp file first:\n\n```bash\ncat > /tmp/wa_draft.json <<'JSON'\n{\"contact\":\"6591234567@s.whatsapp.net\",\"contact_name\":\"John\",\"summary\":\"…\",\"draft_text\":\"It's nice to hear from you!\"}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_draft.json\n```\n\n### Networking\n\n`https://jimmy-draft.huper.asia` is the public HTTPS endpoint of the sidecar (TLS-terminated by Traefik on ubuntu-server). HTTPS is required because Hermes blocks plain-HTTP commands via tirith. The internal Docker DNS hostname http://jimmy-wa-v64mqe:8080 also works for non-shell calls but the agent is forbidden from using it.\n\n### Sidecar status & logs\n\n- Source: (auto-deploys on push)\n- Public URL: (`/health` for ping)\n- Container: `jimmy-wa-v64mqe.1.` on ubuntu-server\n- Bot: `@jimmy_WA_bot`\n\n## Handling a Redraft Trigger (`wa-redraft` webhook)\n\nWhen Vincent clicks the **Context** button on a draft, the sidecar:\n1. Edits the original Telegram message to \"πŸ”„ Adding context for X… agent is redrafting.\" (removes the old buttons so Vincent can't approve the now-stale text)\n2. POSTs to your `wa-redraft` webhook (`http://localhost:8644/webhooks/wa-redraft`) with the original draft fields plus Vincent's `context` instructions and the `extras` you stored when you first created the draft.\n\nWhen you receive a `wa-redraft` activation, the prompt template will lay out the fields. Your job:\n\n1. Read Vincent's `context` carefully β€” that's the steering signal. The `previous_draft` is just history, not necessarily a starting point.\n2. Re-fetch recent WA history if the original context isn't enough (`python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`). Use `extras.group_jid` for groups or `contact` for DMs.\n3. Write a NEW draft that addresses Vincent's instructions while staying faithful to the original WA context (in `extras.original_message`, `extras.extracted`).\n4. POST the new draft back to the sidecar **with the same `draft_id`** so the existing draft is updated, not duplicated:\n\n```bash\ncat > /tmp/wa_redraft.json <\",\n \"contact\": \"\",\n \"contact_name\": \"\",\n \"summary\": \"\",\n \"draft_text\": \"\",\n \"extras\": \n}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_redraft.json\n```\n\nThe sidecar will send a NEW Telegram message with 4 fresh buttons. Vincent can Approve, Edit, Context-again (loop), or Pass.\n\nDo **not** send to WhatsApp directly during a redraft β€” the approval flow is the sidecar's job.\n\n## Incoming Message Processing\n\n### Delivery Pipeline (current)\n1. Evolution API POSTs each `MESSAGES_UPSERT` webhook directly to Hermes at `http://localhost:8644/webhooks/whatsapp-incoming` (proxied via `jimmy.huper.asia/webhooks/whatsapp-incoming`)\n2. Hermes webhook route (`whatsapp-incoming`, dynamic subscription) validates and spawns an agent\n3. Agent loads `whatsapp-evolution` skill, follows the rules below\n4. Webhook subscription uses `deliver: log` because webhooks do not support `deliver: local`; the agent explicitly calls `send_message(target=\"telegram\", ...)` only for important/actionable/VIP/event/birthday/prayer/error items, OR posts a draft to the sidecar for approval items.\n\n**No more OpenClaw, no Python webhook server, no port 5001, no keepalive cron.** If messages aren't being processed, check: nginx proxy to port 8644, `hermes webhook list` for the subscription, Evolution API webhook URL setting.\n\n## Contact Rules & Escalation\n\nConfig: `memory/whatsapp-vip-contacts.json`\n\n### Priority Levels\n\n| Priority | Action | Example |\n|----------|--------|---------|\n| **VIP (call-always)** | **CALL Vincent immediately via Twilio without asking.** Do NOT ask for approval first. During quiet hours (1am-10am SGT) suppress call and use Telegram notification only, unless life-and-death emergency. Match VIPs by both phone JID (`659...@s.whatsapp.net`) and LID JID (`...@lid`) when present, because recent chats may arrive with `key.remoteJid` as an LID and the phone number only in `key.remoteJidAlt`. | Dn Jeff (+6597630553) |\n| **Family** | Always notify Vincent on Telegram | ps112 group, family members |\n| **Auto-reply** | Reply based on context file, notify Vincent | Repair shops |\n| **Normal** | Analyze intent, reply if appropriate, notify if important | Everyone else |\n| **Broadcast/Spam** | Ignore silently | Forwarded job posts, chain messages |\n\n### Quiet Hours\n- 1am-10am SGT (17:00-02:00 UTC): avoid phone calls, including for Dn Jeff / VIP call-always contacts\n- **Exception:** life-and-death emergencies always call regardless of quiet hours\n- During quiet hours, use Telegram notification only and mention that a call was suppressed due to quiet hours\n\n## Making a Call via Twilio\n\nCredentials: `/opt/data/secrets/.twilio-credentials-new` (`TWILIO_ACCOUNT_SID`, `TWILIO_AUTH_TOKEN`, `TWILIO_PHONE_NUMBER`). From: `+16614809287`, Vincent: `+6590054438`.\n\nPreferred script: `/opt/data/scripts/call_vincent.py`, now using the Twilio Python SDK baked into the container.\n\n```bash\npython3 /opt/data/scripts/call_vincent.py \"Hello Vincent, this is Jimmy. Urgent summary here.\"\n```\n\n## Action Rules\n\n### Default: DO NOT auto-reply directly, and do not spam main Telegram\nCollect intent and stay local unless something truly needs Vincent's immediate attention.\n\n**When you need Vincent to approve a reply, POST a draft to the sidecar (see \"Draft Approvals\" above).** The sidecar handles the Telegram notification with buttons, the WebApp for editing, and sending to WhatsApp on approve. Do **not** duplicate that draft or summary in the main Telegram chat.\n\nMain Telegram WhatsApp summaries are muted by default. Only use `send_message(target=\"telegram\", ...)` for urgent/important/actionable items that need Vincent now: VIP actionable messages/calls, missed/ringing family calls, emergencies, safety/medical/school-pickup issues, money/legal/account/security matters, explicit call-back/decision-now requests, or pipeline errors. Routine logistics, casual family pings, generic group events, links, standalone codes, normal confirmations, and suggested replies should be local-only or sidecar-only.\n\nDo NOT send WhatsApp replies directly unless Vincent explicitly approves them or the category is pre-approved.\n\n### Tone (when proposing or sending replies)\n- Casual, direct, no sign-offs (no \"Thanks & Regards\", no \"- Vincent\")\n- Ask questions instead of giving instructions\n- Keep it natural β€” like texting a friend\n- Christian and grace-based: warm, humble, encouraging, non-judgmental, peace-making\n- Never use em dashes in drafted replies or approval messages. Use commas, periods, colons, or parentheses instead.\n- For prayer/birthday contexts, use gentle faith-filled language; avoid sounding preachy, performative, or overly formal\n\n### What to Do With Each Message\n\n**DMs (personal chats):**\n- Analyze intent\n- Notify Vincent on Telegram (via `send_message(target=\"telegram\", ...)`) only if important/actionable/time-sensitive, requires a decision, comes from a VIP contact\n- If you have a useful suggested reply, POST it as a draft to the sidecar instead of a Telegram notification β€” Vincent gets the buttons inline\n- Casual/social/non-actionable DMs β†’ local log only, no Telegram notification, no draft\n- Do NOT reply unless Vincent says to\n- VIP contacts β†’ **call Vincent immediately without asking**. Do NOT post a draft. Just call. During quiet hours (1am-10am SGT), suppress the call and send Telegram notification only, unless life-and-death emergency.\n\n**Group chats β€” events/meetups:**\n- If someone posts about an event, gathering, meetup, dinner, etc:\n 1. Extract: what, when, where, who\n 2. Notify Vincent on Telegram with details (no draft needed unless you also want to suggest an RSVP reply)\n 3. Ask: \"Want me to add this to Google Calendar and set a reminder?\"\n\n**Group chats β€” prayer requests / praying for others:**\n- If someone asks for prayer, shares a burden, illness, grief, struggle, thanksgiving, or asks the group to pray:\n 1. Fetch recent group context first: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n 2. Identify who/what to pray for and any sensitive details\n 3. Draft a short, grace-based Christian reply/prayer\n 4. POST it to the sidecar with `summary` describing the request and `draft_text` being the prayer\n 5. Do NOT post to WhatsApp unless Vincent approves\n\n**Group chats β€” birthday greetings:**\n- If you see birthday wishes for someone in a group:\n 1. Fetch recent group context first: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n 2. Note who the birthday person is and relationship/context if inferable\n 3. Draft a warm, grace-based Christian birthday greeting\n 4. POST it to the sidecar\n 5. Do NOT post to WhatsApp unless Vincent approves\n\n**Group chats β€” general:**\n- Casual banter, memes, forwards, broadcast spam β†’ ignore silently\n- Religious content shares β†’ ignore unless specifically addressed to Vincent\n\n### When to Notify Vincent (Telegram) vs Post a Draft\n\n**Use `send_message(target=\"telegram\", ...)` sparingly** for plain notifications only when the message needs Vincent's immediate attention: VIP actionable DMs/calls, emergencies, school pickup/child logistics requiring action now, money/legal/account/security matters, explicit call-back/decision-now requests, or pipeline errors.\n\n**Use the sidecar (POST `/drafts`)** whenever you have a suggested reply for Vincent to approve: prayer responses, birthday greetings, DM replies, group RSVPs. Do not also send the same summary to the main Telegram chat.\n\nDo **not** notify or draft for casual/social/non-actionable DMs, routine logistics, read receipts/status updates, religious shares not addressed to Vincent, memes, forwarded/broadcast/spam, generic group events, standalone codes, normal confirmations, or general group chatter. Family messages are not automatically main-chat alerts unless urgent, safety-related, school/pickup critical, or explicitly requiring Vincent now.\n\n## Intent Categories\n\nWhen analyzing incoming messages, classify as:\n\n1. **Urgent** β€” time-sensitive, emergencies β†’ call Vincent outside quiet hours; during quiet hours use Telegram only unless life-and-death\n2. **Important** β€” decisions, money, appointments, anything consequential β†’ call Vincent outside quiet hours; during quiet hours notify on Telegram only unless life-threatening\n3. **Request** β€” asking Vincent to do something β†’ **notify Vincent** (Telegram) or post a draft if you have a safe reply\n4. **Enquiry** β€” asking for info, pricing, availability β†’ auto-reply if context available, otherwise post a draft\n5. **Confirmation** β€” confirming plans, bookings β†’ acknowledge and notify Vincent\n6. **Casual/Social** β€” greetings, memes, forwards β†’ usually ignore or react\n7. **Broadcast/Spam** β€” forwarded job posts, chain messages β†’ ignore\n\n## Finding Individual Contacts and JIDs\n\nWhen Vincent asks to monitor or prioritize a specific WhatsApp contact and only gives a name:\n\n1. Query contacts and chats, then search both display names and serialized message metadata:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n python3 - <<'PY'\n import os, requests, json\n base=os.environ['EVOLUTION_API_URL']; inst=os.environ['EVOLUTION_INSTANCE']; key=os.environ['EVOLUTION_API_KEY']\n headers={'apikey':key,'Content-Type':'application/json'}\n terms=['whye','keong'] # replace with lowercase search terms\n for ep in ['findContacts','findChats']:\n r=requests.post(f'{base}/chat/{ep}/{inst}', headers=headers, json={}, timeout=30)\n print('endpoint', ep, 'status', r.status_code)\n for obj in (r.json() if r.ok else []):\n if any(t in json.dumps(obj, ensure_ascii=False).lower() for t in terms):\n print(json.dumps(obj, ensure_ascii=False)[:1000])\n PY\n ```\n2. If `findChats` shows `remoteJid` ending in `@lid`, inspect `lastMessage.key.remoteJidAlt` for the phone JID. Store both in `memory/whatsapp-vip-contacts.json` when configuring VIP/call rules.\n3. Verify the number exists with `/chat/whatsappNumbers/{instance}` before finalizing.\n4. Validate the JSON config after editing: `python3 -m json.tool /opt/data/agent-state/memory/whatsapp-vip-contacts.json >/tmp/wa_vip_validated.json`.\n\n## Finding Group JIDs\n\nThe `group/fetchGroups` endpoint returns 404 on some Evolution API instances (including ours). To find a group JID when you don't have it:\n\n1. **Use `chat/findMessages` with an empty `where` filter** to pull recent messages across all chats, then extract unique `remoteJid` values containing `@g.us`.\n2. **Match by content**: Fetch messages from each group JID using `where.key.remoteJid` and match sender names or message text to identify the target group.\n3. **Known group JIDs** (store in `whatsapp-vip-contacts.json` as you discover them):\n - ps112 family group: `6591887488-1464061170@g.us`\n - Overcomers group: `6591444911-1614821287@g.us`\n - η₯žηš„εŽηΎŽ group: `6596756578-1405873840@g.us` (content-sharing, casual/broadcast, local log only)\n - Overcomers/TCians group: `120363230236878544@g.us` (church fellowship, PJ sends weekly attendance reports, routine attendance = local log only)\n\nExample search:\n```bash\ncurl -s -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"remoteJid\": \"6591444911-1614821287@g.us\"}}, \"limit\": 10, \"order\": \"desc\"}'\n```\n\n## Processing Image/Media Messages\n\nWhen an incoming WhatsApp message is an image (imageMessage), video, or other media with no text or only a caption:\n\n1. **Fetch conversation context** as usual with `wa_recent_messages.py`\n2. **Download the media** using the Evolution API:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"message\": {\"key\": {\"id\": \"MESSAGE_ID\", \"fromMe\": false, \"remoteJid\": \"GROUP_OR_CHAT_JID\"}}, \"convertToMp4\": false}' \\\n -o /tmp/wa_media_response.json\n ```\n Then decode: `python3 -c \"import json,base64; d=json.load(open('/tmp/wa_media_response.json')); open('/tmp/wa_media.jpg','wb').write(base64.b64decode(d['base64']))\"`\n3. **Analyze the image** using `vision_analyze` tool with `/tmp/wa_media.jpg` path and a context-aware question\n4. **Fallback if vision is unavailable**: Use `tesseract /tmp/wa_media.jpg stdout` for OCR, combined with pixel analysis (PIL/numpy) and conversation context to infer what the image contains\n5. **Album messages** (`albumMessage`): These bundle multiple images. Each image arrives as a separate `imageMessage` with a `messageAssociation.parentMessageKey` linking them to the album. Download each image separately. The `expectedImageCount` field tells how many images to expect\n6. **messageAssociation**: When images have `parentMessageKey` with `fromMe: true`, they are replying to a message Vincent sent. Use this for context\n\n### Audio / Voice Note Messages\nWhen an incoming WhatsApp message is `audioMessage` or a voice note (`ptt: true`), treat it like other media, but transcribe before intent classification when possible:\n\n1. **Fetch conversation context first**, especially in groups:\n ```bash\n python3 /opt/data/scripts/wa_recent_messages.py \"\" 50\n ```\n2. **Download the audio** with Evolution API, saving the response to a file:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n cat > /tmp/wa_audio_req.json <<'JSON'\n {\"message\":{\"key\":{\"id\":\"MESSAGE_ID\",\"fromMe\":false,\"remoteJid\":\"GROUP_OR_CHAT_JID\"}},\"convertToMp4\":false}\n JSON\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_audio_req.json -o /tmp/wa_audio_response.json\n python3 - <<'PY'\n import json, base64\n d=json.load(open('/tmp/wa_audio_response.json'))\n b=d.get('base64') or d.get('data',{}).get('base64')\n if ',' in b[:100]: b=b.split(',',1)[1]\n open('/tmp/wa_audio.ogg','wb').write(base64.b64decode(b))\n PY\n ```\n3. **Convert OGG/Opus to WAV**:\n ```bash\n ffmpeg -y -i /tmp/wa_audio.ogg -ar 16000 -ac 1 /tmp/wa_audio.wav\n ```\n4. **Transcribe fallback when Whisper is unavailable**: the container may not have `whisper`, `faster-whisper`, `torch`, or `speech_recognition` installed globally. A reliable lightweight fallback is a temp venv with SpeechRecognition and Google web speech:\n ```bash\n python3 -m venv /tmp/srvenv\n /tmp/srvenv/bin/pip install --quiet SpeechRecognition\n /tmp/srvenv/bin/python - <<'PY'\n import speech_recognition as sr\n r=sr.Recognizer()\n with sr.AudioFile('/tmp/wa_audio.wav') as source:\n audio=r.record(source)\n print(r.recognize_google(audio, language='en-SG'))\n PY\n ```\n5. **Check adjacent voice notes from the same sender**. WhatsApp voice notes are often sent in quick sequential chunks. Download and transcribe nearby `audioMessage` records from the same sender before deciding whether to notify Vincent.\n6. Use the transcript plus recent context to classify the message. For group meetup planning, extract what/when/where/who and notify Vincent if actionable, including transcript confidence.\n\n### Video Messages\n\nWhen an incoming WhatsApp message is `videoMessage` (with or without a caption):\n\n1. **Fetch conversation context** first, as with any media message: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n2. **Download the video** using Evolution API (same pattern as images/audio β€” save response to file, then decode):\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n cat > /tmp/wa_video_req.json <<'JSON'\n {\"message\":{\"key\":{\"id\":\"MESSAGE_ID\",\"fromMe\":false,\"remoteJid\":\"CONTACT_JID\"}},\"convertToMp4\":false}\n JSON\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_video_req.json -o /tmp/wa_video_response.json\n python3 - <<'PY'\n import json, base64\n d = json.load(open('/tmp/wa_video_response.json'))\n b = d.get('base64') or d.get('data', {}).get('base64', '')\n if ',' in b[:100]: b = b.split(',', 1)[1]\n open('/tmp/wa_video.mp4', 'wb').write(base64.b64decode(b))\n print(f\"Video saved: {len(base64.b64decode(b))} bytes\")\n PY\n ```\n3. **Extract representative frames** with ffmpeg (do NOT try to analyze video directly β€” vision tools work on images):\n ```bash\n # Frame at start\n ffmpeg -y -ss 0 -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame0.jpg\n # Frame at ~1/3 of the way (good for short videos)\n ffmpeg -y -ss 3 -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame3s.jpg\n # If video is longer (>15s), also extract at midpoint\n # ffprobe -v quiet -print_format json -show_format /tmp/wa_video.mp4 # get duration\n # ffmpeg -y -ss $HALF_DURATION -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame_mid.jpg\n ```\n4. **Analyze the frames** with `vision_analyze`. Ask specific questions about content: what objects/people/text are visible, what type of video this appears to be (screen recording, personal clip, forwarded meme, etc.), and any text or identifying details.\n5. **Use the caption** from `videoMessage.caption` if present (can be empty/null). Combine frame analysis + caption + conversation context to classify the message intent.\n6. **Transcription of audio in video** is generally not feasible with available tools. If the video likely contains important spoken content (e.g. voice memo style), note this limitation and flag it to Vincent.\n\n**Key points:**\n- Always use `-o` flag and file-based decode; videos can be 1-10MB and base64 responses will be even larger\n- `--max-time 25` on curl is essential for video downloads\n- Short casual clips (showing a key fob, a parking spot, quick view of surroundings) are usually casual/social β€” classify accordingly and don't over-escalate\n- If `vision_analyze` fails (400 on models that don't support images, or 429 rate limit), fall back to conversation context inference alone\n\n### Poll Messages\nWhen `pollUpdateMessage` appears in chat history (someone voted), the message contains a `pollCreationMessageKey` with the original poll's message `id`. To get the poll question and options, query:\n```bash\ncurl -s -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"id\": \"POLL_MESSAGE_ID\", \"remoteJid\": \"GROUP_JID\"}}, \"limit\": 1}'\n```\nThe response will have `pollCreationMessageV3` with `name` (the poll question) and `options` array. Always look up the poll when votes or poll references appear, since the poll text contains the event details.\n\n### `associatedChildMessage` (Motion Photos / Companion Media)\nWhatsApp motion photos and companion media arrive as `associatedChildMessage` type. These are short videos or supplementary images attached to a parent message via `messageAssociation.parentMessageKey`. The structure is:\n\n```\nmessage: {\n messageContextInfo: {\n messageAssociation: {\n associationType: 12,\n parentMessageKey: { id, fromMe, remoteJid }\n }\n },\n associatedChildMessage: {\n message: {\n videoMessage: { ... } // or imageMessage: { ... }\n }\n }\n}\n```\n\nKey points:\n- The `parentMessageKey.id` links back to the main message (the image/video they accompany)\n- When `parentMessageKey.fromMe: true`, the companion media is replying to a message Vincent sent\n- Captions may appear on the child's `videoMessage.caption` or `imageMessage.caption` inside the nested message, NOT at the top level\n- These often contain motion-photo short videos (3 seconds) or album companion images\n- Treat them as media attachments to the parent; don't process them as standalone messages requiring independent intent analysis\n\n### Sequential Image Sets (Non-Album)\nMultiple images sent rapidly by the same sender at similar timestamps may NOT be wrapped in an `albumMessage`. They appear as separate `imageMessage` entries. Always check for adjacent images from the same sender and download+OCR all of them together, as they often form a coherent multi-page set (e.g., route info, comparison tables, maps). Missing one can lose important context.\n\n### Key pitfalls\n- The Evolution API `getBase64FromMediaMessage` endpoint returns the full base64 payload (can be 100KB+ for images). Use `-o` flag to save to file, then parse with Python. Do NOT try to pipe it directly\n- `curl` with `--max-time 25` is essential; without it, large media responses can hang\n- The `albumMessage` type itself has no `fromMe` field; check each individual `imageMessage` instead\n- Image thumbnails in the webhook payload are truncated; always download the full image for analysis\n- **WhatsApp media URLs expire**: CDN download links (`mmg.whatsapp.net`) expire after roughly 24-48 hours. When `getBase64FromMediaMessage` returns a 400/404 error like \"Failed to fetch stream from https://mmg.whatsapp.net/...\", the media is no longer retrievable via the API. This means you CANNOT download adjacent/older voice notes, images, or documents for context if they're more than a day or two old. Fall back to conversation text context only. Fresh/incoming messages are almost always still downloadable.\n- **Voice note transcription reliability**: Short voice notes (<15s), especially in Singlish/Chinese-English mix common in Singapore contacts, often produce unreliable transcriptions via SpeechRecognition+Google. Always flag low-confidence transcriptions to Vincent rather than acting on potentially garbled text.\n- **Vision API failures**: `vision_analyze` can fail with 429 (rate limit) **or 400** (model doesn't support image input, e.g. glm-5.1 via ollama-cloud). Always have a fallback plan. When vision is unavailable, try tesseract OCR, then pixel analysis (PIL/numpy for brightness zones, text region detection via row-level dark-pixel density)\n- **OCR can fail on decorative/complex images**: Obituaries, memorial notices, religious flyers with ornate fonts, or dark-background images often defeat tesseract even with aggressive preprocessing (upscaling, thresholding, inverting, PSM mode variations). When OCR returns gibberish or empty, fall back to conversation context inference: use preceding/surrounding messages to determine what the image likely contains, then draft an appropriate response based on that context\n- **OCR on devotional/dark-background images**: These often defeat tesseract on full-image scans. Use a multi-strategy approach: (1) run tesseract on the full image first, (2) if text is sparse, crop into regions (top 15%, middle, bottom 15%), upscale 2-3x, enhance contrast (2-3x), and try both normal and inverted (`ImageOps.invert`) versions with `--psm 6`, (3) combine all partial OCR results. For Exodus/verse images, the title (e.g. \"WHEN MOSES' HANDS WERE\") is often in decorative font that OCR misses, while the verse body text is usually readable.\n- **Prioritize conversation context**: When processing group images, always fetch recent context FIRST (before attempting vision/OCR). If image analysis fails, the conversation history often makes the image's purpose clear enough to draft a meaningful reply\n- **DM conversation context is one-sided (fromMe only)**: The Evolution API `findMessages` endpoint often returns only Vincent's sent messages (`fromMe: true`) for DM chats, not the contact's incoming messages. This means `wa_recent_messages.py` can show only one side of the conversation. When processing a DM webhook, do NOT rely solely on fetched history for the contact's side. The webhook payload already contains the incoming message text. Use what the webhook gives you plus Vincent's side from the API. If you need the contact's recent messages, check if `fromMe: false` records exist; if none are returned, the API simply doesn't have them and you must work with what you have (webhook message + Vincent's replies for context).\n- **`findMessages` ignores `key.fromMe` filter**: Passing `\"key.fromMe\": false` in the `where` clause still returns all messages (including `fromMe: true`). To find messages from a specific sender, fetch results and filter client-side by checking `fromMe` in Python.\n- **`findMessages` caps at ~50 records per request**: Even with `limit: 200` or `limit: 500`, the API returns at most 50 records. When one party sends many consecutive messages, the other party's messages may not appear in the first 50 results. Use pagination or fetch multiple pages if you need older messages.\n- **Video frame extraction**: Use `ffmpeg -y -ss N -i video.mp4 -frames:v 1 -update 1 output.jpg` (the `-update 1` flag is needed; without it ffmpeg treats the output as an image sequence and fails). Extract frames at multiple timestamps (start, ~1/3, midpoint for longer videos) since a single frame may miss the key content.\n- **ffmpeg \"Invalid color space\" failure on WhatsApp videos**: Some WhatsApp H.264 videos have `color_space=reserved` and `color_primaries=reserved` metadata that ffmpeg cannot decode at all. Every codec/format/filter combination fails with \"Invalid color space\" / return code 234. This is NOT fixable with `-vf setparams`, `-color_trc`, `-colorspace`, `-color_primaries`, or `-vf format=` flags because the error occurs at the decoder level before any filter runs. **Do not waste time trying ffmpeg workarounds.** When this happens, fall back to: (1) conversation context inference (who sent it, what thread it's in, adjacent messages), (2) the jpegThumbnail embedded in the videoMessage (small but gives a rough idea; base64-encoded JPEG, typically 600-900 bytes, use `vision_analyze` on it if available), or (3) if all analysis fails, just classify based on surrounding text context.\n- **`wa_recent_messages.py` JSON output breaks `json.loads()`**: The script's stdout contains WhatsApp message content with control characters (emoji, CJK, deviceListMetadata binary blobs) that cause `json.loads()` to fail with `Invalid control character` or `Expecting ',' delimiter` errors. Even `strict=False` does not reliably fix it. **Reliable approach**: use `curl -o /tmp/wa_msgs.json` to save the Evolution API response directly to a file (the file write handles binary safely), then parse with `json.load(open('/tmp/wa_msgs.json'))` in Python. Example:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n curl -s --max-time 15 -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"remoteJid\": \"CONTACT_JID\"}}, \"limit\": 20}' -o /tmp/wa_msgs.json\n ```\n Then in Python: `data = json.load(open('/tmp/wa_msgs.json')); records = data['messages']['records']`\n\n## API Reference\n\nFor full API endpoints (media, contacts, location, buttons) β€” see [references/api.md](references/api.md).\n\n[Skill directory: /opt/data/skills/personal/whatsapp-evolution]\nResolve any relative paths in this skill (e.g. `scripts/foo.js`, `templates/config.yaml`) against that directory, then run them with the terminal tool using the absolute path.\n\n[This skill has supporting files:]\n- references/api.md -> /opt/data/skills/personal/whatsapp-evolution/references/api.md\n- scripts/process_queue.py -> /opt/data/skills/personal/whatsapp-evolution/scripts/process_queue.py\n- scripts/send_text.sh -> /opt/data/skills/personal/whatsapp-evolution/scripts/send_text.sh\n\nLoad any of these with skill_view(name=\"personal/whatsapp-evolution\", file_path=\"\"), or run scripts directly by absolute path (e.g. `node /opt/data/skills/personal/whatsapp-evolution/scripts/foo.js`).\n\nThe user has provided the following instruction alongside the skill invocation: Incoming WhatsApp webhook received.\n\nEvent: messages.upsert\nInstance: n8n-WA\nSender: Shalom Evonne\nSender JID: 120363418526182134@g.us\nFrom me: False\nMessage type: conversation\nMessage: {\n \"event\": \"messages.upsert\",\n \"instance\": \"n8n-WA\",\n \"data\": {\n \"key\": {\n \"remoteJid\": \"120363418526182134@g.us\",\n \"fromMe\": false,\n \"id\": \"3EB04D3EBD896045906E39\",\n \"participant\": \"131503308673050@lid\",\n \"participantAlt\": \"6591887488@s.whatsapp.net\",\n \"addressingMode\": \"lid\"\n },\n \"pushName\": \"Shalom Evonne\",\n \"status\": \"DELIVERY_ACK\",\n \"message\": {\n \"conversation\": \"Good evening Shalom Family!\\n\\nTomorrow is Monday, Week Even.\\n\\nNaomi Schedule (G2/G3):\\n8:00 - Morning Assembly\\n8:25 - CMTime/CCE\\n9:00 - English Language\\n9:35 - English Language\\n10:10 - Recess\\n10:45 - Mother Tongue\\n11:20 - Basic Leadership\\n11:55 - PE (Mrs Ku)\\n12:30 - Lunch\\n1:05 - Literature\\n1:40 - G2/G3 Subject\\n2:15 - Dismissal\\n\\nPE TOMORROW! Please pack PE gear tonight. Teacher is Mrs Ku.\\n\\nHave a blessed rest of the weekend and may God go with Naomi into the new week!\\n\\nWith love,\\nShalom Evonne\"\n },\n \"contextInfo\": null,\n \"messageType\": \"conversation\",\n \"messageTimestamp\": 1779620446,\n \"instanceId\": \"b2bfaa30-d5f1-4b22-b979-791892619414\",\n \"source\": \"web\"\n },\n \"destination\": \"https://jimmy.huper.asia/webhooks/whatsapp-incoming\",\n \"date_time\": \"2026-05-24T08:00:46.675Z\",\n \"sender\": \"6590054438@s.whatsapp.net\",\n \"server_url\": \"https://evolution.huper.asia\",\n \"apikey\": \"5E4C532223A0-460B-A749-16C584F56E74\"\n}\n\nPrimary goal: REDUCE NOISE. Stay silent in the main Telegram chat unless the message truly needs Vincent's immediate attention. Approval-worthy reply drafts should go to the jimmy-wa-drafts sidecar, NOT to the main Telegram chat.\n\nFollow the whatsapp-evolution skill and /opt/data/agent-state/memory/whatsapp-vip-contacts.json, with these stricter notification rules:\n\nABSOLUTE SILENCE RULES:\n- If fromMe is true: ignore. Do NOT call send_message. Final response local log only.\n- Ignore status/update/read-receipt events or payloads with no real message content.\n- Do NOT send main Telegram summaries for ordinary DMs, routine logistics, casual family messages, generic group events, group chatter, forwards, memes, devotional/religious shares, links, standalone codes, normal confirmations, or suggested replies.\n- Do NOT ask \"Want me to add this to Calendar?\" for WhatsApp group events unless Vincent is explicitly invited, named, responsible, or the event is time-sensitive and clearly relevant to him today.\n- Do NOT surface family messages just because they are family. Only surface if urgent, safety-related, school/pickup critical, money/legal/medical, or explicitly requires Vincent now.\n\nSIDEcar RULES:\n- If a reply may be useful but it is not urgent, POST it to https://jimmy-draft.huper.asia/drafts using the whatsapp-evolution Draft Approvals flow. Do NOT duplicate that draft/summary in the main Telegram chat.\n- Prayer and birthday replies should go to the sidecar when appropriate, not the main Telegram chat, unless the underlying situation is urgent/sensitive and Vincent needs immediate awareness.\n- Before drafting any reply or approval suggestion, fetch recent context: python3 /opt/data/scripts/wa_recent_messages.py \"120363418526182134@g.us\" 30. For active/ambiguous groups use 50.\n\nMAIN TELEGRAM ALERTS ARE ALLOWED ONLY FOR:\n1. VIP contacts marked always_call or always_notify when the message is important/actionable, or any VIP call event.\n2. Calls from family/VIP that appear missed/ringing and likely need immediate attention. Avoid repeated alerts for accepted/terminated duplicate call state updates.\n3. Emergencies, safety issues, medical issues, school pickup/child logistics requiring action now, legal/money/account/security matters, or time-sensitive commitments in the next few hours.\n4. Messages explicitly asking Vincent to call back, decide now, approve payment, provide OTP/code for an active task, or respond urgently.\n5. Pipeline errors or webhook processing failures.\n\nCALL RULES:\n- VIP contacts (e.g. Dn Jeff, Ng Whye Keong): call Vincent immediately outside quiet hours when actionably important or on ringing/missed call. During 1am-10am SGT, Telegram only unless life-and-death emergency.\n- Do not call for casual VIP confirmations or non-actionable courtesy messages.\n\nTONE/RULES FOR DRAFTS:\n- Proposed replies must be Christian and grace-based where appropriate: warm, humble, encouraging, non-judgmental, peace-making, not preachy.\n- Never use em dashes in drafted replies or approval messages. Use commas, periods, colons, or parentheses instead.\n- NEVER auto-reply directly on WhatsApp unless Vincent explicitly approves or future config marks that exact category pre-approved.\n\nDelivery rule:\n- For allowed main alerts only: call send_message(target=\"telegram\", message=...). Keep it concise.\n- For non-urgent reply suggestions: use the sidecar only.\n- For everything else: DO NOT call send_message. Keep final response as local-only log.\n", "timestamp": 1779620466.436012, "anchor": true}, {"id": 81567, "role": "assistant", "content": "", "timestamp": 1779620466.456067, "tool_calls": [{"id": "call_0kH1suJ6FmC7vLz5ovvloBOT", "call_id": "call_0kH1suJ6FmC7vLz5ovvloBOT", "response_item_id": "fc_092d8a3173a1f34c016a12da67dce88191a438382037c7a589", "type": "function", "function": {"name": "read_file", "arguments": "{\"path\":\"/opt/data/agent-state/memory/whatsapp-vip-contacts.json\",\"limit\":200}"}}]}, {"id": 81569, "role": "assistant", "content": "Local-only log: Routine family/group school schedule reminder from Shalom Evonne about Naomi’s Monday timetable and PE gear tomorrow. No urgent action, no reply draft needed, no main Telegram alert sent.", "timestamp": 1779620466.505175}], "bookend_end": [], "messages_before": 0, "messages_after": 4}, {"session_id": "20260521_110129_233d69e8", "when": "May 21, 2026 at 11:01 AM", "source": "webhook", "model": "gpt-5.5", "title": "WhatsApp Group Schedule Update", "matched_role": "user", "match_message_id": 75250, "snippet": "...draft_text\": \"Thanks >>>Karen<<<, will miss you both β€” praying for a good time too πŸ™\",\n \"extras\": {\n \"group_jid\": \"120363351787895353@g.us\",\n \"sender_name\": \">>>Karen<<< >>>Tan<<<\",\n \"sender_jid\": \">>>6597630553<<<@s.whatsapp.net\",\n \"category\": \">>>prayer<<>>blessing<<<\",\n \"original_message\": \"Happy >>>cycling<<<, to those who...", "bookend_start": [], "messages": [{"id": 75250, "role": "user", "content": "[IMPORTANT: The user has invoked the \"whatsapp-evolution\" skill, indicating they want you to follow its instructions. The full skill content is loaded below.]\n\n---\nname: whatsapp-evolution\ndescription: WhatsApp messaging via Evolution API β€” send, receive, analyze, and auto-reply. Use when processing an incoming WhatsApp message delivered via the webhook, replying on Vincent's behalf, liaising with contacts, sending messages, or handling WhatsApp-related tasks. Covers intent analysis, auto-reply rules, escalation (VIP β†’ phone call), conversation context fetching, and draft approvals via the jimmy-wa-drafts sidecar.\n---\n\n# WhatsApp via Evolution API\n\n## Credentials\nStored in `/opt/data/secrets/.whatsapp-evolution-credentials`:\n- `EVOLUTION_API_URL`, `EVOLUTION_API_KEY`, `EVOLUTION_INSTANCE`\n\nLoad with: `export $(cat .whatsapp-evolution-credentials | xargs)`\n\n## Sending Messages (direct, no approval)\n\n```bash\nexport $(cat .whatsapp-evolution-credentials | xargs)\ncurl -s -X POST \"${EVOLUTION_API_URL}/message/sendText/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"number\": \"6591234567\", \"text\": \"Hello!\"}'\n```\n\n- Phone format: country code, no `+` (e.g. `6590054438`)\n- Groups: use JID `{groupid}@g.us`\n- For media/location/contacts β€” see [references/api.md](references/api.md)\n\nUse direct send only when Vincent has explicitly commanded/approved the exact send, or for pre-approved auto-replies/follow-ups. For any reply that needs Vincent's review, use the sidecar (next section).\n\n### Sending trip/camp details from retrieved emails\n\nWhen Vincent explicitly asks to send extracted school trip/camp details to a child or family member, send directly. Keep it practical and grounded:\n- Include dates, reporting/pick-up times, flights, venue/itinerary highlights, deadlines, forms, and safety/medication notes when found.\n- If an official packing list is missing, say that clearly. Do not invent one. You may add a short β€œbased on itinerary/weather, prepare…” list, clearly labeled as practical guidance rather than official school instructions.\n- Avoid long explanations about the search process in WhatsApp. Send the useful details.\n\n### Sending a local image/file as media\n\nFor user-approved media sends, base64-encode the local file and call `sendMedia`:\n\n```bash\nexport $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\npython3 - <<'PY'\nimport base64, json\npath = '/tmp/file.png'\npayload = {\n 'number': '6591234567',\n 'mediatype': 'image',\n 'mimetype': 'image/png',\n 'caption': 'Caption here',\n 'media': base64.b64encode(open(path, 'rb').read()).decode(),\n 'fileName': 'file.png',\n}\nopen('/tmp/wa_media_payload.json', 'w').write(json.dumps(payload))\nPY\ncurl -s --max-time 60 -X POST \"${EVOLUTION_API_URL}/message/sendMedia/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_media_payload.json\n```\n\nWhen sending QR tickets or passes, verify the crop/image contains the full QR code and the intended attendee name before sending; the QR screenshot is effectively the redeemable ticket.\n\n## Fetching Conversation Context\n\nBefore drafting or proposing any reply, always fetch recent chat history for context:\n\n```bash\npython3 /opt/data/scripts/wa_recent_messages.py \"6591234567@s.whatsapp.net\" 30\n```\n\nUse `limit=30` by default. For active group chats or ambiguous situations, fetch up to 50 messages. Read enough context to understand who is being discussed, current tone, previous commitments, and whether Vincent has already replied. The script returns newest messages first (descending order).\n\n## Draft Approvals β€” `jimmy-wa-drafts` sidecar\n\nWhen you need Vincent to approve a WhatsApp reply, POST the draft to the sidecar. It owns the entire approval flow: Telegram message with 4 inline buttons, Telegram WebApp for Edit/Context, sending to WhatsApp via Evolution API on Approve.\n\n### Posting a draft\n\nAlways include an `extras` object with the original WA context. The sidecar passes `extras` through verbatim if Vincent later clicks \"Context\" β€” without it, your redraft will be flying blind.\n\n```bash\ncat > /tmp/wa_draft.json <<'JSON'\n{\n \"contact\": \"120363351787895353@g.us\",\n \"contact_name\": \"Cycling group\",\n \"summary\": \"Karen sent a blessing over the bike ride; she's not joining\",\n \"draft_text\": \"Thanks Karen, will miss you both β€” praying for a good time too πŸ™\",\n \"extras\": {\n \"group_jid\": \"120363351787895353@g.us\",\n \"sender_name\": \"Karen Tan\",\n \"sender_jid\": \"6597630553@s.whatsapp.net\",\n \"category\": \"prayer/blessing\",\n \"original_message\": \"Happy cycling, to those who can make it… Dn Jeff and I had made plans for this PM so we won't be able to join y'all. Praying the Lord's favor, protection and good-weather blessings be upon y'all and your route!\",\n \"extracted\": {\n \"what\": \"Cycling from East Coast Park to Changi Village, then dinner\",\n \"when\": \"Today, 4:00 PM\",\n \"where\": \"East Coast Park to Changi Village\",\n \"who\": \"Group members who voted yes for cycling/dinner or dinner\"\n },\n \"confidence\": \"high\",\n \"why_appropriate\": \"Multiple group messages are blessings/wishes; Vincent has not replied; tone matches grace-based norm.\",\n \"wa_message_id\": \"ABC123XYZ\"\n }\n}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_draft.json\n```\n\nResponse: `{\"draft_id\":\"abc123def\",\"telegram_message_id\":42}`\n\nWhat to put in `extras` (use whatever makes sense for the message type):\n\n- **Always**: `original_message` (verbatim WA text), `sender_name`, `category` (e.g. `prayer`, `birthday`, `event`, `dm-reply`, `enquiry`)\n- **Group messages**: `group_jid`, `sender_jid`\n- **DMs**: `sender_jid`\n- **Strongly recommended (drives the rich Telegram render)**: `extracted` (dict of who/what/when/where), `confidence` (`high`/`medium`/`low`), `why_appropriate` (one-line reasoning shown to Vincent so he can decide)\n- **Optional**: `wa_message_id`, `tone_hint` (e.g. `warm-grace-based`), `sender_phone` (auto-derived from `sender_jid` if omitted)\n\nVincent receives a Telegram message in the `@jimmy_WA_bot` chat with 4 inline buttons. The sidecar handles every interaction; **you do not track replies, callbacks, or message IDs**. Just POST and move on.\n\n| Button | Behavior |\n|--------|----------|\n| **Approve** | Sidecar sends draft to WhatsApp |\n| **Edit** | Opens a Telegram WebApp with draft pre-filled; Vincent edits + sends; sidecar pushes to WhatsApp |\n| **Context** | Opens a Telegram WebApp with empty textarea + the original WA message visible; Vincent provides context. (When `HERMES_REDRAFT_WEBHOOK` is wired in the sidecar's env, this will POST back to Jimmy for a redraft. Until then, the context is just stored on the draft.) |\n| **Pass** | Sidecar deletes the draft |\n\n### JSON escaping when `draft_text` contains quotes\n\nSingle quotes (`It's`, `God's`) break shell interpolation. Write the JSON to a temp file first:\n\n```bash\ncat > /tmp/wa_draft.json <<'JSON'\n{\"contact\":\"6591234567@s.whatsapp.net\",\"contact_name\":\"John\",\"summary\":\"…\",\"draft_text\":\"It's nice to hear from you!\"}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_draft.json\n```\n\n### Networking\n\n`https://jimmy-draft.huper.asia` is the public HTTPS endpoint of the sidecar (TLS-terminated by Traefik on ubuntu-server). HTTPS is required because Hermes blocks plain-HTTP commands via tirith. The internal Docker DNS hostname http://jimmy-wa-v64mqe:8080 also works for non-shell calls but the agent is forbidden from using it.\n\n### Sidecar status & logs\n\n- Source: (auto-deploys on push)\n- Public URL: (`/health` for ping)\n- Container: `jimmy-wa-v64mqe.1.` on ubuntu-server\n- Bot: `@jimmy_WA_bot`\n\n## Handling a Redraft Trigger (`wa-redraft` webhook)\n\nWhen Vincent clicks the **Context** button on a draft, the sidecar:\n1. Edits the original Telegram message to \"πŸ”„ Adding context for X… agent is redrafting.\" (removes the old buttons so Vincent can't approve the now-stale text)\n2. POSTs to your `wa-redraft` webhook (`http://localhost:8644/webhooks/wa-redraft`) with the original draft fields plus Vincent's `context` instructions and the `extras` you stored when you first created the draft.\n\nWhen you receive a `wa-redraft` activation, the prompt template will lay out the fields. Your job:\n\n1. Read Vincent's `context` carefully β€” that's the steering signal. The `previous_draft` is just history, not necessarily a starting point.\n2. Re-fetch recent WA history if the original context isn't enough (`python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`). Use `extras.group_jid` for groups or `contact` for DMs.\n3. Write a NEW draft that addresses Vincent's instructions while staying faithful to the original WA context (in `extras.original_message`, `extras.extracted`).\n4. POST the new draft back to the sidecar **with the same `draft_id`** so the existing draft is updated, not duplicated:\n\n```bash\ncat > /tmp/wa_redraft.json <\",\n \"contact\": \"\",\n \"contact_name\": \"\",\n \"summary\": \"\",\n \"draft_text\": \"\",\n \"extras\": \n}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_redraft.json\n```\n\nThe sidecar will send a NEW Telegram message with 4 fresh buttons. Vincent can Approve, Edit, Context-again (loop), or Pass.\n\nDo **not** send to WhatsApp directly during a redraft β€” the approval flow is the sidecar's job.\n\n## Incoming Message Processing\n\n### Delivery Pipeline (current)\n1. Evolution API POSTs each `MESSAGES_UPSERT` webhook directly to Hermes at `http://localhost:8644/webhooks/whatsapp-incoming` (proxied via `jimmy.huper.asia/webhooks/whatsapp-incoming`)\n2. Hermes webhook route (`whatsapp-incoming`, dynamic subscription) validates and spawns an agent\n3. Agent loads `whatsapp-evolution` skill, follows the rules below\n4. Webhook subscription uses `deliver: log` because webhooks do not support `deliver: local`; the agent explicitly calls `send_message(target=\"telegram\", ...)` only for important/actionable/VIP/event/birthday/prayer/error items, OR posts a draft to the sidecar for approval items.\n\n**No more OpenClaw, no Python webhook server, no port 5001, no keepalive cron.** If messages aren't being processed, check: nginx proxy to port 8644, `hermes webhook list` for the subscription, Evolution API webhook URL setting.\n\n## Contact Rules & Escalation\n\nConfig: `memory/whatsapp-vip-contacts.json`\n\n### Priority Levels\n\n| Priority | Action | Example |\n|----------|--------|---------|\n| **VIP (call-always)** | **CALL Vincent immediately via Twilio without asking.** Do NOT ask for approval first. During quiet hours (1am-10am SGT) suppress call and use Telegram notification only, unless life-and-death emergency. Match VIPs by both phone JID (`659...@s.whatsapp.net`) and LID JID (`...@lid`) when present, because recent chats may arrive with `key.remoteJid` as an LID and the phone number only in `key.remoteJidAlt`. | Dn Jeff (+6597630553) |\n| **Family** | Always notify Vincent on Telegram | ps112 group, family members |\n| **Auto-reply** | Reply based on context file, notify Vincent | Repair shops |\n| **Normal** | Analyze intent, reply if appropriate, notify if important | Everyone else |\n| **Broadcast/Spam** | Ignore silently | Forwarded job posts, chain messages |\n\n### Quiet Hours\n- 1am-10am SGT (17:00-02:00 UTC): avoid phone calls, including for Dn Jeff / VIP call-always contacts\n- **Exception:** life-and-death emergencies always call regardless of quiet hours\n- During quiet hours, use Telegram notification only and mention that a call was suppressed due to quiet hours\n\n## Making a Call via Twilio\n\nCredentials: `/opt/data/secrets/.twilio-credentials-new` (`TWILIO_ACCOUNT_SID`, `TWILIO_AUTH_TOKEN`, `TWILIO_PHONE_NUMBER`). From: `+16614809287`, Vincent: `+6590054438`.\n\nPreferred script: `/opt/data/scripts/call_vincent.py`, now using the Twilio Python SDK baked into the container.\n\n```bash\npython3 /opt/data/scripts/call_vincent.py \"Hello Vincent, this is Jimmy. Urgent summary here.\"\n```\n\n## Action Rules\n\n### Default: DO NOT auto-reply directly, and do not spam main Telegram\nCollect intent and stay local unless something truly needs Vincent's immediate attention.\n\n**When you need Vincent to approve a reply, POST a draft to the sidecar (see \"Draft Approvals\" above).** The sidecar handles the Telegram notification with buttons, the WebApp for editing, and sending to WhatsApp on approve. Do **not** duplicate that draft or summary in the main Telegram chat.\n\nMain Telegram WhatsApp summaries are muted by default. Only use `send_message(target=\"telegram\", ...)` for urgent/important/actionable items that need Vincent now: VIP actionable messages/calls, missed/ringing family calls, emergencies, safety/medical/school-pickup issues, money/legal/account/security matters, explicit call-back/decision-now requests, or pipeline errors. Routine logistics, casual family pings, generic group events, links, standalone codes, normal confirmations, and suggested replies should be local-only or sidecar-only.\n\nDo NOT send WhatsApp replies directly unless Vincent explicitly approves them or the category is pre-approved.\n\n### Tone (when proposing or sending replies)\n- Casual, direct, no sign-offs (no \"Thanks & Regards\", no \"- Vincent\")\n- Ask questions instead of giving instructions\n- Keep it natural β€” like texting a friend\n- Christian and grace-based: warm, humble, encouraging, non-judgmental, peace-making\n- Never use em dashes in drafted replies or approval messages. Use commas, periods, colons, or parentheses instead.\n- For prayer/birthday contexts, use gentle faith-filled language; avoid sounding preachy, performative, or overly formal\n\n### What to Do With Each Message\n\n**DMs (personal chats):**\n- Analyze intent\n- Notify Vincent on Telegram (via `send_message(target=\"telegram\", ...)`) only if important/actionable/time-sensitive, requires a decision, comes from a VIP contact\n- If you have a useful suggested reply, POST it as a draft to the sidecar instead of a Telegram notification β€” Vincent gets the buttons inline\n- Casual/social/non-actionable DMs β†’ local log only, no Telegram notification, no draft\n- Do NOT reply unless Vincent says to\n- VIP contacts β†’ **call Vincent immediately without asking**. Do NOT post a draft. Just call. During quiet hours (1am-10am SGT), suppress the call and send Telegram notification only, unless life-and-death emergency.\n\n**Group chats β€” events/meetups:**\n- If someone posts about an event, gathering, meetup, dinner, etc:\n 1. Extract: what, when, where, who\n 2. Notify Vincent on Telegram with details (no draft needed unless you also want to suggest an RSVP reply)\n 3. Ask: \"Want me to add this to Google Calendar and set a reminder?\"\n\n**Group chats β€” prayer requests / praying for others:**\n- If someone asks for prayer, shares a burden, illness, grief, struggle, thanksgiving, or asks the group to pray:\n 1. Fetch recent group context first: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n 2. Identify who/what to pray for and any sensitive details\n 3. Draft a short, grace-based Christian reply/prayer\n 4. POST it to the sidecar with `summary` describing the request and `draft_text` being the prayer\n 5. Do NOT post to WhatsApp unless Vincent approves\n\n**Group chats β€” birthday greetings:**\n- If you see birthday wishes for someone in a group:\n 1. Fetch recent group context first: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n 2. Note who the birthday person is and relationship/context if inferable\n 3. Draft a warm, grace-based Christian birthday greeting\n 4. POST it to the sidecar\n 5. Do NOT post to WhatsApp unless Vincent approves\n\n**Group chats β€” general:**\n- Casual banter, memes, forwards, broadcast spam β†’ ignore silently\n- Religious content shares β†’ ignore unless specifically addressed to Vincent\n\n### When to Notify Vincent (Telegram) vs Post a Draft\n\n**Use `send_message(target=\"telegram\", ...)` sparingly** for plain notifications only when the message needs Vincent's immediate attention: VIP actionable DMs/calls, emergencies, school pickup/child logistics requiring action now, money/legal/account/security matters, explicit call-back/decision-now requests, or pipeline errors.\n\n**Use the sidecar (POST `/drafts`)** whenever you have a suggested reply for Vincent to approve: prayer responses, birthday greetings, DM replies, group RSVPs. Do not also send the same summary to the main Telegram chat.\n\nDo **not** notify or draft for casual/social/non-actionable DMs, routine logistics, read receipts/status updates, religious shares not addressed to Vincent, memes, forwarded/broadcast/spam, generic group events, standalone codes, normal confirmations, or general group chatter. Family messages are not automatically main-chat alerts unless urgent, safety-related, school/pickup critical, or explicitly requiring Vincent now.\n\n## Intent Categories\n\nWhen analyzing incoming messages, classify as:\n\n1. **Urgent** β€” time-sensitive, emergencies β†’ call Vincent outside quiet hours; during quiet hours use Telegram only unless life-and-death\n2. **Important** β€” decisions, money, appointments, anything consequential β†’ call Vincent outside quiet hours; during quiet hours notify on Telegram only unless life-threatening\n3. **Request** β€” asking Vincent to do something β†’ **notify Vincent** (Telegram) or post a draft if you have a safe reply\n4. **Enquiry** β€” asking for info, pricing, availability β†’ auto-reply if context available, otherwise post a draft\n5. **Confirmation** β€” confirming plans, bookings β†’ acknowledge and notify Vincent\n6. **Casual/Social** β€” greetings, memes, forwards β†’ usually ignore or react\n7. **Broadcast/Spam** β€” forwarded job posts, chain messages β†’ ignore\n\n## Finding Individual Contacts and JIDs\n\nWhen Vincent asks to monitor or prioritize a specific WhatsApp contact and only gives a name:\n\n1. Query contacts and chats, then search both display names and serialized message metadata:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n python3 - <<'PY'\n import os, requests, json\n base=os.environ['EVOLUTION_API_URL']; inst=os.environ['EVOLUTION_INSTANCE']; key=os.environ['EVOLUTION_API_KEY']\n headers={'apikey':key,'Content-Type':'application/json'}\n terms=['whye','keong'] # replace with lowercase search terms\n for ep in ['findContacts','findChats']:\n r=requests.post(f'{base}/chat/{ep}/{inst}', headers=headers, json={}, timeout=30)\n print('endpoint', ep, 'status', r.status_code)\n for obj in (r.json() if r.ok else []):\n if any(t in json.dumps(obj, ensure_ascii=False).lower() for t in terms):\n print(json.dumps(obj, ensure_ascii=False)[:1000])\n PY\n ```\n2. If `findChats` shows `remoteJid` ending in `@lid`, inspect `lastMessage.key.remoteJidAlt` for the phone JID. Store both in `memory/whatsapp-vip-contacts.json` when configuring VIP/call rules.\n3. Verify the number exists with `/chat/whatsappNumbers/{instance}` before finalizing.\n4. Validate the JSON config after editing: `python3 -m json.tool /opt/data/agent-state/memory/whatsapp-vip-contacts.json >/tmp/wa_vip_validated.json`.\n\n## Finding Group JIDs\n\nThe `group/fetchGroups` endpoint returns 404 on some Evolution API instances (including ours). To find a group JID when you don't have it:\n\n1. **Use `chat/findMessages` with an empty `where` filter** to pull recent messages across all chats, then extract unique `remoteJid` values containing `@g.us`.\n2. **Match by content**: Fetch messages from each group JID using `where.key.remoteJid` and match sender names or message text to identify the target group.\n3. **Known group JIDs** (store in `whatsapp-vip-contacts.json` as you discover them):\n - ps112 family group: `6591887488-1464061170@g.us`\n - Overcomers group: `6591444911-1614821287@g.us`\n - η₯žηš„εŽηΎŽ group: `6596756578-1405873840@g.us` (content-sharing, casual/broadcast, local log only)\n - Overcomers/TCians group: `120363230236878544@g.us` (church fellowship, PJ sends weekly attendance reports, routine attendance = local log only)\n\nExample search:\n```bash\ncurl -s -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"remoteJid\": \"6591444911-1614821287@g.us\"}}, \"limit\": 10, \"order\": \"desc\"}'\n```\n\n## Processing Image/Media Messages\n\nWhen an incoming WhatsApp message is an image (imageMessage), video, or other media with no text or only a caption:\n\n1. **Fetch conversation context** as usual with `wa_recent_messages.py`\n2. **Download the media** using the Evolution API:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"message\": {\"key\": {\"id\": \"MESSAGE_ID\", \"fromMe\": false, \"remoteJid\": \"GROUP_OR_CHAT_JID\"}}, \"convertToMp4\": false}' \\\n -o /tmp/wa_media_response.json\n ```\n Then decode: `python3 -c \"import json,base64; d=json.load(open('/tmp/wa_media_response.json')); open('/tmp/wa_media.jpg','wb').write(base64.b64decode(d['base64']))\"`\n3. **Analyze the image** using `vision_analyze` tool with `/tmp/wa_media.jpg` path and a context-aware question\n4. **Fallback if vision is unavailable**: Use `tesseract /tmp/wa_media.jpg stdout` for OCR, combined with pixel analysis (PIL/numpy) and conversation context to infer what the image contains\n5. **Album messages** (`albumMessage`): These bundle multiple images. Each image arrives as a separate `imageMessage` with a `messageAssociation.parentMessageKey` linking them to the album. Download each image separately. The `expectedImageCount` field tells how many images to expect\n6. **messageAssociation**: When images have `parentMessageKey` with `fromMe: true`, they are replying to a message Vincent sent. Use this for context\n\n### Audio / Voice Note Messages\nWhen an incoming WhatsApp message is `audioMessage` or a voice note (`ptt: true`), treat it like other media, but transcribe before intent classification when possible:\n\n1. **Fetch conversation context first**, especially in groups:\n ```bash\n python3 /opt/data/scripts/wa_recent_messages.py \"\" 50\n ```\n2. **Download the audio** with Evolution API, saving the response to a file:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n cat > /tmp/wa_audio_req.json <<'JSON'\n {\"message\":{\"key\":{\"id\":\"MESSAGE_ID\",\"fromMe\":false,\"remoteJid\":\"GROUP_OR_CHAT_JID\"}},\"convertToMp4\":false}\n JSON\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_audio_req.json -o /tmp/wa_audio_response.json\n python3 - <<'PY'\n import json, base64\n d=json.load(open('/tmp/wa_audio_response.json'))\n b=d.get('base64') or d.get('data',{}).get('base64')\n if ',' in b[:100]: b=b.split(',',1)[1]\n open('/tmp/wa_audio.ogg','wb').write(base64.b64decode(b))\n PY\n ```\n3. **Convert OGG/Opus to WAV**:\n ```bash\n ffmpeg -y -i /tmp/wa_audio.ogg -ar 16000 -ac 1 /tmp/wa_audio.wav\n ```\n4. **Transcribe fallback when Whisper is unavailable**: the container may not have `whisper`, `faster-whisper`, `torch`, or `speech_recognition` installed globally. A reliable lightweight fallback is a temp venv with SpeechRecognition and Google web speech:\n ```bash\n python3 -m venv /tmp/srvenv\n /tmp/srvenv/bin/pip install --quiet SpeechRecognition\n /tmp/srvenv/bin/python - <<'PY'\n import speech_recognition as sr\n r=sr.Recognizer()\n with sr.AudioFile('/tmp/wa_audio.wav') as source:\n audio=r.record(source)\n print(r.recognize_google(audio, language='en-SG'))\n PY\n ```\n5. **Check adjacent voice notes from the same sender**. WhatsApp voice notes are often sent in quick sequential chunks. Download and transcribe nearby `audioMessage` records from the same sender before deciding whether to notify Vincent.\n6. Use the transcript plus recent context to classify the message. For group meetup planning, extract what/when/where/who and notify Vincent if actionable, including transcript confidence.\n\n### Video Messages\n\nWhen an incoming WhatsApp message is `videoMessage` (with or without a caption):\n\n1. **Fetch conversation context** first, as with any media message: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n2. **Download the video** using Evolution API (same pattern as images/audio β€” save response to file, then decode):\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n cat > /tmp/wa_video_req.json <<'JSON'\n {\"message\":{\"key\":{\"id\":\"MESSAGE_ID\",\"fromMe\":false,\"remoteJid\":\"CONTACT_JID\"}},\"convertToMp4\":false}\n JSON\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_video_req.json -o /tmp/wa_video_response.json\n python3 - <<'PY'\n import json, base64\n d = json.load(open('/tmp/wa_video_response.json'))\n b = d.get('base64') or d.get('data', {}).get('base64', '')\n if ',' in b[:100]: b = b.split(',', 1)[1]\n open('/tmp/wa_video.mp4', 'wb').write(base64.b64decode(b))\n print(f\"Video saved: {len(base64.b64decode(b))} bytes\")\n PY\n ```\n3. **Extract representative frames** with ffmpeg (do NOT try to analyze video directly β€” vision tools work on images):\n ```bash\n # Frame at start\n ffmpeg -y -ss 0 -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame0.jpg\n # Frame at ~1/3 of the way (good for short videos)\n ffmpeg -y -ss 3 -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame3s.jpg\n # If video is longer (>15s), also extract at midpoint\n # ffprobe -v quiet -print_format json -show_format /tmp/wa_video.mp4 # get duration\n # ffmpeg -y -ss $HALF_DURATION -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame_mid.jpg\n ```\n4. **Analyze the frames** with `vision_analyze`. Ask specific questions about content: what objects/people/text are visible, what type of video this appears to be (screen recording, personal clip, forwarded meme, etc.), and any text or identifying details.\n5. **Use the caption** from `videoMessage.caption` if present (can be empty/null). Combine frame analysis + caption + conversation context to classify the message intent.\n6. **Transcription of audio in video** is generally not feasible with available tools. If the video likely contains important spoken content (e.g. voice memo style), note this limitation and flag it to Vincent.\n\n**Key points:**\n- Always use `-o` flag and file-based decode; videos can be 1-10MB and base64 responses will be even larger\n- `--max-time 25` on curl is essential for video downloads\n- Short casual clips (showing a key fob, a parking spot, quick view of surroundings) are usually casual/social β€” classify accordingly and don't over-escalate\n- If `vision_analyze` fails (400 on models that don't support images, or 429 rate limit), fall back to conversation context inference alone\n\n### Poll Messages\nWhen `pollUpdateMessage` appears in chat history (someone voted), the message contains a `pollCreationMessageKey` with the original poll's message `id`. To get the poll question and options, query:\n```bash\ncurl -s -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"id\": \"POLL_MESSAGE_ID\", \"remoteJid\": \"GROUP_JID\"}}, \"limit\": 1}'\n```\nThe response will have `pollCreationMessageV3` with `name` (the poll question) and `options` array. Always look up the poll when votes or poll references appear, since the poll text contains the event details.\n\n### `associatedChildMessage` (Motion Photos / Companion Media)\nWhatsApp motion photos and companion media arrive as `associatedChildMessage` type. These are short videos or supplementary images attached to a parent message via `messageAssociation.parentMessageKey`. The structure is:\n\n```\nmessage: {\n messageContextInfo: {\n messageAssociation: {\n associationType: 12,\n parentMessageKey: { id, fromMe, remoteJid }\n }\n },\n associatedChildMessage: {\n message: {\n videoMessage: { ... } // or imageMessage: { ... }\n }\n }\n}\n```\n\nKey points:\n- The `parentMessageKey.id` links back to the main message (the image/video they accompany)\n- When `parentMessageKey.fromMe: true`, the companion media is replying to a message Vincent sent\n- Captions may appear on the child's `videoMessage.caption` or `imageMessage.caption` inside the nested message, NOT at the top level\n- These often contain motion-photo short videos (3 seconds) or album companion images\n- Treat them as media attachments to the parent; don't process them as standalone messages requiring independent intent analysis\n\n### Sequential Image Sets (Non-Album)\nMultiple images sent rapidly by the same sender at similar timestamps may NOT be wrapped in an `albumMessage`. They appear as separate `imageMessage` entries. Always check for adjacent images from the same sender and download+OCR all of them together, as they often form a coherent multi-page set (e.g., route info, comparison tables, maps). Missing one can lose important context.\n\n### Key pitfalls\n- The Evolution API `getBase64FromMediaMessage` endpoint returns the full base64 payload (can be 100KB+ for images). Use `-o` flag to save to file, then parse with Python. Do NOT try to pipe it directly\n- `curl` with `--max-time 25` is essential; without it, large media responses can hang\n- The `albumMessage` type itself has no `fromMe` field; check each individual `imageMessage` instead\n- Image thumbnails in the webhook payload are truncated; always download the full image for analysis\n- **WhatsApp media URLs expire**: CDN download links (`mmg.whatsapp.net`) expire after roughly 24-48 hours. When `getBase64FromMediaMessage` returns a 400/404 error like \"Failed to fetch stream from https://mmg.whatsapp.net/...\", the media is no longer retrievable via the API. This means you CANNOT download adjacent/older voice notes, images, or documents for context if they're more than a day or two old. Fall back to conversation text context only. Fresh/incoming messages are almost always still downloadable.\n- **Voice note transcription reliability**: Short voice notes (<15s), especially in Singlish/Chinese-English mix common in Singapore contacts, often produce unreliable transcriptions via SpeechRecognition+Google. Always flag low-confidence transcriptions to Vincent rather than acting on potentially garbled text.\n- **Vision API failures**: `vision_analyze` can fail with 429 (rate limit) **or 400** (model doesn't support image input, e.g. glm-5.1 via ollama-cloud). Always have a fallback plan. When vision is unavailable, try tesseract OCR, then pixel analysis (PIL/numpy for brightness zones, text region detection via row-level dark-pixel density)\n- **OCR can fail on decorative/complex images**: Obituaries, memorial notices, religious flyers with ornate fonts, or dark-background images often defeat tesseract even with aggressive preprocessing (upscaling, thresholding, inverting, PSM mode variations). When OCR returns gibberish or empty, fall back to conversation context inference: use preceding/surrounding messages to determine what the image likely contains, then draft an appropriate response based on that context\n- **OCR on devotional/dark-background images**: These often defeat tesseract on full-image scans. Use a multi-strategy approach: (1) run tesseract on the full image first, (2) if text is sparse, crop into regions (top 15%, middle, bottom 15%), upscale 2-3x, enhance contrast (2-3x), and try both normal and inverted (`ImageOps.invert`) versions with `--psm 6`, (3) combine all partial OCR results. For Exodus/verse images, the title (e.g. \"WHEN MOSES' HANDS WERE\") is often in decorative font that OCR misses, while the verse body text is usually readable.\n- **Prioritize conversation context**: When processing group images, always fetch recent context FIRST (before attempting vision/OCR). If image analysis fails, the conversation history often makes the image's purpose clear enough to draft a meaningful reply\n- **DM conversation context is one-sided (fromMe only)**: The Evolution API `findMessages` endpoint often returns only Vincent's sent messages (`fromMe: true`) for DM chats, not the contact's incoming messages. This means `wa_recent_messages.py` can show only one side of the conversation. When processing a DM webhook, do NOT rely solely on fetched history for the contact's side. The webhook payload already contains the incoming message text. Use what the webhook gives you plus Vincent's side from the API. If you need the contact's recent messages, check if `fromMe: false` records exist; if none are returned, the API simply doesn't have them and you must work with what you have (webhook message + Vincent's replies for context).\n- **`findMessages` ignores `key.fromMe` filter**: Passing `\"key.fromMe\": false` in the `where` clause still returns all messages (including `fromMe: true`). To find messages from a specific sender, fetch results and filter client-side by checking `fromMe` in Python.\n- **`findMessages` caps at ~50 records per request**: Even with `limit: 200` or `limit: 500`, the API returns at most 50 records. When one party sends many consecutive messages, the other party's messages may not appear in the first 50 results. Use pagination or fetch multiple pages if you need older messages.\n- **Video frame extraction**: Use `ffmpeg -y -ss N -i video.mp4 -frames:v 1 -update 1 output.jpg` (the `-update 1` flag is needed; without it ffmpeg treats the output as an image sequence and fails). Extract frames at multiple timestamps (start, ~1/3, midpoint for longer videos) since a single frame may miss the key content.\n- **ffmpeg \"Invalid color space\" failure on WhatsApp videos**: Some WhatsApp H.264 videos have `color_space=reserved` and `color_primaries=reserved` metadata that ffmpeg cannot decode at all. Every codec/format/filter combination fails with \"Invalid color space\" / return code 234. This is NOT fixable with `-vf setparams`, `-color_trc`, `-colorspace`, `-color_primaries`, or `-vf format=` flags because the error occurs at the decoder level before any filter runs. **Do not waste time trying ffmpeg workarounds.** When this happens, fall back to: (1) conversation context inference (who sent it, what thread it's in, adjacent messages), (2) the jpegThumbnail embedded in the videoMessage (small but gives a rough idea; base64-encoded JPEG, typically 600-900 bytes, use `vision_analyze` on it if available), or (3) if all analysis fails, just classify based on surrounding text context.\n- **`wa_recent_messages.py` JSON output breaks `json.loads()`**: The script's stdout contains WhatsApp message content with control characters (emoji, CJK, deviceListMetadata binary blobs) that cause `json.loads()` to fail with `Invalid control character` or `Expecting ',' delimiter` errors. Even `strict=False` does not reliably fix it. **Reliable approach**: use `curl -o /tmp/wa_msgs.json` to save the Evolution API response directly to a file (the file write handles binary safely), then parse with `json.load(open('/tmp/wa_msgs.json'))` in Python. Example:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n curl -s --max-time 15 -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"remoteJid\": \"CONTACT_JID\"}}, \"limit\": 20}' -o /tmp/wa_msgs.json\n ```\n Then in Python: `data = json.load(open('/tmp/wa_msgs.json')); records = data['messages']['records']`\n\n## API Reference\n\nFor full API endpoints (media, contacts, location, buttons) β€” see [references/api.md](references/api.md).\n\n[Skill directory: /opt/data/skills/personal/whatsapp-evolution]\nResolve any relative paths in this skill (e.g. `scripts/foo.js`, `templates/config.yaml`) against that directory, then run them with the terminal tool using the absolute path.\n\n[This skill has supporting files:]\n- references/api.md -> /opt/data/skills/personal/whatsapp-evolution/references/api.md\n- scripts/process_queue.py -> /opt/data/skills/personal/whatsapp-evolution/scripts/process_queue.py\n- scripts/send_text.sh -> /opt/data/skills/personal/whatsapp-evolution/scripts/send_text.sh\n\nLoad any of these with skill_view(name=\"personal/whatsapp-evolution\", file_path=\"\"), or run scripts directly by absolute path (e.g. `node /opt/data/skills/personal/whatsapp-evolution/scripts/foo.js`).\n\nThe user has provided the following instruction alongside the skill invocation: Incoming WhatsApp webhook received.\n\nEvent: messages.upsert\nInstance: n8n-WA\nSender: Shalom Evonne\nSender JID: 120363418526182134@g.us\nFrom me: False\nMessage type: conversation\nMessage: {\n \"event\": \"messages.upsert\",\n \"instance\": \"n8n-WA\",\n \"data\": {\n \"key\": {\n \"remoteJid\": \"120363418526182134@g.us\",\n \"fromMe\": false,\n \"id\": \"3EB0FCA476B5401C85E5E9\",\n \"participant\": \"131503308673050@lid\",\n \"participantAlt\": \"6591887488@s.whatsapp.net\",\n \"addressingMode\": \"lid\"\n },\n \"pushName\": \"Shalom Evonne\",\n \"status\": \"DELIVERY_ACK\",\n \"message\": {\n \"conversation\": \"Good evening Shalom Family!\\n\\nTomorrow is Friday, Week Odd.\\n\\nNaomi is Schedule (G2/G3):\\n\\u2022 Morning Assembly: 8:00-8:25\\n\\u2022 CCE: 8:25-9:00\\n\\u2022 English Language: 9:00-9:35\\n\\u2022 Recess: 9:35-10:10\\n\\u2022 Recess: 10:10-10:45\\n\\u2022 Music: 10:45-11:20\\n\\u2022 Needlework: 11:20-11:55\\n\\u2022 Geography: 11:55-12:30\\n\\nDISMISSAL: 12:30 PM\\n\\nNo PE tomorrow. Have a blessed evening and see you bright and early tomorrow!\\n\\nWith love,\\nShalom Evonne\"\n },\n \"contextInfo\": null,\n \"messageType\": \"conversation\",\n \"messageTimestamp\": 1779361289,\n \"instanceId\": \"b2bfaa30-d5f1-4b22-b979-791892619414\",\n \"source\": \"web\"\n },\n \"destination\": \"https://jimmy.huper.asia/webhooks/whatsapp-incoming\",\n \"date_time\": \"2026-05-21T08:01:29.820Z\",\n \"sender\": \"6590054438@s.whatsapp.net\",\n \"server_url\": \"https://evolution.huper.asia\",\n \"apikey\": \"5E4C532223A0-460B-A749-16C584F56E74\"\n}\n\nPrimary goal: REDUCE NOISE. Stay silent in the main Telegram chat unless the message truly needs Vincent's immediate attention. Approval-worthy reply drafts should go to the jimmy-wa-drafts sidecar, NOT to the main Telegram chat.\n\nFollow the whatsapp-evolution skill and /opt/data/agent-state/memory/whatsapp-vip-contacts.json, with these stricter notification rules:\n\nABSOLUTE SILENCE RULES:\n- If fromMe is true: ignore. Do NOT call send_message. Final response local log only.\n- Ignore status/update/read-receipt events or payloads with no real message content.\n- Do NOT send main Telegram summaries for ordinary DMs, routine logistics, casual family messages, generic group events, group chatter, forwards, memes, devotional/religious shares, links, standalone codes, normal confirmations, or suggested replies.\n- Do NOT ask \"Want me to add this to Calendar?\" for WhatsApp group events unless Vincent is explicitly invited, named, responsible, or the event is time-sensitive and clearly relevant to him today.\n- Do NOT surface family messages just because they are family. Only surface if urgent, safety-related, school/pickup critical, money/legal/medical, or explicitly requires Vincent now.\n\nSIDEcar RULES:\n- If a reply may be useful but it is not urgent, POST it to https://jimmy-draft.huper.asia/drafts using the whatsapp-evolution Draft Approvals flow. Do NOT duplicate that draft/summary in the main Telegram chat.\n- Prayer and birthday replies should go to the sidecar when appropriate, not the main Telegram chat, unless the underlying situation is urgent/sensitive and Vincent needs immediate awareness.\n- Before drafting any reply or approval suggestion, fetch recent context: python3 /opt/data/scripts/wa_recent_messages.py \"120363418526182134@g.us\" 30. For active/ambiguous groups use 50.\n\nMAIN TELEGRAM ALERTS ARE ALLOWED ONLY FOR:\n1. VIP contacts marked always_call or always_notify when the message is important/actionable, or any VIP call event.\n2. Calls from family/VIP that appear missed/ringing and likely need immediate attention. Avoid repeated alerts for accepted/terminated duplicate call state updates.\n3. Emergencies, safety issues, medical issues, school pickup/child logistics requiring action now, legal/money/account/security matters, or time-sensitive commitments in the next few hours.\n4. Messages explicitly asking Vincent to call back, decide now, approve payment, provide OTP/code for an active task, or respond urgently.\n5. Pipeline errors or webhook processing failures.\n\nCALL RULES:\n- VIP contacts (e.g. Dn Jeff, Ng Whye Keong): call Vincent immediately outside quiet hours when actionably important or on ringing/missed call. During 1am-10am SGT, Telegram only unless life-and-death emergency.\n- Do not call for casual VIP confirmations or non-actionable courtesy messages.\n\nTONE/RULES FOR DRAFTS:\n- Proposed replies must be Christian and grace-based where appropriate: warm, humble, encouraging, non-judgmental, peace-making, not preachy.\n- Never use em dashes in drafted replies or approval messages. Use commas, periods, colons, or parentheses instead.\n- NEVER auto-reply directly on WhatsApp unless Vincent explicitly approves or future config marks that exact category pre-approved.\n\nDelivery rule:\n- For allowed main alerts only: call send_message(target=\"telegram\", message=...). Keep it concise.\n- For non-urgent reply suggestions: use the sidecar only.\n- For everything else: DO NOT call send_message. Keep final response as local-only log.\n", "timestamp": 1779361297.1647046, "anchor": true}, {"id": 75251, "role": "assistant", "content": "Local-only log: Routine group schedule update for tomorrow, no urgent action needed. No Telegram alert, no draft, no WhatsApp reply sent.", "timestamp": 1779361297.2074366}], "bookend_end": [], "messages_before": 0, "messages_after": 2}, {"session_id": "20260518_110108_894e3ff1", "when": "May 18, 2026 at 11:01 AM", "source": "webhook", "model": "gpt-5.5", "title": "WhatsApp School Schedule Reminder", "matched_role": "user", "match_message_id": 67331, "snippet": "...draft_text\": \"Thanks >>>Karen<<<, will miss you both β€” praying for a good time too πŸ™\",\n \"extras\": {\n \"group_jid\": \"120363351787895353@g.us\",\n \"sender_name\": \">>>Karen<<< >>>Tan<<<\",\n \"sender_jid\": \">>>6597630553<<<@s.whatsapp.net\",\n \"category\": \">>>prayer<<>>blessing<<<\",\n \"original_message\": \"Happy >>>cycling<<<, to those who...", "bookend_start": [], "messages": [{"id": 67331, "role": "user", "content": "[IMPORTANT: The user has invoked the \"whatsapp-evolution\" skill, indicating they want you to follow its instructions. The full skill content is loaded below.]\n\n---\nname: whatsapp-evolution\ndescription: WhatsApp messaging via Evolution API β€” send, receive, analyze, and auto-reply. Use when processing an incoming WhatsApp message delivered via the webhook, replying on Vincent's behalf, liaising with contacts, sending messages, or handling WhatsApp-related tasks. Covers intent analysis, auto-reply rules, escalation (VIP β†’ phone call), conversation context fetching, and draft approvals via the jimmy-wa-drafts sidecar.\n---\n\n# WhatsApp via Evolution API\n\n## Credentials\nStored in `/opt/data/secrets/.whatsapp-evolution-credentials`:\n- `EVOLUTION_API_URL`, `EVOLUTION_API_KEY`, `EVOLUTION_INSTANCE`\n\nLoad with: `export $(cat .whatsapp-evolution-credentials | xargs)`\n\n## Sending Messages (direct, no approval)\n\n```bash\nexport $(cat .whatsapp-evolution-credentials | xargs)\ncurl -s -X POST \"${EVOLUTION_API_URL}/message/sendText/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"number\": \"6591234567\", \"text\": \"Hello!\"}'\n```\n\n- Phone format: country code, no `+` (e.g. `6590054438`)\n- Groups: use JID `{groupid}@g.us`\n- For media/location/contacts β€” see [references/api.md](references/api.md)\n\nUse direct send only when Vincent has explicitly commanded/approved the exact send, or for pre-approved auto-replies/follow-ups. For any reply that needs Vincent's review, use the sidecar (next section).\n\n### Sending trip/camp details from retrieved emails\n\nWhen Vincent explicitly asks to send extracted school trip/camp details to a child or family member, send directly. Keep it practical and grounded:\n- Include dates, reporting/pick-up times, flights, venue/itinerary highlights, deadlines, forms, and safety/medication notes when found.\n- If an official packing list is missing, say that clearly. Do not invent one. You may add a short β€œbased on itinerary/weather, prepare…” list, clearly labeled as practical guidance rather than official school instructions.\n- Avoid long explanations about the search process in WhatsApp. Send the useful details.\n\n### Sending a local image/file as media\n\nFor user-approved media sends, base64-encode the local file and call `sendMedia`:\n\n```bash\nexport $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\npython3 - <<'PY'\nimport base64, json\npath = '/tmp/file.png'\npayload = {\n 'number': '6591234567',\n 'mediatype': 'image',\n 'mimetype': 'image/png',\n 'caption': 'Caption here',\n 'media': base64.b64encode(open(path, 'rb').read()).decode(),\n 'fileName': 'file.png',\n}\nopen('/tmp/wa_media_payload.json', 'w').write(json.dumps(payload))\nPY\ncurl -s --max-time 60 -X POST \"${EVOLUTION_API_URL}/message/sendMedia/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_media_payload.json\n```\n\nWhen sending QR tickets or passes, verify the crop/image contains the full QR code and the intended attendee name before sending; the QR screenshot is effectively the redeemable ticket.\n\n## Fetching Conversation Context\n\nBefore drafting or proposing any reply, always fetch recent chat history for context:\n\n```bash\npython3 /opt/data/scripts/wa_recent_messages.py \"6591234567@s.whatsapp.net\" 30\n```\n\nUse `limit=30` by default. For active group chats or ambiguous situations, fetch up to 50 messages. Read enough context to understand who is being discussed, current tone, previous commitments, and whether Vincent has already replied. The script returns newest messages first (descending order).\n\n## Draft Approvals β€” `jimmy-wa-drafts` sidecar\n\nWhen you need Vincent to approve a WhatsApp reply, POST the draft to the sidecar. It owns the entire approval flow: Telegram message with 4 inline buttons, Telegram WebApp for Edit/Context, sending to WhatsApp via Evolution API on Approve.\n\n### Posting a draft\n\nAlways include an `extras` object with the original WA context. The sidecar passes `extras` through verbatim if Vincent later clicks \"Context\" β€” without it, your redraft will be flying blind.\n\n```bash\ncat > /tmp/wa_draft.json <<'JSON'\n{\n \"contact\": \"120363351787895353@g.us\",\n \"contact_name\": \"Cycling group\",\n \"summary\": \"Karen sent a blessing over the bike ride; she's not joining\",\n \"draft_text\": \"Thanks Karen, will miss you both β€” praying for a good time too πŸ™\",\n \"extras\": {\n \"group_jid\": \"120363351787895353@g.us\",\n \"sender_name\": \"Karen Tan\",\n \"sender_jid\": \"6597630553@s.whatsapp.net\",\n \"category\": \"prayer/blessing\",\n \"original_message\": \"Happy cycling, to those who can make it… Dn Jeff and I had made plans for this PM so we won't be able to join y'all. Praying the Lord's favor, protection and good-weather blessings be upon y'all and your route!\",\n \"extracted\": {\n \"what\": \"Cycling from East Coast Park to Changi Village, then dinner\",\n \"when\": \"Today, 4:00 PM\",\n \"where\": \"East Coast Park to Changi Village\",\n \"who\": \"Group members who voted yes for cycling/dinner or dinner\"\n },\n \"confidence\": \"high\",\n \"why_appropriate\": \"Multiple group messages are blessings/wishes; Vincent has not replied; tone matches grace-based norm.\",\n \"wa_message_id\": \"ABC123XYZ\"\n }\n}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_draft.json\n```\n\nResponse: `{\"draft_id\":\"abc123def\",\"telegram_message_id\":42}`\n\nWhat to put in `extras` (use whatever makes sense for the message type):\n\n- **Always**: `original_message` (verbatim WA text), `sender_name`, `category` (e.g. `prayer`, `birthday`, `event`, `dm-reply`, `enquiry`)\n- **Group messages**: `group_jid`, `sender_jid`\n- **DMs**: `sender_jid`\n- **Strongly recommended (drives the rich Telegram render)**: `extracted` (dict of who/what/when/where), `confidence` (`high`/`medium`/`low`), `why_appropriate` (one-line reasoning shown to Vincent so he can decide)\n- **Optional**: `wa_message_id`, `tone_hint` (e.g. `warm-grace-based`), `sender_phone` (auto-derived from `sender_jid` if omitted)\n\nVincent receives a Telegram message in the `@jimmy_WA_bot` chat with 4 inline buttons. The sidecar handles every interaction; **you do not track replies, callbacks, or message IDs**. Just POST and move on.\n\n| Button | Behavior |\n|--------|----------|\n| **Approve** | Sidecar sends draft to WhatsApp |\n| **Edit** | Opens a Telegram WebApp with draft pre-filled; Vincent edits + sends; sidecar pushes to WhatsApp |\n| **Context** | Opens a Telegram WebApp with empty textarea + the original WA message visible; Vincent provides context. (When `HERMES_REDRAFT_WEBHOOK` is wired in the sidecar's env, this will POST back to Jimmy for a redraft. Until then, the context is just stored on the draft.) |\n| **Pass** | Sidecar deletes the draft |\n\n### JSON escaping when `draft_text` contains quotes\n\nSingle quotes (`It's`, `God's`) break shell interpolation. Write the JSON to a temp file first:\n\n```bash\ncat > /tmp/wa_draft.json <<'JSON'\n{\"contact\":\"6591234567@s.whatsapp.net\",\"contact_name\":\"John\",\"summary\":\"…\",\"draft_text\":\"It's nice to hear from you!\"}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_draft.json\n```\n\n### Networking\n\n`https://jimmy-draft.huper.asia` is the public HTTPS endpoint of the sidecar (TLS-terminated by Traefik on ubuntu-server). HTTPS is required because Hermes blocks plain-HTTP commands via tirith. The internal Docker DNS hostname http://jimmy-wa-v64mqe:8080 also works for non-shell calls but the agent is forbidden from using it.\n\n### Sidecar status & logs\n\n- Source: (auto-deploys on push)\n- Public URL: (`/health` for ping)\n- Container: `jimmy-wa-v64mqe.1.` on ubuntu-server\n- Bot: `@jimmy_WA_bot`\n\n## Handling a Redraft Trigger (`wa-redraft` webhook)\n\nWhen Vincent clicks the **Context** button on a draft, the sidecar:\n1. Edits the original Telegram message to \"πŸ”„ Adding context for X… agent is redrafting.\" (removes the old buttons so Vincent can't approve the now-stale text)\n2. POSTs to your `wa-redraft` webhook (`http://localhost:8644/webhooks/wa-redraft`) with the original draft fields plus Vincent's `context` instructions and the `extras` you stored when you first created the draft.\n\nWhen you receive a `wa-redraft` activation, the prompt template will lay out the fields. Your job:\n\n1. Read Vincent's `context` carefully β€” that's the steering signal. The `previous_draft` is just history, not necessarily a starting point.\n2. Re-fetch recent WA history if the original context isn't enough (`python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`). Use `extras.group_jid` for groups or `contact` for DMs.\n3. Write a NEW draft that addresses Vincent's instructions while staying faithful to the original WA context (in `extras.original_message`, `extras.extracted`).\n4. POST the new draft back to the sidecar **with the same `draft_id`** so the existing draft is updated, not duplicated:\n\n```bash\ncat > /tmp/wa_redraft.json <\",\n \"contact\": \"\",\n \"contact_name\": \"\",\n \"summary\": \"\",\n \"draft_text\": \"\",\n \"extras\": \n}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_redraft.json\n```\n\nThe sidecar will send a NEW Telegram message with 4 fresh buttons. Vincent can Approve, Edit, Context-again (loop), or Pass.\n\nDo **not** send to WhatsApp directly during a redraft β€” the approval flow is the sidecar's job.\n\n## Incoming Message Processing\n\n### Delivery Pipeline (current)\n1. Evolution API POSTs each `MESSAGES_UPSERT` webhook directly to Hermes at `http://localhost:8644/webhooks/whatsapp-incoming` (proxied via `jimmy.huper.asia/webhooks/whatsapp-incoming`)\n2. Hermes webhook route (`whatsapp-incoming`, dynamic subscription) validates and spawns an agent\n3. Agent loads `whatsapp-evolution` skill, follows the rules below\n4. Webhook subscription uses `deliver: log` because webhooks do not support `deliver: local`; the agent explicitly calls `send_message(target=\"telegram\", ...)` only for important/actionable/VIP/event/birthday/prayer/error items, OR posts a draft to the sidecar for approval items.\n\n**No more OpenClaw, no Python webhook server, no port 5001, no keepalive cron.** If messages aren't being processed, check: nginx proxy to port 8644, `hermes webhook list` for the subscription, Evolution API webhook URL setting.\n\n## Contact Rules & Escalation\n\nConfig: `memory/whatsapp-vip-contacts.json`\n\n### Priority Levels\n\n| Priority | Action | Example |\n|----------|--------|---------|\n| **VIP (call-always)** | **CALL Vincent immediately via Twilio without asking.** Do NOT ask for approval first. During quiet hours (1am-10am SGT) suppress call and use Telegram notification only, unless life-and-death emergency. Match VIPs by both phone JID (`659...@s.whatsapp.net`) and LID JID (`...@lid`) when present, because recent chats may arrive with `key.remoteJid` as an LID and the phone number only in `key.remoteJidAlt`. | Dn Jeff (+6597630553) |\n| **Family** | Always notify Vincent on Telegram | ps112 group, family members |\n| **Auto-reply** | Reply based on context file, notify Vincent | Repair shops |\n| **Normal** | Analyze intent, reply if appropriate, notify if important | Everyone else |\n| **Broadcast/Spam** | Ignore silently | Forwarded job posts, chain messages |\n\n### Quiet Hours\n- 1am-10am SGT (17:00-02:00 UTC): avoid phone calls, including for Dn Jeff / VIP call-always contacts\n- **Exception:** life-and-death emergencies always call regardless of quiet hours\n- During quiet hours, use Telegram notification only and mention that a call was suppressed due to quiet hours\n\n## Making a Call via Twilio\n\nCredentials: `/opt/data/secrets/.twilio-credentials-new` (`TWILIO_ACCOUNT_SID`, `TWILIO_AUTH_TOKEN`, `TWILIO_PHONE_NUMBER`). From: `+16614809287`, Vincent: `+6590054438`.\n\nPreferred script: `/opt/data/scripts/call_vincent.py`, now using the Twilio Python SDK baked into the container.\n\n```bash\npython3 /opt/data/scripts/call_vincent.py \"Hello Vincent, this is Jimmy. Urgent summary here.\"\n```\n\n## Action Rules\n\n### Default: DO NOT auto-reply directly, and do not spam main Telegram\nCollect intent and stay local unless something truly needs Vincent's immediate attention.\n\n**When you need Vincent to approve a reply, POST a draft to the sidecar (see \"Draft Approvals\" above).** The sidecar handles the Telegram notification with buttons, the WebApp for editing, and sending to WhatsApp on approve. Do **not** duplicate that draft or summary in the main Telegram chat.\n\nMain Telegram WhatsApp summaries are muted by default. Only use `send_message(target=\"telegram\", ...)` for urgent/important/actionable items that need Vincent now: VIP actionable messages/calls, missed/ringing family calls, emergencies, safety/medical/school-pickup issues, money/legal/account/security matters, explicit call-back/decision-now requests, or pipeline errors. Routine logistics, casual family pings, generic group events, links, standalone codes, normal confirmations, and suggested replies should be local-only or sidecar-only.\n\nDo NOT send WhatsApp replies directly unless Vincent explicitly approves them or the category is pre-approved.\n\n### Tone (when proposing or sending replies)\n- Casual, direct, no sign-offs (no \"Thanks & Regards\", no \"- Vincent\")\n- Ask questions instead of giving instructions\n- Keep it natural β€” like texting a friend\n- Christian and grace-based: warm, humble, encouraging, non-judgmental, peace-making\n- Never use em dashes in drafted replies or approval messages. Use commas, periods, colons, or parentheses instead.\n- For prayer/birthday contexts, use gentle faith-filled language; avoid sounding preachy, performative, or overly formal\n\n### What to Do With Each Message\n\n**DMs (personal chats):**\n- Analyze intent\n- Notify Vincent on Telegram (via `send_message(target=\"telegram\", ...)`) only if important/actionable/time-sensitive, requires a decision, comes from a VIP contact\n- If you have a useful suggested reply, POST it as a draft to the sidecar instead of a Telegram notification β€” Vincent gets the buttons inline\n- Casual/social/non-actionable DMs β†’ local log only, no Telegram notification, no draft\n- Do NOT reply unless Vincent says to\n- VIP contacts β†’ **call Vincent immediately without asking**. Do NOT post a draft. Just call. During quiet hours (1am-10am SGT), suppress the call and send Telegram notification only, unless life-and-death emergency.\n\n**Group chats β€” events/meetups:**\n- If someone posts about an event, gathering, meetup, dinner, etc:\n 1. Extract: what, when, where, who\n 2. Notify Vincent on Telegram with details (no draft needed unless you also want to suggest an RSVP reply)\n 3. Ask: \"Want me to add this to Google Calendar and set a reminder?\"\n\n**Group chats β€” prayer requests / praying for others:**\n- If someone asks for prayer, shares a burden, illness, grief, struggle, thanksgiving, or asks the group to pray:\n 1. Fetch recent group context first: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n 2. Identify who/what to pray for and any sensitive details\n 3. Draft a short, grace-based Christian reply/prayer\n 4. POST it to the sidecar with `summary` describing the request and `draft_text` being the prayer\n 5. Do NOT post to WhatsApp unless Vincent approves\n\n**Group chats β€” birthday greetings:**\n- If you see birthday wishes for someone in a group:\n 1. Fetch recent group context first: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n 2. Note who the birthday person is and relationship/context if inferable\n 3. Draft a warm, grace-based Christian birthday greeting\n 4. POST it to the sidecar\n 5. Do NOT post to WhatsApp unless Vincent approves\n\n**Group chats β€” general:**\n- Casual banter, memes, forwards, broadcast spam β†’ ignore silently\n- Religious content shares β†’ ignore unless specifically addressed to Vincent\n\n### When to Notify Vincent (Telegram) vs Post a Draft\n\n**Use `send_message(target=\"telegram\", ...)` sparingly** for plain notifications only when the message needs Vincent's immediate attention: VIP actionable DMs/calls, emergencies, school pickup/child logistics requiring action now, money/legal/account/security matters, explicit call-back/decision-now requests, or pipeline errors.\n\n**Use the sidecar (POST `/drafts`)** whenever you have a suggested reply for Vincent to approve: prayer responses, birthday greetings, DM replies, group RSVPs. Do not also send the same summary to the main Telegram chat.\n\nDo **not** notify or draft for casual/social/non-actionable DMs, routine logistics, read receipts/status updates, religious shares not addressed to Vincent, memes, forwarded/broadcast/spam, generic group events, standalone codes, normal confirmations, or general group chatter. Family messages are not automatically main-chat alerts unless urgent, safety-related, school/pickup critical, or explicitly requiring Vincent now.\n\n## Intent Categories\n\nWhen analyzing incoming messages, classify as:\n\n1. **Urgent** β€” time-sensitive, emergencies β†’ call Vincent outside quiet hours; during quiet hours use Telegram only unless life-and-death\n2. **Important** β€” decisions, money, appointments, anything consequential β†’ call Vincent outside quiet hours; during quiet hours notify on Telegram only unless life-threatening\n3. **Request** β€” asking Vincent to do something β†’ **notify Vincent** (Telegram) or post a draft if you have a safe reply\n4. **Enquiry** β€” asking for info, pricing, availability β†’ auto-reply if context available, otherwise post a draft\n5. **Confirmation** β€” confirming plans, bookings β†’ acknowledge and notify Vincent\n6. **Casual/Social** β€” greetings, memes, forwards β†’ usually ignore or react\n7. **Broadcast/Spam** β€” forwarded job posts, chain messages β†’ ignore\n\n## Finding Individual Contacts and JIDs\n\nWhen Vincent asks to monitor or prioritize a specific WhatsApp contact and only gives a name:\n\n1. Query contacts and chats, then search both display names and serialized message metadata:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n python3 - <<'PY'\n import os, requests, json\n base=os.environ['EVOLUTION_API_URL']; inst=os.environ['EVOLUTION_INSTANCE']; key=os.environ['EVOLUTION_API_KEY']\n headers={'apikey':key,'Content-Type':'application/json'}\n terms=['whye','keong'] # replace with lowercase search terms\n for ep in ['findContacts','findChats']:\n r=requests.post(f'{base}/chat/{ep}/{inst}', headers=headers, json={}, timeout=30)\n print('endpoint', ep, 'status', r.status_code)\n for obj in (r.json() if r.ok else []):\n if any(t in json.dumps(obj, ensure_ascii=False).lower() for t in terms):\n print(json.dumps(obj, ensure_ascii=False)[:1000])\n PY\n ```\n2. If `findChats` shows `remoteJid` ending in `@lid`, inspect `lastMessage.key.remoteJidAlt` for the phone JID. Store both in `memory/whatsapp-vip-contacts.json` when configuring VIP/call rules.\n3. Verify the number exists with `/chat/whatsappNumbers/{instance}` before finalizing.\n4. Validate the JSON config after editing: `python3 -m json.tool /opt/data/agent-state/memory/whatsapp-vip-contacts.json >/tmp/wa_vip_validated.json`.\n\n## Finding Group JIDs\n\nThe `group/fetchGroups` endpoint returns 404 on some Evolution API instances (including ours). To find a group JID when you don't have it:\n\n1. **Use `chat/findMessages` with an empty `where` filter** to pull recent messages across all chats, then extract unique `remoteJid` values containing `@g.us`.\n2. **Match by content**: Fetch messages from each group JID using `where.key.remoteJid` and match sender names or message text to identify the target group.\n3. **Known group JIDs** (store in `whatsapp-vip-contacts.json` as you discover them):\n - ps112 family group: `6591887488-1464061170@g.us`\n - Overcomers group: `6591444911-1614821287@g.us`\n - η₯žηš„εŽηΎŽ group: `6596756578-1405873840@g.us` (content-sharing, casual/broadcast, local log only)\n - Overcomers/TCians group: `120363230236878544@g.us` (church fellowship, PJ sends weekly attendance reports, routine attendance = local log only)\n\nExample search:\n```bash\ncurl -s -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"remoteJid\": \"6591444911-1614821287@g.us\"}}, \"limit\": 10, \"order\": \"desc\"}'\n```\n\n## Processing Image/Media Messages\n\nWhen an incoming WhatsApp message is an image (imageMessage), video, or other media with no text or only a caption:\n\n1. **Fetch conversation context** as usual with `wa_recent_messages.py`\n2. **Download the media** using the Evolution API:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"message\": {\"key\": {\"id\": \"MESSAGE_ID\", \"fromMe\": false, \"remoteJid\": \"GROUP_OR_CHAT_JID\"}}, \"convertToMp4\": false}' \\\n -o /tmp/wa_media_response.json\n ```\n Then decode: `python3 -c \"import json,base64; d=json.load(open('/tmp/wa_media_response.json')); open('/tmp/wa_media.jpg','wb').write(base64.b64decode(d['base64']))\"`\n3. **Analyze the image** using `vision_analyze` tool with `/tmp/wa_media.jpg` path and a context-aware question\n4. **Fallback if vision is unavailable**: Use `tesseract /tmp/wa_media.jpg stdout` for OCR, combined with pixel analysis (PIL/numpy) and conversation context to infer what the image contains\n5. **Album messages** (`albumMessage`): These bundle multiple images. Each image arrives as a separate `imageMessage` with a `messageAssociation.parentMessageKey` linking them to the album. Download each image separately. The `expectedImageCount` field tells how many images to expect\n6. **messageAssociation**: When images have `parentMessageKey` with `fromMe: true`, they are replying to a message Vincent sent. Use this for context\n\n### Audio / Voice Note Messages\nWhen an incoming WhatsApp message is `audioMessage` or a voice note (`ptt: true`), treat it like other media, but transcribe before intent classification when possible:\n\n1. **Fetch conversation context first**, especially in groups:\n ```bash\n python3 /opt/data/scripts/wa_recent_messages.py \"\" 50\n ```\n2. **Download the audio** with Evolution API, saving the response to a file:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n cat > /tmp/wa_audio_req.json <<'JSON'\n {\"message\":{\"key\":{\"id\":\"MESSAGE_ID\",\"fromMe\":false,\"remoteJid\":\"GROUP_OR_CHAT_JID\"}},\"convertToMp4\":false}\n JSON\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_audio_req.json -o /tmp/wa_audio_response.json\n python3 - <<'PY'\n import json, base64\n d=json.load(open('/tmp/wa_audio_response.json'))\n b=d.get('base64') or d.get('data',{}).get('base64')\n if ',' in b[:100]: b=b.split(',',1)[1]\n open('/tmp/wa_audio.ogg','wb').write(base64.b64decode(b))\n PY\n ```\n3. **Convert OGG/Opus to WAV**:\n ```bash\n ffmpeg -y -i /tmp/wa_audio.ogg -ar 16000 -ac 1 /tmp/wa_audio.wav\n ```\n4. **Transcribe fallback when Whisper is unavailable**: the container may not have `whisper`, `faster-whisper`, `torch`, or `speech_recognition` installed globally. A reliable lightweight fallback is a temp venv with SpeechRecognition and Google web speech:\n ```bash\n python3 -m venv /tmp/srvenv\n /tmp/srvenv/bin/pip install --quiet SpeechRecognition\n /tmp/srvenv/bin/python - <<'PY'\n import speech_recognition as sr\n r=sr.Recognizer()\n with sr.AudioFile('/tmp/wa_audio.wav') as source:\n audio=r.record(source)\n print(r.recognize_google(audio, language='en-SG'))\n PY\n ```\n5. **Check adjacent voice notes from the same sender**. WhatsApp voice notes are often sent in quick sequential chunks. Download and transcribe nearby `audioMessage` records from the same sender before deciding whether to notify Vincent.\n6. Use the transcript plus recent context to classify the message. For group meetup planning, extract what/when/where/who and notify Vincent if actionable, including transcript confidence.\n\n### Video Messages\n\nWhen an incoming WhatsApp message is `videoMessage` (with or without a caption):\n\n1. **Fetch conversation context** first, as with any media message: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n2. **Download the video** using Evolution API (same pattern as images/audio β€” save response to file, then decode):\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n cat > /tmp/wa_video_req.json <<'JSON'\n {\"message\":{\"key\":{\"id\":\"MESSAGE_ID\",\"fromMe\":false,\"remoteJid\":\"CONTACT_JID\"}},\"convertToMp4\":false}\n JSON\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_video_req.json -o /tmp/wa_video_response.json\n python3 - <<'PY'\n import json, base64\n d = json.load(open('/tmp/wa_video_response.json'))\n b = d.get('base64') or d.get('data', {}).get('base64', '')\n if ',' in b[:100]: b = b.split(',', 1)[1]\n open('/tmp/wa_video.mp4', 'wb').write(base64.b64decode(b))\n print(f\"Video saved: {len(base64.b64decode(b))} bytes\")\n PY\n ```\n3. **Extract representative frames** with ffmpeg (do NOT try to analyze video directly β€” vision tools work on images):\n ```bash\n # Frame at start\n ffmpeg -y -ss 0 -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame0.jpg\n # Frame at ~1/3 of the way (good for short videos)\n ffmpeg -y -ss 3 -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame3s.jpg\n # If video is longer (>15s), also extract at midpoint\n # ffprobe -v quiet -print_format json -show_format /tmp/wa_video.mp4 # get duration\n # ffmpeg -y -ss $HALF_DURATION -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame_mid.jpg\n ```\n4. **Analyze the frames** with `vision_analyze`. Ask specific questions about content: what objects/people/text are visible, what type of video this appears to be (screen recording, personal clip, forwarded meme, etc.), and any text or identifying details.\n5. **Use the caption** from `videoMessage.caption` if present (can be empty/null). Combine frame analysis + caption + conversation context to classify the message intent.\n6. **Transcription of audio in video** is generally not feasible with available tools. If the video likely contains important spoken content (e.g. voice memo style), note this limitation and flag it to Vincent.\n\n**Key points:**\n- Always use `-o` flag and file-based decode; videos can be 1-10MB and base64 responses will be even larger\n- `--max-time 25` on curl is essential for video downloads\n- Short casual clips (showing a key fob, a parking spot, quick view of surroundings) are usually casual/social β€” classify accordingly and don't over-escalate\n- If `vision_analyze` fails (400 on models that don't support images, or 429 rate limit), fall back to conversation context inference alone\n\n### Poll Messages\nWhen `pollUpdateMessage` appears in chat history (someone voted), the message contains a `pollCreationMessageKey` with the original poll's message `id`. To get the poll question and options, query:\n```bash\ncurl -s -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"id\": \"POLL_MESSAGE_ID\", \"remoteJid\": \"GROUP_JID\"}}, \"limit\": 1}'\n```\nThe response will have `pollCreationMessageV3` with `name` (the poll question) and `options` array. Always look up the poll when votes or poll references appear, since the poll text contains the event details.\n\n### `associatedChildMessage` (Motion Photos / Companion Media)\nWhatsApp motion photos and companion media arrive as `associatedChildMessage` type. These are short videos or supplementary images attached to a parent message via `messageAssociation.parentMessageKey`. The structure is:\n\n```\nmessage: {\n messageContextInfo: {\n messageAssociation: {\n associationType: 12,\n parentMessageKey: { id, fromMe, remoteJid }\n }\n },\n associatedChildMessage: {\n message: {\n videoMessage: { ... } // or imageMessage: { ... }\n }\n }\n}\n```\n\nKey points:\n- The `parentMessageKey.id` links back to the main message (the image/video they accompany)\n- When `parentMessageKey.fromMe: true`, the companion media is replying to a message Vincent sent\n- Captions may appear on the child's `videoMessage.caption` or `imageMessage.caption` inside the nested message, NOT at the top level\n- These often contain motion-photo short videos (3 seconds) or album companion images\n- Treat them as media attachments to the parent; don't process them as standalone messages requiring independent intent analysis\n\n### Sequential Image Sets (Non-Album)\nMultiple images sent rapidly by the same sender at similar timestamps may NOT be wrapped in an `albumMessage`. They appear as separate `imageMessage` entries. Always check for adjacent images from the same sender and download+OCR all of them together, as they often form a coherent multi-page set (e.g., route info, comparison tables, maps). Missing one can lose important context.\n\n### Key pitfalls\n- The Evolution API `getBase64FromMediaMessage` endpoint returns the full base64 payload (can be 100KB+ for images). Use `-o` flag to save to file, then parse with Python. Do NOT try to pipe it directly\n- `curl` with `--max-time 25` is essential; without it, large media responses can hang\n- The `albumMessage` type itself has no `fromMe` field; check each individual `imageMessage` instead\n- Image thumbnails in the webhook payload are truncated; always download the full image for analysis\n- **WhatsApp media URLs expire**: CDN download links (`mmg.whatsapp.net`) expire after roughly 24-48 hours. When `getBase64FromMediaMessage` returns a 400/404 error like \"Failed to fetch stream from https://mmg.whatsapp.net/...\", the media is no longer retrievable via the API. This means you CANNOT download adjacent/older voice notes, images, or documents for context if they're more than a day or two old. Fall back to conversation text context only. Fresh/incoming messages are almost always still downloadable.\n- **Voice note transcription reliability**: Short voice notes (<15s), especially in Singlish/Chinese-English mix common in Singapore contacts, often produce unreliable transcriptions via SpeechRecognition+Google. Always flag low-confidence transcriptions to Vincent rather than acting on potentially garbled text.\n- **Vision API failures**: `vision_analyze` can fail with 429 (rate limit) **or 400** (model doesn't support image input, e.g. glm-5.1 via ollama-cloud). Always have a fallback plan. When vision is unavailable, try tesseract OCR, then pixel analysis (PIL/numpy for brightness zones, text region detection via row-level dark-pixel density)\n- **OCR can fail on decorative/complex images**: Obituaries, memorial notices, religious flyers with ornate fonts, or dark-background images often defeat tesseract even with aggressive preprocessing (upscaling, thresholding, inverting, PSM mode variations). When OCR returns gibberish or empty, fall back to conversation context inference: use preceding/surrounding messages to determine what the image likely contains, then draft an appropriate response based on that context\n- **OCR on devotional/dark-background images**: These often defeat tesseract on full-image scans. Use a multi-strategy approach: (1) run tesseract on the full image first, (2) if text is sparse, crop into regions (top 15%, middle, bottom 15%), upscale 2-3x, enhance contrast (2-3x), and try both normal and inverted (`ImageOps.invert`) versions with `--psm 6`, (3) combine all partial OCR results. For Exodus/verse images, the title (e.g. \"WHEN MOSES' HANDS WERE\") is often in decorative font that OCR misses, while the verse body text is usually readable.\n- **Prioritize conversation context**: When processing group images, always fetch recent context FIRST (before attempting vision/OCR). If image analysis fails, the conversation history often makes the image's purpose clear enough to draft a meaningful reply\n- **DM conversation context is one-sided (fromMe only)**: The Evolution API `findMessages` endpoint often returns only Vincent's sent messages (`fromMe: true`) for DM chats, not the contact's incoming messages. This means `wa_recent_messages.py` can show only one side of the conversation. When processing a DM webhook, do NOT rely solely on fetched history for the contact's side. The webhook payload already contains the incoming message text. Use what the webhook gives you plus Vincent's side from the API. If you need the contact's recent messages, check if `fromMe: false` records exist; if none are returned, the API simply doesn't have them and you must work with what you have (webhook message + Vincent's replies for context).\n- **`findMessages` ignores `key.fromMe` filter**: Passing `\"key.fromMe\": false` in the `where` clause still returns all messages (including `fromMe: true`). To find messages from a specific sender, fetch results and filter client-side by checking `fromMe` in Python.\n- **`findMessages` caps at ~50 records per request**: Even with `limit: 200` or `limit: 500`, the API returns at most 50 records. When one party sends many consecutive messages, the other party's messages may not appear in the first 50 results. Use pagination or fetch multiple pages if you need older messages.\n- **Video frame extraction**: Use `ffmpeg -y -ss N -i video.mp4 -frames:v 1 -update 1 output.jpg` (the `-update 1` flag is needed; without it ffmpeg treats the output as an image sequence and fails). Extract frames at multiple timestamps (start, ~1/3, midpoint for longer videos) since a single frame may miss the key content.\n- **ffmpeg \"Invalid color space\" failure on WhatsApp videos**: Some WhatsApp H.264 videos have `color_space=reserved` and `color_primaries=reserved` metadata that ffmpeg cannot decode at all. Every codec/format/filter combination fails with \"Invalid color space\" / return code 234. This is NOT fixable with `-vf setparams`, `-color_trc`, `-colorspace`, `-color_primaries`, or `-vf format=` flags because the error occurs at the decoder level before any filter runs. **Do not waste time trying ffmpeg workarounds.** When this happens, fall back to: (1) conversation context inference (who sent it, what thread it's in, adjacent messages), (2) the jpegThumbnail embedded in the videoMessage (small but gives a rough idea; base64-encoded JPEG, typically 600-900 bytes, use `vision_analyze` on it if available), or (3) if all analysis fails, just classify based on surrounding text context.\n- **`wa_recent_messages.py` JSON output breaks `json.loads()`**: The script's stdout contains WhatsApp message content with control characters (emoji, CJK, deviceListMetadata binary blobs) that cause `json.loads()` to fail with `Invalid control character` or `Expecting ',' delimiter` errors. Even `strict=False` does not reliably fix it. **Reliable approach**: use `curl -o /tmp/wa_msgs.json` to save the Evolution API response directly to a file (the file write handles binary safely), then parse with `json.load(open('/tmp/wa_msgs.json'))` in Python. Example:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n curl -s --max-time 15 -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"remoteJid\": \"CONTACT_JID\"}}, \"limit\": 20}' -o /tmp/wa_msgs.json\n ```\n Then in Python: `data = json.load(open('/tmp/wa_msgs.json')); records = data['messages']['records']`\n\n## API Reference\n\nFor full API endpoints (media, contacts, location, buttons) β€” see [references/api.md](references/api.md).\n\n[Skill directory: /opt/data/skills/personal/whatsapp-evolution]\nResolve any relative paths in this skill (e.g. `scripts/foo.js`, `templates/config.yaml`) against that directory, then run them with the terminal tool using the absolute path.\n\n[This skill has supporting files:]\n- references/api.md -> /opt/data/skills/personal/whatsapp-evolution/references/api.md\n- scripts/process_queue.py -> /opt/data/skills/personal/whatsapp-evolution/scripts/process_queue.py\n- scripts/send_text.sh -> /opt/data/skills/personal/whatsapp-evolution/scripts/send_text.sh\n\nLoad any of these with skill_view(name=\"personal/whatsapp-evolution\", file_path=\"\"), or run scripts directly by absolute path (e.g. `node /opt/data/skills/personal/whatsapp-evolution/scripts/foo.js`).\n\nThe user has provided the following instruction alongside the skill invocation: Incoming WhatsApp webhook received.\n\nEvent: messages.upsert\nInstance: n8n-WA\nSender: Shalom Evonne\nSender JID: 120363418526182134@g.us\nFrom me: False\nMessage type: conversation\nMessage: {\n \"event\": \"messages.upsert\",\n \"instance\": \"n8n-WA\",\n \"data\": {\n \"key\": {\n \"remoteJid\": \"120363418526182134@g.us\",\n \"fromMe\": false,\n \"id\": \"3EB0F8682093702C1BD272\",\n \"participant\": \"131503308673050@lid\",\n \"participantAlt\": \"6591887488@s.whatsapp.net\",\n \"addressingMode\": \"lid\"\n },\n \"pushName\": \"Shalom Evonne\",\n \"status\": \"DELIVERY_ACK\",\n \"message\": {\n \"conversation\": \"Good evening Shalom Family!\\n\\nTomorrow is Tuesday, Week Odd.\\n\\nNaomi's Schedule:\\n8:00-8:25 - Morning Assembly\\n8:25-9:35 - Math\\n9:35-10:45 - PE (Mrs Ku)\\n10:45-11:20 - Recess\\n11:20-11:55 - G2/G3\\n11:55-12:30 - Geog\\n12:30-1:05 - Lunch\\n1:05-1:40 - MTL\\n1:40-2:15 - Sci\\n\\nDismissal: 2:15 PM\\n\\nPE reminder: Pack PE gear tonight! Mrs Ku will be teaching.\\n\\nMay the Lord bless Naomi with energy and joy for tomorrow. Have a restful evening!\\n\\nWith love,\\nShalom Evonne\"\n },\n \"contextInfo\": null,\n \"messageType\": \"conversation\",\n \"messageTimestamp\": 1779102068,\n \"instanceId\": \"b2bfaa30-d5f1-4b22-b979-791892619414\",\n \"source\": \"web\"\n },\n \"destination\": \"https://jimmy.huper.asia/webhooks/whatsapp-incoming\",\n \"date_time\": \"2026-05-18T08:01:08.337Z\",\n \"sender\": \"6590054438@s.whatsapp.net\",\n \"server_url\": \"https://evolution.huper.asia\",\n \"apikey\": \"5E4C532223A0-460B-A749-16C584F56E74\"\n}\n\nPrimary goal: REDUCE NOISE. Stay silent in the main Telegram chat unless the message truly needs Vincent's immediate attention. Approval-worthy reply drafts should go to the jimmy-wa-drafts sidecar, NOT to the main Telegram chat.\n\nFollow the whatsapp-evolution skill and /opt/data/agent-state/memory/whatsapp-vip-contacts.json, with these stricter notification rules:\n\nABSOLUTE SILENCE RULES:\n- If fromMe is true: ignore. Do NOT call send_message. Final response local log only.\n- Ignore status/update/read-receipt events or payloads with no real message content.\n- Do NOT send main Telegram summaries for ordinary DMs, routine logistics, casual family messages, generic group events, group chatter, forwards, memes, devotional/religious shares, links, standalone codes, normal confirmations, or suggested replies.\n- Do NOT ask \"Want me to add this to Calendar?\" for WhatsApp group events unless Vincent is explicitly invited, named, responsible, or the event is time-sensitive and clearly relevant to him today.\n- Do NOT surface family messages just because they are family. Only surface if urgent, safety-related, school/pickup critical, money/legal/medical, or explicitly requires Vincent now.\n\nSIDEcar RULES:\n- If a reply may be useful but it is not urgent, POST it to https://jimmy-draft.huper.asia/drafts using the whatsapp-evolution Draft Approvals flow. Do NOT duplicate that draft/summary in the main Telegram chat.\n- Prayer and birthday replies should go to the sidecar when appropriate, not the main Telegram chat, unless the underlying situation is urgent/sensitive and Vincent needs immediate awareness.\n- Before drafting any reply or approval suggestion, fetch recent context: python3 /opt/data/scripts/wa_recent_messages.py \"120363418526182134@g.us\" 30. For active/ambiguous groups use 50.\n\nMAIN TELEGRAM ALERTS ARE ALLOWED ONLY FOR:\n1. VIP contacts marked always_call or always_notify when the message is important/actionable, or any VIP call event.\n2. Calls from family/VIP that appear missed/ringing and likely need immediate attention. Avoid repeated alerts for accepted/terminated duplicate call state updates.\n3. Emergencies, safety issues, medical issues, school pickup/child logistics requiring action now, legal/money/account/security matters, or time-sensitive commitments in the next few hours.\n4. Messages explicitly asking Vincent to call back, decide now, approve payment, provide OTP/code for an active task, or respond urgently.\n5. Pipeline errors or webhook processing failures.\n\nCALL RULES:\n- VIP contacts (e.g. Dn Jeff, Ng Whye Keong): call Vincent immediately outside quiet hours when actionably important or on ringing/missed call. During 1am-10am SGT, Telegram only unless life-and-death emergency.\n- Do not call for casual VIP confirmations or non-actionable courtesy messages.\n\nTONE/RULES FOR DRAFTS:\n- Proposed replies must be Christian and grace-based where appropriate: warm, humble, encouraging, non-judgmental, peace-making, not preachy.\n- Never use em dashes in drafted replies or approval messages. Use commas, periods, colons, or parentheses instead.\n- NEVER auto-reply directly on WhatsApp unless Vincent explicitly approves or future config marks that exact category pre-approved.\n\nDelivery rule:\n- For allowed main alerts only: call send_message(target=\"telegram\", message=...). Keep it concise.\n- For non-urgent reply suggestions: use the sidecar only.\n- For everything else: DO NOT call send_message. Keep final response as local-only log.\n", "timestamp": 1779102080.7047033, "anchor": true}, {"id": 67332, "role": "assistant", "content": "", "timestamp": 1779102080.7307496, "tool_calls": [{"id": "call_G8niK6FzXLDeSuFZgC1C73AZ", "call_id": "call_G8niK6FzXLDeSuFZgC1C73AZ", "response_item_id": "fc_0f3deeb2b9fdf8d1016a0af17950d481918312e9ff57fc0088", "type": "function", "function": {"name": "read_file", "arguments": "{\"path\":\"/opt/data/agent-state/memory/whatsapp-vip-contacts.json\",\"limit\":500}"}}]}, {"id": 67334, "role": "assistant", "content": "Local-only log: routine school schedule/reminder for Naomi from Shalom Evonne in group `120363418526182134@g.us`.\n\nAction taken: no Telegram alert, no WhatsApp draft, no call. Reason: routine logistics, not urgent or requiring Vincent’s immediate attention.", "timestamp": 1779102080.7347953}], "bookend_end": [], "messages_before": 0, "messages_after": 4}, {"session_id": "20260513_110032_e1553b05", "when": "May 13, 2026 at 11:00 AM", "source": "webhook", "model": "gpt-5.5", "title": "WhatsApp School Timetable Update", "matched_role": "user", "match_message_id": 53493, "snippet": "...draft_text\": \"Thanks >>>Karen<<<, will miss you both β€” praying for a good time too πŸ™\",\n \"extras\": {\n \"group_jid\": \"120363351787895353@g.us\",\n \"sender_name\": \">>>Karen<<< >>>Tan<<<\",\n \"sender_jid\": \">>>6597630553<<<@s.whatsapp.net\",\n \"category\": \">>>prayer<<>>blessing<<<\",\n \"original_message\": \"Happy >>>cycling<<<, to those who...", "bookend_start": [], "messages": [{"id": 53493, "role": "user", "content": "[IMPORTANT: The user has invoked the \"whatsapp-evolution\" skill, indicating they want you to follow its instructions. The full skill content is loaded below.]\n\n---\nname: whatsapp-evolution\ndescription: WhatsApp messaging via Evolution API β€” send, receive, analyze, and auto-reply. Use when processing an incoming WhatsApp message delivered via the webhook, replying on Vincent's behalf, liaising with contacts, sending messages, or handling WhatsApp-related tasks. Covers intent analysis, auto-reply rules, escalation (VIP β†’ phone call), conversation context fetching, and draft approvals via the jimmy-wa-drafts sidecar.\n---\n\n# WhatsApp via Evolution API\n\n## Credentials\nStored in `/opt/data/secrets/.whatsapp-evolution-credentials`:\n- `EVOLUTION_API_URL`, `EVOLUTION_API_KEY`, `EVOLUTION_INSTANCE`\n\nLoad with: `export $(cat .whatsapp-evolution-credentials | xargs)`\n\n## Sending Messages (direct, no approval)\n\n```bash\nexport $(cat .whatsapp-evolution-credentials | xargs)\ncurl -s -X POST \"${EVOLUTION_API_URL}/message/sendText/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"number\": \"6591234567\", \"text\": \"Hello!\"}'\n```\n\n- Phone format: country code, no `+` (e.g. `6590054438`)\n- Groups: use JID `{groupid}@g.us`\n- For media/location/contacts β€” see [references/api.md](references/api.md)\n\nUse direct send only when Vincent has explicitly commanded/approved the exact send, or for pre-approved auto-replies/follow-ups. For any reply that needs Vincent's review, use the sidecar (next section).\n\n### Sending trip/camp details from retrieved emails\n\nWhen Vincent explicitly asks to send extracted school trip/camp details to a child or family member, send directly. Keep it practical and grounded:\n- Include dates, reporting/pick-up times, flights, venue/itinerary highlights, deadlines, forms, and safety/medication notes when found.\n- If an official packing list is missing, say that clearly. Do not invent one. You may add a short β€œbased on itinerary/weather, prepare…” list, clearly labeled as practical guidance rather than official school instructions.\n- Avoid long explanations about the search process in WhatsApp. Send the useful details.\n\n### Sending a local image/file as media\n\nFor user-approved media sends, base64-encode the local file and call `sendMedia`:\n\n```bash\nexport $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\npython3 - <<'PY'\nimport base64, json\npath = '/tmp/file.png'\npayload = {\n 'number': '6591234567',\n 'mediatype': 'image',\n 'mimetype': 'image/png',\n 'caption': 'Caption here',\n 'media': base64.b64encode(open(path, 'rb').read()).decode(),\n 'fileName': 'file.png',\n}\nopen('/tmp/wa_media_payload.json', 'w').write(json.dumps(payload))\nPY\ncurl -s --max-time 60 -X POST \"${EVOLUTION_API_URL}/message/sendMedia/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_media_payload.json\n```\n\nWhen sending QR tickets or passes, verify the crop/image contains the full QR code and the intended attendee name before sending; the QR screenshot is effectively the redeemable ticket.\n\n## Fetching Conversation Context\n\nBefore drafting or proposing any reply, always fetch recent chat history for context:\n\n```bash\npython3 /opt/data/scripts/wa_recent_messages.py \"6591234567@s.whatsapp.net\" 30\n```\n\nUse `limit=30` by default. For active group chats or ambiguous situations, fetch up to 50 messages. Read enough context to understand who is being discussed, current tone, previous commitments, and whether Vincent has already replied. The script returns newest messages first (descending order).\n\n## Draft Approvals β€” `jimmy-wa-drafts` sidecar\n\nWhen you need Vincent to approve a WhatsApp reply, POST the draft to the sidecar. It owns the entire approval flow: Telegram message with 4 inline buttons, Telegram WebApp for Edit/Context, sending to WhatsApp via Evolution API on Approve.\n\n### Posting a draft\n\nAlways include an `extras` object with the original WA context. The sidecar passes `extras` through verbatim if Vincent later clicks \"Context\" β€” without it, your redraft will be flying blind.\n\n```bash\ncat > /tmp/wa_draft.json <<'JSON'\n{\n \"contact\": \"120363351787895353@g.us\",\n \"contact_name\": \"Cycling group\",\n \"summary\": \"Karen sent a blessing over the bike ride; she's not joining\",\n \"draft_text\": \"Thanks Karen, will miss you both β€” praying for a good time too πŸ™\",\n \"extras\": {\n \"group_jid\": \"120363351787895353@g.us\",\n \"sender_name\": \"Karen Tan\",\n \"sender_jid\": \"6597630553@s.whatsapp.net\",\n \"category\": \"prayer/blessing\",\n \"original_message\": \"Happy cycling, to those who can make it… Dn Jeff and I had made plans for this PM so we won't be able to join y'all. Praying the Lord's favor, protection and good-weather blessings be upon y'all and your route!\",\n \"extracted\": {\n \"what\": \"Cycling from East Coast Park to Changi Village, then dinner\",\n \"when\": \"Today, 4:00 PM\",\n \"where\": \"East Coast Park to Changi Village\",\n \"who\": \"Group members who voted yes for cycling/dinner or dinner\"\n },\n \"confidence\": \"high\",\n \"why_appropriate\": \"Multiple group messages are blessings/wishes; Vincent has not replied; tone matches grace-based norm.\",\n \"wa_message_id\": \"ABC123XYZ\"\n }\n}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_draft.json\n```\n\nResponse: `{\"draft_id\":\"abc123def\",\"telegram_message_id\":42}`\n\nWhat to put in `extras` (use whatever makes sense for the message type):\n\n- **Always**: `original_message` (verbatim WA text), `sender_name`, `category` (e.g. `prayer`, `birthday`, `event`, `dm-reply`, `enquiry`)\n- **Group messages**: `group_jid`, `sender_jid`\n- **DMs**: `sender_jid`\n- **Strongly recommended (drives the rich Telegram render)**: `extracted` (dict of who/what/when/where), `confidence` (`high`/`medium`/`low`), `why_appropriate` (one-line reasoning shown to Vincent so he can decide)\n- **Optional**: `wa_message_id`, `tone_hint` (e.g. `warm-grace-based`), `sender_phone` (auto-derived from `sender_jid` if omitted)\n\nVincent receives a Telegram message in the `@jimmy_WA_bot` chat with 4 inline buttons. The sidecar handles every interaction; **you do not track replies, callbacks, or message IDs**. Just POST and move on.\n\n| Button | Behavior |\n|--------|----------|\n| **Approve** | Sidecar sends draft to WhatsApp |\n| **Edit** | Opens a Telegram WebApp with draft pre-filled; Vincent edits + sends; sidecar pushes to WhatsApp |\n| **Context** | Opens a Telegram WebApp with empty textarea + the original WA message visible; Vincent provides context. (When `HERMES_REDRAFT_WEBHOOK` is wired in the sidecar's env, this will POST back to Jimmy for a redraft. Until then, the context is just stored on the draft.) |\n| **Pass** | Sidecar deletes the draft |\n\n### JSON escaping when `draft_text` contains quotes\n\nSingle quotes (`It's`, `God's`) break shell interpolation. Write the JSON to a temp file first:\n\n```bash\ncat > /tmp/wa_draft.json <<'JSON'\n{\"contact\":\"6591234567@s.whatsapp.net\",\"contact_name\":\"John\",\"summary\":\"…\",\"draft_text\":\"It's nice to hear from you!\"}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_draft.json\n```\n\n### Networking\n\n`https://jimmy-draft.huper.asia` is the public HTTPS endpoint of the sidecar (TLS-terminated by Traefik on ubuntu-server). HTTPS is required because Hermes blocks plain-HTTP commands via tirith. The internal Docker DNS hostname http://jimmy-wa-v64mqe:8080 also works for non-shell calls but the agent is forbidden from using it.\n\n### Sidecar status & logs\n\n- Source: (auto-deploys on push)\n- Public URL: (`/health` for ping)\n- Container: `jimmy-wa-v64mqe.1.` on ubuntu-server\n- Bot: `@jimmy_WA_bot`\n\n## Handling a Redraft Trigger (`wa-redraft` webhook)\n\nWhen Vincent clicks the **Context** button on a draft, the sidecar:\n1. Edits the original Telegram message to \"πŸ”„ Adding context for X… agent is redrafting.\" (removes the old buttons so Vincent can't approve the now-stale text)\n2. POSTs to your `wa-redraft` webhook (`http://localhost:8644/webhooks/wa-redraft`) with the original draft fields plus Vincent's `context` instructions and the `extras` you stored when you first created the draft.\n\nWhen you receive a `wa-redraft` activation, the prompt template will lay out the fields. Your job:\n\n1. Read Vincent's `context` carefully β€” that's the steering signal. The `previous_draft` is just history, not necessarily a starting point.\n2. Re-fetch recent WA history if the original context isn't enough (`python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`). Use `extras.group_jid` for groups or `contact` for DMs.\n3. Write a NEW draft that addresses Vincent's instructions while staying faithful to the original WA context (in `extras.original_message`, `extras.extracted`).\n4. POST the new draft back to the sidecar **with the same `draft_id`** so the existing draft is updated, not duplicated:\n\n```bash\ncat > /tmp/wa_redraft.json <\",\n \"contact\": \"\",\n \"contact_name\": \"\",\n \"summary\": \"\",\n \"draft_text\": \"\",\n \"extras\": \n}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_redraft.json\n```\n\nThe sidecar will send a NEW Telegram message with 4 fresh buttons. Vincent can Approve, Edit, Context-again (loop), or Pass.\n\nDo **not** send to WhatsApp directly during a redraft β€” the approval flow is the sidecar's job.\n\n## Incoming Message Processing\n\n### Delivery Pipeline (current)\n1. Evolution API POSTs each `MESSAGES_UPSERT` webhook directly to Hermes at `http://localhost:8644/webhooks/whatsapp-incoming` (proxied via `jimmy.huper.asia/webhooks/whatsapp-incoming`)\n2. Hermes webhook route (`whatsapp-incoming`, dynamic subscription) validates and spawns an agent\n3. Agent loads `whatsapp-evolution` skill, follows the rules below\n4. Webhook subscription uses `deliver: log` because webhooks do not support `deliver: local`; the agent explicitly calls `send_message(target=\"telegram\", ...)` only for important/actionable/VIP/event/birthday/prayer/error items, OR posts a draft to the sidecar for approval items.\n\n**No more OpenClaw, no Python webhook server, no port 5001, no keepalive cron.** If messages aren't being processed, check: nginx proxy to port 8644, `hermes webhook list` for the subscription, Evolution API webhook URL setting.\n\n## Contact Rules & Escalation\n\nConfig: `memory/whatsapp-vip-contacts.json`\n\n### Priority Levels\n\n| Priority | Action | Example |\n|----------|--------|---------|\n| **VIP (call-always)** | **CALL Vincent immediately via Twilio without asking.** Do NOT ask for approval first. During quiet hours (1am-10am SGT) suppress call and use Telegram notification only, unless life-and-death emergency. Match VIPs by both phone JID (`659...@s.whatsapp.net`) and LID JID (`...@lid`) when present, because recent chats may arrive with `key.remoteJid` as an LID and the phone number only in `key.remoteJidAlt`. | Dn Jeff (+6597630553) |\n| **Family** | Always notify Vincent on Telegram | ps112 group, family members |\n| **Auto-reply** | Reply based on context file, notify Vincent | Repair shops |\n| **Normal** | Analyze intent, reply if appropriate, notify if important | Everyone else |\n| **Broadcast/Spam** | Ignore silently | Forwarded job posts, chain messages |\n\n### Quiet Hours\n- 1am-10am SGT (17:00-02:00 UTC): avoid phone calls, including for Dn Jeff / VIP call-always contacts\n- **Exception:** life-and-death emergencies always call regardless of quiet hours\n- During quiet hours, use Telegram notification only and mention that a call was suppressed due to quiet hours\n\n## Making a Call via Twilio\n\nCredentials: `/opt/data/secrets/.twilio-credentials-new` (`TWILIO_ACCOUNT_SID`, `TWILIO_AUTH_TOKEN`, `TWILIO_PHONE_NUMBER`). From: `+16614809287`, Vincent: `+6590054438`.\n\nPreferred script: `/opt/data/scripts/call_vincent.py`, now using the Twilio Python SDK baked into the container.\n\n```bash\npython3 /opt/data/scripts/call_vincent.py \"Hello Vincent, this is Jimmy. Urgent summary here.\"\n```\n\n## Action Rules\n\n### Default: DO NOT auto-reply directly, and do not spam main Telegram\nCollect intent and stay local unless something truly needs Vincent's immediate attention.\n\n**When you need Vincent to approve a reply, POST a draft to the sidecar (see \"Draft Approvals\" above).** The sidecar handles the Telegram notification with buttons, the WebApp for editing, and sending to WhatsApp on approve. Do **not** duplicate that draft or summary in the main Telegram chat.\n\nMain Telegram WhatsApp summaries are muted by default. Only use `send_message(target=\"telegram\", ...)` for urgent/important/actionable items that need Vincent now: VIP actionable messages/calls, missed/ringing family calls, emergencies, safety/medical/school-pickup issues, money/legal/account/security matters, explicit call-back/decision-now requests, or pipeline errors. Routine logistics, casual family pings, generic group events, links, standalone codes, normal confirmations, and suggested replies should be local-only or sidecar-only.\n\nDo NOT send WhatsApp replies directly unless Vincent explicitly approves them or the category is pre-approved.\n\n### Tone (when proposing or sending replies)\n- Casual, direct, no sign-offs (no \"Thanks & Regards\", no \"- Vincent\")\n- Ask questions instead of giving instructions\n- Keep it natural β€” like texting a friend\n- Christian and grace-based: warm, humble, encouraging, non-judgmental, peace-making\n- Never use em dashes in drafted replies or approval messages. Use commas, periods, colons, or parentheses instead.\n- For prayer/birthday contexts, use gentle faith-filled language; avoid sounding preachy, performative, or overly formal\n\n### What to Do With Each Message\n\n**DMs (personal chats):**\n- Analyze intent\n- Notify Vincent on Telegram (via `send_message(target=\"telegram\", ...)`) only if important/actionable/time-sensitive, requires a decision, comes from a VIP contact\n- If you have a useful suggested reply, POST it as a draft to the sidecar instead of a Telegram notification β€” Vincent gets the buttons inline\n- Casual/social/non-actionable DMs β†’ local log only, no Telegram notification, no draft\n- Do NOT reply unless Vincent says to\n- VIP contacts β†’ **call Vincent immediately without asking**. Do NOT post a draft. Just call. During quiet hours (1am-10am SGT), suppress the call and send Telegram notification only, unless life-and-death emergency.\n\n**Group chats β€” events/meetups:**\n- If someone posts about an event, gathering, meetup, dinner, etc:\n 1. Extract: what, when, where, who\n 2. Notify Vincent on Telegram with details (no draft needed unless you also want to suggest an RSVP reply)\n 3. Ask: \"Want me to add this to Google Calendar and set a reminder?\"\n\n**Group chats β€” prayer requests / praying for others:**\n- If someone asks for prayer, shares a burden, illness, grief, struggle, thanksgiving, or asks the group to pray:\n 1. Fetch recent group context first: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n 2. Identify who/what to pray for and any sensitive details\n 3. Draft a short, grace-based Christian reply/prayer\n 4. POST it to the sidecar with `summary` describing the request and `draft_text` being the prayer\n 5. Do NOT post to WhatsApp unless Vincent approves\n\n**Group chats β€” birthday greetings:**\n- If you see birthday wishes for someone in a group:\n 1. Fetch recent group context first: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n 2. Note who the birthday person is and relationship/context if inferable\n 3. Draft a warm, grace-based Christian birthday greeting\n 4. POST it to the sidecar\n 5. Do NOT post to WhatsApp unless Vincent approves\n\n**Group chats β€” general:**\n- Casual banter, memes, forwards, broadcast spam β†’ ignore silently\n- Religious content shares β†’ ignore unless specifically addressed to Vincent\n\n### When to Notify Vincent (Telegram) vs Post a Draft\n\n**Use `send_message(target=\"telegram\", ...)` sparingly** for plain notifications only when the message needs Vincent's immediate attention: VIP actionable DMs/calls, emergencies, school pickup/child logistics requiring action now, money/legal/account/security matters, explicit call-back/decision-now requests, or pipeline errors.\n\n**Use the sidecar (POST `/drafts`)** whenever you have a suggested reply for Vincent to approve: prayer responses, birthday greetings, DM replies, group RSVPs. Do not also send the same summary to the main Telegram chat.\n\nDo **not** notify or draft for casual/social/non-actionable DMs, routine logistics, read receipts/status updates, religious shares not addressed to Vincent, memes, forwarded/broadcast/spam, generic group events, standalone codes, normal confirmations, or general group chatter. Family messages are not automatically main-chat alerts unless urgent, safety-related, school/pickup critical, or explicitly requiring Vincent now.\n\n## Intent Categories\n\nWhen analyzing incoming messages, classify as:\n\n1. **Urgent** β€” time-sensitive, emergencies β†’ call Vincent outside quiet hours; during quiet hours use Telegram only unless life-and-death\n2. **Important** β€” decisions, money, appointments, anything consequential β†’ call Vincent outside quiet hours; during quiet hours notify on Telegram only unless life-threatening\n3. **Request** β€” asking Vincent to do something β†’ **notify Vincent** (Telegram) or post a draft if you have a safe reply\n4. **Enquiry** β€” asking for info, pricing, availability β†’ auto-reply if context available, otherwise post a draft\n5. **Confirmation** β€” confirming plans, bookings β†’ acknowledge and notify Vincent\n6. **Casual/Social** β€” greetings, memes, forwards β†’ usually ignore or react\n7. **Broadcast/Spam** β€” forwarded job posts, chain messages β†’ ignore\n\n## Finding Individual Contacts and JIDs\n\nWhen Vincent asks to monitor or prioritize a specific WhatsApp contact and only gives a name:\n\n1. Query contacts and chats, then search both display names and serialized message metadata:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n python3 - <<'PY'\n import os, requests, json\n base=os.environ['EVOLUTION_API_URL']; inst=os.environ['EVOLUTION_INSTANCE']; key=os.environ['EVOLUTION_API_KEY']\n headers={'apikey':key,'Content-Type':'application/json'}\n terms=['whye','keong'] # replace with lowercase search terms\n for ep in ['findContacts','findChats']:\n r=requests.post(f'{base}/chat/{ep}/{inst}', headers=headers, json={}, timeout=30)\n print('endpoint', ep, 'status', r.status_code)\n for obj in (r.json() if r.ok else []):\n if any(t in json.dumps(obj, ensure_ascii=False).lower() for t in terms):\n print(json.dumps(obj, ensure_ascii=False)[:1000])\n PY\n ```\n2. If `findChats` shows `remoteJid` ending in `@lid`, inspect `lastMessage.key.remoteJidAlt` for the phone JID. Store both in `memory/whatsapp-vip-contacts.json` when configuring VIP/call rules.\n3. Verify the number exists with `/chat/whatsappNumbers/{instance}` before finalizing.\n4. Validate the JSON config after editing: `python3 -m json.tool /opt/data/agent-state/memory/whatsapp-vip-contacts.json >/tmp/wa_vip_validated.json`.\n\n## Finding Group JIDs\n\nThe `group/fetchGroups` endpoint returns 404 on some Evolution API instances (including ours). To find a group JID when you don't have it:\n\n1. **Use `chat/findMessages` with an empty `where` filter** to pull recent messages across all chats, then extract unique `remoteJid` values containing `@g.us`.\n2. **Match by content**: Fetch messages from each group JID using `where.key.remoteJid` and match sender names or message text to identify the target group.\n3. **Known group JIDs** (store in `whatsapp-vip-contacts.json` as you discover them):\n - ps112 family group: `6591887488-1464061170@g.us`\n - Overcomers group: `6591444911-1614821287@g.us`\n - η₯žηš„εŽηΎŽ group: `6596756578-1405873840@g.us` (content-sharing, casual/broadcast, local log only)\n - Overcomers/TCians group: `120363230236878544@g.us` (church fellowship, PJ sends weekly attendance reports, routine attendance = local log only)\n\nExample search:\n```bash\ncurl -s -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"remoteJid\": \"6591444911-1614821287@g.us\"}}, \"limit\": 10, \"order\": \"desc\"}'\n```\n\n## Processing Image/Media Messages\n\nWhen an incoming WhatsApp message is an image (imageMessage), video, or other media with no text or only a caption:\n\n1. **Fetch conversation context** as usual with `wa_recent_messages.py`\n2. **Download the media** using the Evolution API:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"message\": {\"key\": {\"id\": \"MESSAGE_ID\", \"fromMe\": false, \"remoteJid\": \"GROUP_OR_CHAT_JID\"}}, \"convertToMp4\": false}' \\\n -o /tmp/wa_media_response.json\n ```\n Then decode: `python3 -c \"import json,base64; d=json.load(open('/tmp/wa_media_response.json')); open('/tmp/wa_media.jpg','wb').write(base64.b64decode(d['base64']))\"`\n3. **Analyze the image** using `vision_analyze` tool with `/tmp/wa_media.jpg` path and a context-aware question\n4. **Fallback if vision is unavailable**: Use `tesseract /tmp/wa_media.jpg stdout` for OCR, combined with pixel analysis (PIL/numpy) and conversation context to infer what the image contains\n5. **Album messages** (`albumMessage`): These bundle multiple images. Each image arrives as a separate `imageMessage` with a `messageAssociation.parentMessageKey` linking them to the album. Download each image separately. The `expectedImageCount` field tells how many images to expect\n6. **messageAssociation**: When images have `parentMessageKey` with `fromMe: true`, they are replying to a message Vincent sent. Use this for context\n\n### Audio / Voice Note Messages\nWhen an incoming WhatsApp message is `audioMessage` or a voice note (`ptt: true`), treat it like other media, but transcribe before intent classification when possible:\n\n1. **Fetch conversation context first**, especially in groups:\n ```bash\n python3 /opt/data/scripts/wa_recent_messages.py \"\" 50\n ```\n2. **Download the audio** with Evolution API, saving the response to a file:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n cat > /tmp/wa_audio_req.json <<'JSON'\n {\"message\":{\"key\":{\"id\":\"MESSAGE_ID\",\"fromMe\":false,\"remoteJid\":\"GROUP_OR_CHAT_JID\"}},\"convertToMp4\":false}\n JSON\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_audio_req.json -o /tmp/wa_audio_response.json\n python3 - <<'PY'\n import json, base64\n d=json.load(open('/tmp/wa_audio_response.json'))\n b=d.get('base64') or d.get('data',{}).get('base64')\n if ',' in b[:100]: b=b.split(',',1)[1]\n open('/tmp/wa_audio.ogg','wb').write(base64.b64decode(b))\n PY\n ```\n3. **Convert OGG/Opus to WAV**:\n ```bash\n ffmpeg -y -i /tmp/wa_audio.ogg -ar 16000 -ac 1 /tmp/wa_audio.wav\n ```\n4. **Transcribe fallback when Whisper is unavailable**: the container may not have `whisper`, `faster-whisper`, `torch`, or `speech_recognition` installed globally. A reliable lightweight fallback is a temp venv with SpeechRecognition and Google web speech:\n ```bash\n python3 -m venv /tmp/srvenv\n /tmp/srvenv/bin/pip install --quiet SpeechRecognition\n /tmp/srvenv/bin/python - <<'PY'\n import speech_recognition as sr\n r=sr.Recognizer()\n with sr.AudioFile('/tmp/wa_audio.wav') as source:\n audio=r.record(source)\n print(r.recognize_google(audio, language='en-SG'))\n PY\n ```\n5. **Check adjacent voice notes from the same sender**. WhatsApp voice notes are often sent in quick sequential chunks. Download and transcribe nearby `audioMessage` records from the same sender before deciding whether to notify Vincent.\n6. Use the transcript plus recent context to classify the message. For group meetup planning, extract what/when/where/who and notify Vincent if actionable, including transcript confidence.\n\n### Video Messages\n\nWhen an incoming WhatsApp message is `videoMessage` (with or without a caption):\n\n1. **Fetch conversation context** first, as with any media message: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n2. **Download the video** using Evolution API (same pattern as images/audio β€” save response to file, then decode):\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n cat > /tmp/wa_video_req.json <<'JSON'\n {\"message\":{\"key\":{\"id\":\"MESSAGE_ID\",\"fromMe\":false,\"remoteJid\":\"CONTACT_JID\"}},\"convertToMp4\":false}\n JSON\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_video_req.json -o /tmp/wa_video_response.json\n python3 - <<'PY'\n import json, base64\n d = json.load(open('/tmp/wa_video_response.json'))\n b = d.get('base64') or d.get('data', {}).get('base64', '')\n if ',' in b[:100]: b = b.split(',', 1)[1]\n open('/tmp/wa_video.mp4', 'wb').write(base64.b64decode(b))\n print(f\"Video saved: {len(base64.b64decode(b))} bytes\")\n PY\n ```\n3. **Extract representative frames** with ffmpeg (do NOT try to analyze video directly β€” vision tools work on images):\n ```bash\n # Frame at start\n ffmpeg -y -ss 0 -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame0.jpg\n # Frame at ~1/3 of the way (good for short videos)\n ffmpeg -y -ss 3 -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame3s.jpg\n # If video is longer (>15s), also extract at midpoint\n # ffprobe -v quiet -print_format json -show_format /tmp/wa_video.mp4 # get duration\n # ffmpeg -y -ss $HALF_DURATION -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame_mid.jpg\n ```\n4. **Analyze the frames** with `vision_analyze`. Ask specific questions about content: what objects/people/text are visible, what type of video this appears to be (screen recording, personal clip, forwarded meme, etc.), and any text or identifying details.\n5. **Use the caption** from `videoMessage.caption` if present (can be empty/null). Combine frame analysis + caption + conversation context to classify the message intent.\n6. **Transcription of audio in video** is generally not feasible with available tools. If the video likely contains important spoken content (e.g. voice memo style), note this limitation and flag it to Vincent.\n\n**Key points:**\n- Always use `-o` flag and file-based decode; videos can be 1-10MB and base64 responses will be even larger\n- `--max-time 25` on curl is essential for video downloads\n- Short casual clips (showing a key fob, a parking spot, quick view of surroundings) are usually casual/social β€” classify accordingly and don't over-escalate\n- If `vision_analyze` fails (400 on models that don't support images, or 429 rate limit), fall back to conversation context inference alone\n\n### Poll Messages\nWhen `pollUpdateMessage` appears in chat history (someone voted), the message contains a `pollCreationMessageKey` with the original poll's message `id`. To get the poll question and options, query:\n```bash\ncurl -s -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"id\": \"POLL_MESSAGE_ID\", \"remoteJid\": \"GROUP_JID\"}}, \"limit\": 1}'\n```\nThe response will have `pollCreationMessageV3` with `name` (the poll question) and `options` array. Always look up the poll when votes or poll references appear, since the poll text contains the event details.\n\n### `associatedChildMessage` (Motion Photos / Companion Media)\nWhatsApp motion photos and companion media arrive as `associatedChildMessage` type. These are short videos or supplementary images attached to a parent message via `messageAssociation.parentMessageKey`. The structure is:\n\n```\nmessage: {\n messageContextInfo: {\n messageAssociation: {\n associationType: 12,\n parentMessageKey: { id, fromMe, remoteJid }\n }\n },\n associatedChildMessage: {\n message: {\n videoMessage: { ... } // or imageMessage: { ... }\n }\n }\n}\n```\n\nKey points:\n- The `parentMessageKey.id` links back to the main message (the image/video they accompany)\n- When `parentMessageKey.fromMe: true`, the companion media is replying to a message Vincent sent\n- Captions may appear on the child's `videoMessage.caption` or `imageMessage.caption` inside the nested message, NOT at the top level\n- These often contain motion-photo short videos (3 seconds) or album companion images\n- Treat them as media attachments to the parent; don't process them as standalone messages requiring independent intent analysis\n\n### Sequential Image Sets (Non-Album)\nMultiple images sent rapidly by the same sender at similar timestamps may NOT be wrapped in an `albumMessage`. They appear as separate `imageMessage` entries. Always check for adjacent images from the same sender and download+OCR all of them together, as they often form a coherent multi-page set (e.g., route info, comparison tables, maps). Missing one can lose important context.\n\n### Key pitfalls\n- The Evolution API `getBase64FromMediaMessage` endpoint returns the full base64 payload (can be 100KB+ for images). Use `-o` flag to save to file, then parse with Python. Do NOT try to pipe it directly\n- `curl` with `--max-time 25` is essential; without it, large media responses can hang\n- The `albumMessage` type itself has no `fromMe` field; check each individual `imageMessage` instead\n- Image thumbnails in the webhook payload are truncated; always download the full image for analysis\n- **WhatsApp media URLs expire**: CDN download links (`mmg.whatsapp.net`) expire after roughly 24-48 hours. When `getBase64FromMediaMessage` returns a 400/404 error like \"Failed to fetch stream from https://mmg.whatsapp.net/...\", the media is no longer retrievable via the API. This means you CANNOT download adjacent/older voice notes, images, or documents for context if they're more than a day or two old. Fall back to conversation text context only. Fresh/incoming messages are almost always still downloadable.\n- **Voice note transcription reliability**: Short voice notes (<15s), especially in Singlish/Chinese-English mix common in Singapore contacts, often produce unreliable transcriptions via SpeechRecognition+Google. Always flag low-confidence transcriptions to Vincent rather than acting on potentially garbled text.\n- **Vision API failures**: `vision_analyze` can fail with 429 (rate limit) **or 400** (model doesn't support image input, e.g. glm-5.1 via ollama-cloud). Always have a fallback plan. When vision is unavailable, try tesseract OCR, then pixel analysis (PIL/numpy for brightness zones, text region detection via row-level dark-pixel density)\n- **OCR can fail on decorative/complex images**: Obituaries, memorial notices, religious flyers with ornate fonts, or dark-background images often defeat tesseract even with aggressive preprocessing (upscaling, thresholding, inverting, PSM mode variations). When OCR returns gibberish or empty, fall back to conversation context inference: use preceding/surrounding messages to determine what the image likely contains, then draft an appropriate response based on that context\n- **OCR on devotional/dark-background images**: These often defeat tesseract on full-image scans. Use a multi-strategy approach: (1) run tesseract on the full image first, (2) if text is sparse, crop into regions (top 15%, middle, bottom 15%), upscale 2-3x, enhance contrast (2-3x), and try both normal and inverted (`ImageOps.invert`) versions with `--psm 6`, (3) combine all partial OCR results. For Exodus/verse images, the title (e.g. \"WHEN MOSES' HANDS WERE\") is often in decorative font that OCR misses, while the verse body text is usually readable.\n- **Prioritize conversation context**: When processing group images, always fetch recent context FIRST (before attempting vision/OCR). If image analysis fails, the conversation history often makes the image's purpose clear enough to draft a meaningful reply\n- **DM conversation context is one-sided (fromMe only)**: The Evolution API `findMessages` endpoint often returns only Vincent's sent messages (`fromMe: true`) for DM chats, not the contact's incoming messages. This means `wa_recent_messages.py` can show only one side of the conversation. When processing a DM webhook, do NOT rely solely on fetched history for the contact's side. The webhook payload already contains the incoming message text. Use what the webhook gives you plus Vincent's side from the API. If you need the contact's recent messages, check if `fromMe: false` records exist; if none are returned, the API simply doesn't have them and you must work with what you have (webhook message + Vincent's replies for context).\n- **`findMessages` ignores `key.fromMe` filter**: Passing `\"key.fromMe\": false` in the `where` clause still returns all messages (including `fromMe: true`). To find messages from a specific sender, fetch results and filter client-side by checking `fromMe` in Python.\n- **`findMessages` caps at ~50 records per request**: Even with `limit: 200` or `limit: 500`, the API returns at most 50 records. When one party sends many consecutive messages, the other party's messages may not appear in the first 50 results. Use pagination or fetch multiple pages if you need older messages.\n- **Video frame extraction**: Use `ffmpeg -y -ss N -i video.mp4 -frames:v 1 -update 1 output.jpg` (the `-update 1` flag is needed; without it ffmpeg treats the output as an image sequence and fails). Extract frames at multiple timestamps (start, ~1/3, midpoint for longer videos) since a single frame may miss the key content.\n- **ffmpeg \"Invalid color space\" failure on WhatsApp videos**: Some WhatsApp H.264 videos have `color_space=reserved` and `color_primaries=reserved` metadata that ffmpeg cannot decode at all. Every codec/format/filter combination fails with \"Invalid color space\" / return code 234. This is NOT fixable with `-vf setparams`, `-color_trc`, `-colorspace`, `-color_primaries`, or `-vf format=` flags because the error occurs at the decoder level before any filter runs. **Do not waste time trying ffmpeg workarounds.** When this happens, fall back to: (1) conversation context inference (who sent it, what thread it's in, adjacent messages), (2) the jpegThumbnail embedded in the videoMessage (small but gives a rough idea; base64-encoded JPEG, typically 600-900 bytes, use `vision_analyze` on it if available), or (3) if all analysis fails, just classify based on surrounding text context.\n- **`wa_recent_messages.py` JSON output breaks `json.loads()`**: The script's stdout contains WhatsApp message content with control characters (emoji, CJK, deviceListMetadata binary blobs) that cause `json.loads()` to fail with `Invalid control character` or `Expecting ',' delimiter` errors. Even `strict=False` does not reliably fix it. **Reliable approach**: use `curl -o /tmp/wa_msgs.json` to save the Evolution API response directly to a file (the file write handles binary safely), then parse with `json.load(open('/tmp/wa_msgs.json'))` in Python. Example:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n curl -s --max-time 15 -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"remoteJid\": \"CONTACT_JID\"}}, \"limit\": 20}' -o /tmp/wa_msgs.json\n ```\n Then in Python: `data = json.load(open('/tmp/wa_msgs.json')); records = data['messages']['records']`\n\n## API Reference\n\nFor full API endpoints (media, contacts, location, buttons) β€” see [references/api.md](references/api.md).\n\n[Skill directory: /opt/data/skills/personal/whatsapp-evolution]\nResolve any relative paths in this skill (e.g. `scripts/foo.js`, `templates/config.yaml`) against that directory, then run them with the terminal tool using the absolute path.\n\n[This skill has supporting files:]\n- references/api.md -> /opt/data/skills/personal/whatsapp-evolution/references/api.md\n- scripts/process_queue.py -> /opt/data/skills/personal/whatsapp-evolution/scripts/process_queue.py\n- scripts/send_text.sh -> /opt/data/skills/personal/whatsapp-evolution/scripts/send_text.sh\n\nLoad any of these with skill_view(name=\"personal/whatsapp-evolution\", file_path=\"\"), or run scripts directly by absolute path (e.g. `node /opt/data/skills/personal/whatsapp-evolution/scripts/foo.js`).\n\nThe user has provided the following instruction alongside the skill invocation: Incoming WhatsApp webhook received.\n\nEvent: messages.upsert\nInstance: n8n-WA\nSender: Shalom Evonne\nSender JID: 120363418526182134@g.us\nFrom me: False\nMessage type: conversation\nMessage: {\n \"event\": \"messages.upsert\",\n \"instance\": \"n8n-WA\",\n \"data\": {\n \"key\": {\n \"remoteJid\": \"120363418526182134@g.us\",\n \"fromMe\": false,\n \"id\": \"3EB04AC47FA8DFA23E754D\",\n \"participant\": \"131503308673050@lid\",\n \"participantAlt\": \"6591887488@s.whatsapp.net\",\n \"addressingMode\": \"lid\"\n },\n \"pushName\": \"Shalom Evonne\",\n \"status\": \"DELIVERY_ACK\",\n \"message\": {\n \"conversation\": \"Good evening Shalom Family!\\n\\nTomorrow is Thursday, Week Even.\\n\\nNaomi timetable:\\n8:00-8:25 Morning Assembly\\n8:25-9:35 D&T\\n9:35-10:10 Recess\\n10:10-10:45 Math\\n10:45-11:20 BL\\n11:20-11:55 MTL\\n11:55-12:30 Lunch\\n12:30-13:05 G2/G3\\n13:05-13:40 CCE and Assembly\\n\\nDismissal: 1:40 PM\\n\\nMay Naomi have a smooth, focused and blessed school day tomorrow.\"\n },\n \"contextInfo\": null,\n \"messageType\": \"conversation\",\n \"messageTimestamp\": 1778670032,\n \"instanceId\": \"b2bfaa30-d5f1-4b22-b979-791892619414\",\n \"source\": \"web\"\n },\n \"destination\": \"https://jimmy.huper.asia/webhooks/whatsapp-incoming\",\n \"date_time\": \"2026-05-13T08:00:32.646Z\",\n \"sender\": \"6590054438@s.whatsapp.net\",\n \"server_url\": \"https://evolution.huper.asia\",\n \"apikey\": \"5E4C532223A0-460B-A749-16C584F56E74\"\n}\n\nPrimary goal: REDUCE NOISE. Stay silent in the main Telegram chat unless the message truly needs Vincent's immediate attention. Approval-worthy reply drafts should go to the jimmy-wa-drafts sidecar, NOT to the main Telegram chat.\n\nFollow the whatsapp-evolution skill and /opt/data/agent-state/memory/whatsapp-vip-contacts.json, with these stricter notification rules:\n\nABSOLUTE SILENCE RULES:\n- If fromMe is true: ignore. Do NOT call send_message. Final response local log only.\n- Ignore status/update/read-receipt events or payloads with no real message content.\n- Do NOT send main Telegram summaries for ordinary DMs, routine logistics, casual family messages, generic group events, group chatter, forwards, memes, devotional/religious shares, links, standalone codes, normal confirmations, or suggested replies.\n- Do NOT ask \"Want me to add this to Calendar?\" for WhatsApp group events unless Vincent is explicitly invited, named, responsible, or the event is time-sensitive and clearly relevant to him today.\n- Do NOT surface family messages just because they are family. Only surface if urgent, safety-related, school/pickup critical, money/legal/medical, or explicitly requires Vincent now.\n\nSIDEcar RULES:\n- If a reply may be useful but it is not urgent, POST it to https://jimmy-draft.huper.asia/drafts using the whatsapp-evolution Draft Approvals flow. Do NOT duplicate that draft/summary in the main Telegram chat.\n- Prayer and birthday replies should go to the sidecar when appropriate, not the main Telegram chat, unless the underlying situation is urgent/sensitive and Vincent needs immediate awareness.\n- Before drafting any reply or approval suggestion, fetch recent context: python3 /opt/data/scripts/wa_recent_messages.py \"120363418526182134@g.us\" 30. For active/ambiguous groups use 50.\n\nMAIN TELEGRAM ALERTS ARE ALLOWED ONLY FOR:\n1. VIP contacts marked always_call or always_notify when the message is important/actionable, or any VIP call event.\n2. Calls from family/VIP that appear missed/ringing and likely need immediate attention. Avoid repeated alerts for accepted/terminated duplicate call state updates.\n3. Emergencies, safety issues, medical issues, school pickup/child logistics requiring action now, legal/money/account/security matters, or time-sensitive commitments in the next few hours.\n4. Messages explicitly asking Vincent to call back, decide now, approve payment, provide OTP/code for an active task, or respond urgently.\n5. Pipeline errors or webhook processing failures.\n\nCALL RULES:\n- VIP contacts (e.g. Dn Jeff, Ng Whye Keong): call Vincent immediately outside quiet hours when actionably important or on ringing/missed call. During 1am-10am SGT, Telegram only unless life-and-death emergency.\n- Do not call for casual VIP confirmations or non-actionable courtesy messages.\n\nTONE/RULES FOR DRAFTS:\n- Proposed replies must be Christian and grace-based where appropriate: warm, humble, encouraging, non-judgmental, peace-making, not preachy.\n- Never use em dashes in drafted replies or approval messages. Use commas, periods, colons, or parentheses instead.\n- NEVER auto-reply directly on WhatsApp unless Vincent explicitly approves or future config marks that exact category pre-approved.\n\nDelivery rule:\n- For allowed main alerts only: call send_message(target=\"telegram\", message=...). Keep it concise.\n- For non-urgent reply suggestions: use the sidecar only.\n- For everything else: DO NOT call send_message. Keep final response as local-only log.\n", "timestamp": 1778670044.0381198, "anchor": true}, {"id": 53494, "role": "assistant", "content": "", "timestamp": 1778670044.058389, "tool_calls": [{"id": "call_e8x0PcxcSHQFLvPHe37CdkEN", "call_id": "call_e8x0PcxcSHQFLvPHe37CdkEN", "response_item_id": "fc_0cfd643e3d42afaa016a0459d5e1788191863ae5853c51c15f", "type": "function", "function": {"name": "read_file", "arguments": "{\"path\":\"/opt/data/agent-state/memory/whatsapp-vip-contacts.json\",\"limit\":500}"}}]}, {"id": 53496, "role": "assistant", "content": "Local log only. Ignored main Telegram.\n\nClassification: routine family/school timetable update for tomorrow, not urgent, no immediate action required, no reply draft needed.", "timestamp": 1778670044.0624557}], "bookend_end": [], "messages_before": 0, "messages_after": 4}, {"session_id": "20260511_110224_b27deb76", "when": "May 11, 2026 at 11:02 AM", "source": "webhook", "model": "gpt-5.5", "title": "WhatsApp School Logistics Update", "matched_role": "user", "match_message_id": 47200, "snippet": "...draft_text\": \"Thanks >>>Karen<<<, will miss you both β€” praying for a good time too πŸ™\",\n \"extras\": {\n \"group_jid\": \"120363351787895353@g.us\",\n \"sender_name\": \">>>Karen<<< >>>Tan<<<\",\n \"sender_jid\": \">>>6597630553<<<@s.whatsapp.net\",\n \"category\": \">>>prayer<<>>blessing<<<\",\n \"original_message\": \"Happy >>>cycling<<<, to those who...", "bookend_start": [], "messages": [{"id": 47200, "role": "user", "content": "[IMPORTANT: The user has invoked the \"whatsapp-evolution\" skill, indicating they want you to follow its instructions. The full skill content is loaded below.]\n\n---\nname: whatsapp-evolution\ndescription: WhatsApp messaging via Evolution API β€” send, receive, analyze, and auto-reply. Use when processing an incoming WhatsApp message delivered via the webhook, replying on Vincent's behalf, liaising with contacts, sending messages, or handling WhatsApp-related tasks. Covers intent analysis, auto-reply rules, escalation (VIP β†’ phone call), conversation context fetching, and draft approvals via the jimmy-wa-drafts sidecar.\n---\n\n# WhatsApp via Evolution API\n\n## Credentials\nStored in `/opt/data/secrets/.whatsapp-evolution-credentials`:\n- `EVOLUTION_API_URL`, `EVOLUTION_API_KEY`, `EVOLUTION_INSTANCE`\n\nLoad with: `export $(cat .whatsapp-evolution-credentials | xargs)`\n\n## Sending Messages (direct, no approval)\n\n```bash\nexport $(cat .whatsapp-evolution-credentials | xargs)\ncurl -s -X POST \"${EVOLUTION_API_URL}/message/sendText/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"number\": \"6591234567\", \"text\": \"Hello!\"}'\n```\n\n- Phone format: country code, no `+` (e.g. `6590054438`)\n- Groups: use JID `{groupid}@g.us`\n- For media/location/contacts β€” see [references/api.md](references/api.md)\n\nUse direct send only when Vincent has explicitly commanded/approved the exact send, or for pre-approved auto-replies/follow-ups. For any reply that needs Vincent's review, use the sidecar (next section).\n\n### Sending a local image/file as media\n\nFor user-approved media sends, base64-encode the local file and call `sendMedia`:\n\n```bash\nexport $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\npython3 - <<'PY'\nimport base64, json\npath = '/tmp/file.png'\npayload = {\n 'number': '6591234567',\n 'mediatype': 'image',\n 'mimetype': 'image/png',\n 'caption': 'Caption here',\n 'media': base64.b64encode(open(path, 'rb').read()).decode(),\n 'fileName': 'file.png',\n}\nopen('/tmp/wa_media_payload.json', 'w').write(json.dumps(payload))\nPY\ncurl -s --max-time 60 -X POST \"${EVOLUTION_API_URL}/message/sendMedia/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_media_payload.json\n```\n\nWhen sending QR tickets or passes, verify the crop/image contains the full QR code and the intended attendee name before sending; the QR screenshot is effectively the redeemable ticket.\n\n## Fetching Conversation Context\n\nBefore drafting or proposing any reply, always fetch recent chat history for context:\n\n```bash\npython3 /opt/data/scripts/wa_recent_messages.py \"6591234567@s.whatsapp.net\" 30\n```\n\nUse `limit=30` by default. For active group chats or ambiguous situations, fetch up to 50 messages. Read enough context to understand who is being discussed, current tone, previous commitments, and whether Vincent has already replied. The script returns newest messages first (descending order).\n\n## Draft Approvals β€” `jimmy-wa-drafts` sidecar\n\nWhen you need Vincent to approve a WhatsApp reply, POST the draft to the sidecar. It owns the entire approval flow: Telegram message with 4 inline buttons, Telegram WebApp for Edit/Context, sending to WhatsApp via Evolution API on Approve.\n\n### Posting a draft\n\nAlways include an `extras` object with the original WA context. The sidecar passes `extras` through verbatim if Vincent later clicks \"Context\" β€” without it, your redraft will be flying blind.\n\n```bash\ncat > /tmp/wa_draft.json <<'JSON'\n{\n \"contact\": \"120363351787895353@g.us\",\n \"contact_name\": \"Cycling group\",\n \"summary\": \"Karen sent a blessing over the bike ride; she's not joining\",\n \"draft_text\": \"Thanks Karen, will miss you both β€” praying for a good time too πŸ™\",\n \"extras\": {\n \"group_jid\": \"120363351787895353@g.us\",\n \"sender_name\": \"Karen Tan\",\n \"sender_jid\": \"6597630553@s.whatsapp.net\",\n \"category\": \"prayer/blessing\",\n \"original_message\": \"Happy cycling, to those who can make it… Dn Jeff and I had made plans for this PM so we won't be able to join y'all. Praying the Lord's favor, protection and good-weather blessings be upon y'all and your route!\",\n \"extracted\": {\n \"what\": \"Cycling from East Coast Park to Changi Village, then dinner\",\n \"when\": \"Today, 4:00 PM\",\n \"where\": \"East Coast Park to Changi Village\",\n \"who\": \"Group members who voted yes for cycling/dinner or dinner\"\n },\n \"confidence\": \"high\",\n \"why_appropriate\": \"Multiple group messages are blessings/wishes; Vincent has not replied; tone matches grace-based norm.\",\n \"wa_message_id\": \"ABC123XYZ\"\n }\n}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_draft.json\n```\n\nResponse: `{\"draft_id\":\"abc123def\",\"telegram_message_id\":42}`\n\nWhat to put in `extras` (use whatever makes sense for the message type):\n\n- **Always**: `original_message` (verbatim WA text), `sender_name`, `category` (e.g. `prayer`, `birthday`, `event`, `dm-reply`, `enquiry`)\n- **Group messages**: `group_jid`, `sender_jid`\n- **DMs**: `sender_jid`\n- **Strongly recommended (drives the rich Telegram render)**: `extracted` (dict of who/what/when/where), `confidence` (`high`/`medium`/`low`), `why_appropriate` (one-line reasoning shown to Vincent so he can decide)\n- **Optional**: `wa_message_id`, `tone_hint` (e.g. `warm-grace-based`), `sender_phone` (auto-derived from `sender_jid` if omitted)\n\nVincent receives a Telegram message in the `@jimmy_WA_bot` chat with 4 inline buttons. The sidecar handles every interaction; **you do not track replies, callbacks, or message IDs**. Just POST and move on.\n\n| Button | Behavior |\n|--------|----------|\n| **Approve** | Sidecar sends draft to WhatsApp |\n| **Edit** | Opens a Telegram WebApp with draft pre-filled; Vincent edits + sends; sidecar pushes to WhatsApp |\n| **Context** | Opens a Telegram WebApp with empty textarea + the original WA message visible; Vincent provides context. (When `HERMES_REDRAFT_WEBHOOK` is wired in the sidecar's env, this will POST back to Jimmy for a redraft. Until then, the context is just stored on the draft.) |\n| **Pass** | Sidecar deletes the draft |\n\n### JSON escaping when `draft_text` contains quotes\n\nSingle quotes (`It's`, `God's`) break shell interpolation. Write the JSON to a temp file first:\n\n```bash\ncat > /tmp/wa_draft.json <<'JSON'\n{\"contact\":\"6591234567@s.whatsapp.net\",\"contact_name\":\"John\",\"summary\":\"…\",\"draft_text\":\"It's nice to hear from you!\"}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_draft.json\n```\n\n### Networking\n\n`https://jimmy-draft.huper.asia` is the public HTTPS endpoint of the sidecar (TLS-terminated by Traefik on ubuntu-server). HTTPS is required because Hermes blocks plain-HTTP commands via tirith. The internal Docker DNS hostname http://jimmy-wa-v64mqe:8080 also works for non-shell calls but the agent is forbidden from using it.\n\n### Sidecar status & logs\n\n- Source: (auto-deploys on push)\n- Public URL: (`/health` for ping)\n- Container: `jimmy-wa-v64mqe.1.` on ubuntu-server\n- Bot: `@jimmy_WA_bot`\n\n## Handling a Redraft Trigger (`wa-redraft` webhook)\n\nWhen Vincent clicks the **Context** button on a draft, the sidecar:\n1. Edits the original Telegram message to \"πŸ”„ Adding context for X… agent is redrafting.\" (removes the old buttons so Vincent can't approve the now-stale text)\n2. POSTs to your `wa-redraft` webhook (`http://localhost:8644/webhooks/wa-redraft`) with the original draft fields plus Vincent's `context` instructions and the `extras` you stored when you first created the draft.\n\nWhen you receive a `wa-redraft` activation, the prompt template will lay out the fields. Your job:\n\n1. Read Vincent's `context` carefully β€” that's the steering signal. The `previous_draft` is just history, not necessarily a starting point.\n2. Re-fetch recent WA history if the original context isn't enough (`python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`). Use `extras.group_jid` for groups or `contact` for DMs.\n3. Write a NEW draft that addresses Vincent's instructions while staying faithful to the original WA context (in `extras.original_message`, `extras.extracted`).\n4. POST the new draft back to the sidecar **with the same `draft_id`** so the existing draft is updated, not duplicated:\n\n```bash\ncat > /tmp/wa_redraft.json <\",\n \"contact\": \"\",\n \"contact_name\": \"\",\n \"summary\": \"\",\n \"draft_text\": \"\",\n \"extras\": \n}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_redraft.json\n```\n\nThe sidecar will send a NEW Telegram message with 4 fresh buttons. Vincent can Approve, Edit, Context-again (loop), or Pass.\n\nDo **not** send to WhatsApp directly during a redraft β€” the approval flow is the sidecar's job.\n\n## Incoming Message Processing\n\n### Delivery Pipeline (current)\n1. Evolution API POSTs each `MESSAGES_UPSERT` webhook directly to Hermes at `http://localhost:8644/webhooks/whatsapp-incoming` (proxied via `jimmy.huper.asia/webhooks/whatsapp-incoming`)\n2. Hermes webhook route (`whatsapp-incoming`, dynamic subscription) validates and spawns an agent\n3. Agent loads `whatsapp-evolution` skill, follows the rules below\n4. Webhook subscription uses `deliver: log` because webhooks do not support `deliver: local`; the agent explicitly calls `send_message(target=\"telegram\", ...)` only for important/actionable/VIP/event/birthday/prayer/error items, OR posts a draft to the sidecar for approval items.\n\n**No more OpenClaw, no Python webhook server, no port 5001, no keepalive cron.** If messages aren't being processed, check: nginx proxy to port 8644, `hermes webhook list` for the subscription, Evolution API webhook URL setting.\n\n## Contact Rules & Escalation\n\nConfig: `memory/whatsapp-vip-contacts.json`\n\n### Priority Levels\n\n| Priority | Action | Example |\n|----------|--------|---------|\n| **VIP (call-always)** | **CALL Vincent immediately via Twilio without asking.** Do NOT ask for approval first. During quiet hours (1am-10am SGT) suppress call and use Telegram notification only, unless life-and-death emergency. | Dn Jeff (+6597630553) |\n| **Family** | Always notify Vincent on Telegram | ps112 group, family members |\n| **Auto-reply** | Reply based on context file, notify Vincent | Repair shops |\n| **Normal** | Analyze intent, reply if appropriate, notify if important | Everyone else |\n| **Broadcast/Spam** | Ignore silently | Forwarded job posts, chain messages |\n\n### Quiet Hours\n- 1am-10am SGT (17:00-02:00 UTC): avoid phone calls, including for Dn Jeff / VIP call-always contacts\n- **Exception:** life-and-death emergencies always call regardless of quiet hours\n- During quiet hours, use Telegram notification only and mention that a call was suppressed due to quiet hours\n\n## Making a Call via Twilio\n\nCredentials: `/opt/data/secrets/.twilio-credentials-new` (`TWILIO_ACCOUNT_SID`, `TWILIO_AUTH_TOKEN`, `TWILIO_PHONE_NUMBER`). From: `+16614809287`, Vincent: `+6590054438`.\n\nPreferred script: `/opt/data/scripts/call_vincent.py`, now using the Twilio Python SDK baked into the container.\n\n```bash\npython3 /opt/data/scripts/call_vincent.py \"Hello Vincent, this is Jimmy. Urgent summary here.\"\n```\n\n## Action Rules\n\n### Default: DO NOT auto-reply directly\nCollect intent and stay local unless something needs Vincent.\n\n**When you need Vincent to approve a reply, POST a draft to the sidecar (see \"Draft Approvals\" above).** The sidecar handles the Telegram notification with buttons, the WebApp for editing, and sending to WhatsApp on approve.\n\nDo NOT send WhatsApp replies directly unless Vincent explicitly approves them or the category is pre-approved.\n\n### Tone (when proposing or sending replies)\n- Casual, direct, no sign-offs (no \"Thanks & Regards\", no \"- Vincent\")\n- Ask questions instead of giving instructions\n- Keep it natural β€” like texting a friend\n- Christian and grace-based: warm, humble, encouraging, non-judgmental, peace-making\n- Never use em dashes in drafted replies or approval messages. Use commas, periods, colons, or parentheses instead.\n- For prayer/birthday contexts, use gentle faith-filled language; avoid sounding preachy, performative, or overly formal\n\n### What to Do With Each Message\n\n**DMs (personal chats):**\n- Analyze intent\n- Notify Vincent on Telegram (via `send_message(target=\"telegram\", ...)`) only if important/actionable/time-sensitive, requires a decision, comes from a VIP contact\n- If you have a useful suggested reply, POST it as a draft to the sidecar instead of a Telegram notification β€” Vincent gets the buttons inline\n- Casual/social/non-actionable DMs β†’ local log only, no Telegram notification, no draft\n- Do NOT reply unless Vincent says to\n- VIP contacts β†’ **call Vincent immediately without asking**. Do NOT post a draft. Just call. During quiet hours (1am-10am SGT), suppress the call and send Telegram notification only, unless life-and-death emergency.\n\n**Group chats β€” events/meetups:**\n- If someone posts about an event, gathering, meetup, dinner, etc:\n 1. Extract: what, when, where, who\n 2. Notify Vincent on Telegram with details (no draft needed unless you also want to suggest an RSVP reply)\n 3. Ask: \"Want me to add this to Google Calendar and set a reminder?\"\n\n**Group chats β€” prayer requests / praying for others:**\n- If someone asks for prayer, shares a burden, illness, grief, struggle, thanksgiving, or asks the group to pray:\n 1. Fetch recent group context first: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n 2. Identify who/what to pray for and any sensitive details\n 3. Draft a short, grace-based Christian reply/prayer\n 4. POST it to the sidecar with `summary` describing the request and `draft_text` being the prayer\n 5. Do NOT post to WhatsApp unless Vincent approves\n\n**Group chats β€” birthday greetings:**\n- If you see birthday wishes for someone in a group:\n 1. Fetch recent group context first: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n 2. Note who the birthday person is and relationship/context if inferable\n 3. Draft a warm, grace-based Christian birthday greeting\n 4. POST it to the sidecar\n 5. Do NOT post to WhatsApp unless Vincent approves\n\n**Group chats β€” general:**\n- Casual banter, memes, forwards, broadcast spam β†’ ignore silently\n- Religious content shares β†’ ignore unless specifically addressed to Vincent\n\n### When to Notify Vincent (Telegram) vs Post a Draft\n\n**Use `send_message(target=\"telegram\", ...)`** for plain notifications: VIP DMs, events/meetups summaries, errors in the pipeline, group context that doesn't need a reply.\n\n**Use the sidecar (POST `/drafts`)** whenever you have a suggested reply for Vincent to approve: prayer responses, birthday greetings, DM replies, group RSVPs.\n\nDo **not** notify or draft for casual/social/non-actionable DMs, read receipts/status updates, religious shares not addressed to Vincent, memes, forwarded/broadcast/spam, or general group chatter.\n\n## Intent Categories\n\nWhen analyzing incoming messages, classify as:\n\n1. **Urgent** β€” time-sensitive, emergencies β†’ call Vincent outside quiet hours; during quiet hours use Telegram only unless life-and-death\n2. **Important** β€” decisions, money, appointments, anything consequential β†’ call Vincent outside quiet hours; during quiet hours notify on Telegram only unless life-threatening\n3. **Request** β€” asking Vincent to do something β†’ **notify Vincent** (Telegram) or post a draft if you have a safe reply\n4. **Enquiry** β€” asking for info, pricing, availability β†’ auto-reply if context available, otherwise post a draft\n5. **Confirmation** β€” confirming plans, bookings β†’ acknowledge and notify Vincent\n6. **Casual/Social** β€” greetings, memes, forwards β†’ usually ignore or react\n7. **Broadcast/Spam** β€” forwarded job posts, chain messages β†’ ignore\n\n## Finding Group JIDs\n\nThe `group/fetchGroups` endpoint returns 404 on some Evolution API instances (including ours). To find a group JID when you don't have it:\n\n1. **Use `chat/findMessages` with an empty `where` filter** to pull recent messages across all chats, then extract unique `remoteJid` values containing `@g.us`.\n2. **Match by content**: Fetch messages from each group JID using `where.key.remoteJid` and match sender names or message text to identify the target group.\n3. **Known group JIDs** (store in `whatsapp-vip-contacts.json` as you discover them):\n - ps112 family group: `6591887488-1464061170@g.us`\n - Overcomers group: `6591444911-1614821287@g.us`\n - η₯žηš„εŽηΎŽ group: `6596756578-1405873840@g.us` (content-sharing, casual/broadcast, local log only)\n - Overcomers/TCians group: `120363230236878544@g.us` (church fellowship, PJ sends weekly attendance reports, routine attendance = local log only)\n\nExample search:\n```bash\ncurl -s -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"remoteJid\": \"6591444911-1614821287@g.us\"}}, \"limit\": 10, \"order\": \"desc\"}'\n```\n\n## Processing Image/Media Messages\n\nWhen an incoming WhatsApp message is an image (imageMessage), video, or other media with no text or only a caption:\n\n1. **Fetch conversation context** as usual with `wa_recent_messages.py`\n2. **Download the media** using the Evolution API:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"message\": {\"key\": {\"id\": \"MESSAGE_ID\", \"fromMe\": false, \"remoteJid\": \"GROUP_OR_CHAT_JID\"}}, \"convertToMp4\": false}' \\\n -o /tmp/wa_media_response.json\n ```\n Then decode: `python3 -c \"import json,base64; d=json.load(open('/tmp/wa_media_response.json')); open('/tmp/wa_media.jpg','wb').write(base64.b64decode(d['base64']))\"`\n3. **Analyze the image** using `vision_analyze` tool with `/tmp/wa_media.jpg` path and a context-aware question\n4. **Fallback if vision is unavailable**: Use `tesseract /tmp/wa_media.jpg stdout` for OCR, combined with pixel analysis (PIL/numpy) and conversation context to infer what the image contains\n5. **Album messages** (`albumMessage`): These bundle multiple images. Each image arrives as a separate `imageMessage` with a `messageAssociation.parentMessageKey` linking them to the album. Download each image separately. The `expectedImageCount` field tells how many images to expect\n6. **messageAssociation**: When images have `parentMessageKey` with `fromMe: true`, they are replying to a message Vincent sent. Use this for context\n\n### Audio / Voice Note Messages\nWhen an incoming WhatsApp message is `audioMessage` or a voice note (`ptt: true`), treat it like other media, but transcribe before intent classification when possible:\n\n1. **Fetch conversation context first**, especially in groups:\n ```bash\n python3 /opt/data/scripts/wa_recent_messages.py \"\" 50\n ```\n2. **Download the audio** with Evolution API, saving the response to a file:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n cat > /tmp/wa_audio_req.json <<'JSON'\n {\"message\":{\"key\":{\"id\":\"MESSAGE_ID\",\"fromMe\":false,\"remoteJid\":\"GROUP_OR_CHAT_JID\"}},\"convertToMp4\":false}\n JSON\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_audio_req.json -o /tmp/wa_audio_response.json\n python3 - <<'PY'\n import json, base64\n d=json.load(open('/tmp/wa_audio_response.json'))\n b=d.get('base64') or d.get('data',{}).get('base64')\n if ',' in b[:100]: b=b.split(',',1)[1]\n open('/tmp/wa_audio.ogg','wb').write(base64.b64decode(b))\n PY\n ```\n3. **Convert OGG/Opus to WAV**:\n ```bash\n ffmpeg -y -i /tmp/wa_audio.ogg -ar 16000 -ac 1 /tmp/wa_audio.wav\n ```\n4. **Transcribe fallback when Whisper is unavailable**: the container may not have `whisper`, `faster-whisper`, `torch`, or `speech_recognition` installed globally. A reliable lightweight fallback is a temp venv with SpeechRecognition and Google web speech:\n ```bash\n python3 -m venv /tmp/srvenv\n /tmp/srvenv/bin/pip install --quiet SpeechRecognition\n /tmp/srvenv/bin/python - <<'PY'\n import speech_recognition as sr\n r=sr.Recognizer()\n with sr.AudioFile('/tmp/wa_audio.wav') as source:\n audio=r.record(source)\n print(r.recognize_google(audio, language='en-SG'))\n PY\n ```\n5. **Check adjacent voice notes from the same sender**. WhatsApp voice notes are often sent in quick sequential chunks. Download and transcribe nearby `audioMessage` records from the same sender before deciding whether to notify Vincent.\n6. Use the transcript plus recent context to classify the message. For group meetup planning, extract what/when/where/who and notify Vincent if actionable, including transcript confidence.\n\n### Video Messages\n\nWhen an incoming WhatsApp message is `videoMessage` (with or without a caption):\n\n1. **Fetch conversation context** first, as with any media message: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n2. **Download the video** using Evolution API (same pattern as images/audio β€” save response to file, then decode):\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n cat > /tmp/wa_video_req.json <<'JSON'\n {\"message\":{\"key\":{\"id\":\"MESSAGE_ID\",\"fromMe\":false,\"remoteJid\":\"CONTACT_JID\"}},\"convertToMp4\":false}\n JSON\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_video_req.json -o /tmp/wa_video_response.json\n python3 - <<'PY'\n import json, base64\n d = json.load(open('/tmp/wa_video_response.json'))\n b = d.get('base64') or d.get('data', {}).get('base64', '')\n if ',' in b[:100]: b = b.split(',', 1)[1]\n open('/tmp/wa_video.mp4', 'wb').write(base64.b64decode(b))\n print(f\"Video saved: {len(base64.b64decode(b))} bytes\")\n PY\n ```\n3. **Extract representative frames** with ffmpeg (do NOT try to analyze video directly β€” vision tools work on images):\n ```bash\n # Frame at start\n ffmpeg -y -ss 0 -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame0.jpg\n # Frame at ~1/3 of the way (good for short videos)\n ffmpeg -y -ss 3 -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame3s.jpg\n # If video is longer (>15s), also extract at midpoint\n # ffprobe -v quiet -print_format json -show_format /tmp/wa_video.mp4 # get duration\n # ffmpeg -y -ss $HALF_DURATION -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame_mid.jpg\n ```\n4. **Analyze the frames** with `vision_analyze`. Ask specific questions about content: what objects/people/text are visible, what type of video this appears to be (screen recording, personal clip, forwarded meme, etc.), and any text or identifying details.\n5. **Use the caption** from `videoMessage.caption` if present (can be empty/null). Combine frame analysis + caption + conversation context to classify the message intent.\n6. **Transcription of audio in video** is generally not feasible with available tools. If the video likely contains important spoken content (e.g. voice memo style), note this limitation and flag it to Vincent.\n\n**Key points:**\n- Always use `-o` flag and file-based decode; videos can be 1-10MB and base64 responses will be even larger\n- `--max-time 25` on curl is essential for video downloads\n- Short casual clips (showing a key fob, a parking spot, quick view of surroundings) are usually casual/social β€” classify accordingly and don't over-escalate\n- If `vision_analyze` fails (400 on models that don't support images, or 429 rate limit), fall back to conversation context inference alone\n\n### Poll Messages\nWhen `pollUpdateMessage` appears in chat history (someone voted), the message contains a `pollCreationMessageKey` with the original poll's message `id`. To get the poll question and options, query:\n```bash\ncurl -s -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"id\": \"POLL_MESSAGE_ID\", \"remoteJid\": \"GROUP_JID\"}}, \"limit\": 1}'\n```\nThe response will have `pollCreationMessageV3` with `name` (the poll question) and `options` array. Always look up the poll when votes or poll references appear, since the poll text contains the event details.\n\n### `associatedChildMessage` (Motion Photos / Companion Media)\nWhatsApp motion photos and companion media arrive as `associatedChildMessage` type. These are short videos or supplementary images attached to a parent message via `messageAssociation.parentMessageKey`. The structure is:\n\n```\nmessage: {\n messageContextInfo: {\n messageAssociation: {\n associationType: 12,\n parentMessageKey: { id, fromMe, remoteJid }\n }\n },\n associatedChildMessage: {\n message: {\n videoMessage: { ... } // or imageMessage: { ... }\n }\n }\n}\n```\n\nKey points:\n- The `parentMessageKey.id` links back to the main message (the image/video they accompany)\n- When `parentMessageKey.fromMe: true`, the companion media is replying to a message Vincent sent\n- Captions may appear on the child's `videoMessage.caption` or `imageMessage.caption` inside the nested message, NOT at the top level\n- These often contain motion-photo short videos (3 seconds) or album companion images\n- Treat them as media attachments to the parent; don't process them as standalone messages requiring independent intent analysis\n\n### Sequential Image Sets (Non-Album)\nMultiple images sent rapidly by the same sender at similar timestamps may NOT be wrapped in an `albumMessage`. They appear as separate `imageMessage` entries. Always check for adjacent images from the same sender and download+OCR all of them together, as they often form a coherent multi-page set (e.g., route info, comparison tables, maps). Missing one can lose important context.\n\n### Key pitfalls\n- The Evolution API `getBase64FromMediaMessage` endpoint returns the full base64 payload (can be 100KB+ for images). Use `-o` flag to save to file, then parse with Python. Do NOT try to pipe it directly\n- `curl` with `--max-time 25` is essential; without it, large media responses can hang\n- The `albumMessage` type itself has no `fromMe` field; check each individual `imageMessage` instead\n- Image thumbnails in the webhook payload are truncated; always download the full image for analysis\n- **WhatsApp media URLs expire**: CDN download links (`mmg.whatsapp.net`) expire after roughly 24-48 hours. When `getBase64FromMediaMessage` returns a 400/404 error like \"Failed to fetch stream from https://mmg.whatsapp.net/...\", the media is no longer retrievable via the API. This means you CANNOT download adjacent/older voice notes, images, or documents for context if they're more than a day or two old. Fall back to conversation text context only. Fresh/incoming messages are almost always still downloadable.\n- **Voice note transcription reliability**: Short voice notes (<15s), especially in Singlish/Chinese-English mix common in Singapore contacts, often produce unreliable transcriptions via SpeechRecognition+Google. Always flag low-confidence transcriptions to Vincent rather than acting on potentially garbled text.\n- **Vision API failures**: `vision_analyze` can fail with 429 (rate limit) **or 400** (model doesn't support image input, e.g. glm-5.1 via ollama-cloud). Always have a fallback plan. When vision is unavailable, try tesseract OCR, then pixel analysis (PIL/numpy for brightness zones, text region detection via row-level dark-pixel density)\n- **OCR can fail on decorative/complex images**: Obituaries, memorial notices, religious flyers with ornate fonts, or dark-background images often defeat tesseract even with aggressive preprocessing (upscaling, thresholding, inverting, PSM mode variations). When OCR returns gibberish or empty, fall back to conversation context inference: use preceding/surrounding messages to determine what the image likely contains, then draft an appropriate response based on that context\n- **OCR on devotional/dark-background images**: These often defeat tesseract on full-image scans. Use a multi-strategy approach: (1) run tesseract on the full image first, (2) if text is sparse, crop into regions (top 15%, middle, bottom 15%), upscale 2-3x, enhance contrast (2-3x), and try both normal and inverted (`ImageOps.invert`) versions with `--psm 6`, (3) combine all partial OCR results. For Exodus/verse images, the title (e.g. \"WHEN MOSES' HANDS WERE\") is often in decorative font that OCR misses, while the verse body text is usually readable.\n- **Prioritize conversation context**: When processing group images, always fetch recent context FIRST (before attempting vision/OCR). If image analysis fails, the conversation history often makes the image's purpose clear enough to draft a meaningful reply\n- **DM conversation context is one-sided (fromMe only)**: The Evolution API `findMessages` endpoint often returns only Vincent's sent messages (`fromMe: true`) for DM chats, not the contact's incoming messages. This means `wa_recent_messages.py` can show only one side of the conversation. When processing a DM webhook, do NOT rely solely on fetched history for the contact's side. The webhook payload already contains the incoming message text. Use what the webhook gives you plus Vincent's side from the API. If you need the contact's recent messages, check if `fromMe: false` records exist; if none are returned, the API simply doesn't have them and you must work with what you have (webhook message + Vincent's replies for context).\n- **`findMessages` ignores `key.fromMe` filter**: Passing `\"key.fromMe\": false` in the `where` clause still returns all messages (including `fromMe: true`). To find messages from a specific sender, fetch results and filter client-side by checking `fromMe` in Python.\n- **`findMessages` caps at ~50 records per request**: Even with `limit: 200` or `limit: 500`, the API returns at most 50 records. When one party sends many consecutive messages, the other party's messages may not appear in the first 50 results. Use pagination or fetch multiple pages if you need older messages.\n- **Video frame extraction**: Use `ffmpeg -y -ss N -i video.mp4 -frames:v 1 -update 1 output.jpg` (the `-update 1` flag is needed; without it ffmpeg treats the output as an image sequence and fails). Extract frames at multiple timestamps (start, ~1/3, midpoint for longer videos) since a single frame may miss the key content.\n- **ffmpeg \"Invalid color space\" failure on WhatsApp videos**: Some WhatsApp H.264 videos have `color_space=reserved` and `color_primaries=reserved` metadata that ffmpeg cannot decode at all. Every codec/format/filter combination fails with \"Invalid color space\" / return code 234. This is NOT fixable with `-vf setparams`, `-color_trc`, `-colorspace`, `-color_primaries`, or `-vf format=` flags because the error occurs at the decoder level before any filter runs. **Do not waste time trying ffmpeg workarounds.** When this happens, fall back to: (1) conversation context inference (who sent it, what thread it's in, adjacent messages), (2) the jpegThumbnail embedded in the videoMessage (small but gives a rough idea; base64-encoded JPEG, typically 600-900 bytes, use `vision_analyze` on it if available), or (3) if all analysis fails, just classify based on surrounding text context.\n- **`wa_recent_messages.py` JSON output breaks `json.loads()`**: The script's stdout contains WhatsApp message content with control characters (emoji, CJK, deviceListMetadata binary blobs) that cause `json.loads()` to fail with `Invalid control character` or `Expecting ',' delimiter` errors. Even `strict=False` does not reliably fix it. **Reliable approach**: use `curl -o /tmp/wa_msgs.json` to save the Evolution API response directly to a file (the file write handles binary safely), then parse with `json.load(open('/tmp/wa_msgs.json'))` in Python. Example:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n curl -s --max-time 15 -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"remoteJid\": \"CONTACT_JID\"}}, \"limit\": 20}' -o /tmp/wa_msgs.json\n ```\n Then in Python: `data = json.load(open('/tmp/wa_msgs.json')); records = data['messages']['records']`\n\n## API Reference\n\nFor full API endpoints (media, contacts, location, buttons) β€” see [references/api.md](references/api.md).\n\n[Skill directory: /opt/data/skills/personal/whatsapp-evolution]\nResolve any relative paths in this skill (e.g. `scripts/foo.js`, `templates/config.yaml`) against that directory, then run them with the terminal tool using the absolute path.\n\n[This skill has supporting files:]\n- references/api.md -> /opt/data/skills/personal/whatsapp-evolution/references/api.md\n- scripts/process_queue.py -> /opt/data/skills/personal/whatsapp-evolution/scripts/process_queue.py\n- scripts/send_text.sh -> /opt/data/skills/personal/whatsapp-evolution/scripts/send_text.sh\n\nLoad any of these with skill_view(name=\"personal/whatsapp-evolution\", file_path=\"\"), or run scripts directly by absolute path (e.g. `node /opt/data/skills/personal/whatsapp-evolution/scripts/foo.js`).\n\nThe user has provided the following instruction alongside the skill invocation: Incoming WhatsApp webhook received.\n\nEvent: messages.upsert\nInstance: n8n-WA\nSender: Shalom Evonne\nSender JID: 120363418526182134@g.us\nFrom me: False\nMessage type: conversation\nMessage: {\n \"event\": \"messages.upsert\",\n \"instance\": \"n8n-WA\",\n \"data\": {\n \"key\": {\n \"remoteJid\": \"120363418526182134@g.us\",\n \"fromMe\": false,\n \"id\": \"3EB043E18337964BA6A6B4\",\n \"participant\": \"131503308673050@lid\",\n \"participantAlt\": \"6591887488@s.whatsapp.net\",\n \"addressingMode\": \"lid\"\n },\n \"pushName\": \"Shalom Evonne\",\n \"status\": \"DELIVERY_ACK\",\n \"message\": {\n \"conversation\": \"Good evening Shalom Family!\\n\\nTomorrow is Tuesday, Week Even.\\n\\nNaomi timetable:\\n8:00-8:25 Morning Assembly\\n8:25-9:00 Science\\n9:00-9:35 Science\\n9:35-10:10 Recess\\n10:10-10:45 English\\n10:45-11:20 English\\n11:20-11:55 Art\\n11:55-12:30 Art\\n12:30-1:05 D&T\\n1:05-1:40 Lunch\\n1:40-2:15 G2/G3 Humanities\\n\\nDismissal time: 2:15 PM\\n\\nHave a blessed evening and a smooth school day tomorrow. May the Lord guide Naomi with wisdom, peace, and joy.\"\n },\n \"contextInfo\": null,\n \"messageType\": \"conversation\",\n \"messageTimestamp\": 1778497343,\n \"instanceId\": \"b2bfaa30-d5f1-4b22-b979-791892619414\",\n \"source\": \"web\"\n },\n \"destination\": \"https://jimmy.huper.asia/webhooks/whatsapp-incoming\",\n \"date_time\": \"2026-05-11T08:02:23.932Z\",\n \"sender\": \"6590054438@s.whatsapp.net\",\n \"server_url\": \"https://evolution.huper.asia\",\n \"apikey\": \"5E4C532223A0-460B-A749-16C584F56E74\"\n}\n\nGoal: stay silent unless the WhatsApp message is important/actionable OR there is a safe/useful reply draft Vincent should approve. For a start, surface group prayer/praying-for-others messages and group birthday wishes with proposed replies for approval.\n\nFollow the whatsapp-evolution skill and /opt/data/agent-state/memory/whatsapp-vip-contacts.json:\n- If fromMe is true: ignore. Do NOT call send_message. Final response local log only.\n- Ignore status/update/read-receipt events or payloads with no real message content. Do NOT notify Telegram.\n- Analyze intent: Urgent/Important/Request/Enquiry/Confirmation/Prayer/Birthday/Casual/Broadcast-Spam.\n- Before drafting any reply or approval suggestion, fetch recent context: python3 /opt/data/scripts/wa_recent_messages.py \"120363418526182134@g.us\" 30. For active/ambiguous groups use 50.\n- All proposed replies must be Christian and grace-based: warm, humble, encouraging, non-judgmental, peace-making, not preachy.\n- Never use em dashes in drafted replies or approval messages. Use commas, periods, colons, or parentheses instead.\n- Group prayer/praying-for-others: if someone asks for prayer, shares illness/grief/burden/thanksgiving, or invites prayer, fetch context and send Telegram approval request with summary + suggested short prayer/reply + confidence + why appropriate.\n- Group birthday wishes: if someone is being wished happy birthday, fetch context and send Telegram approval request with birthday person + suggested warm grace-based greeting + confidence.\n- If you think you can auto-reply to any DM/group message: DO NOT send WhatsApp reply. Instead call send_message(target=\"telegram\", message=...) with sender/group, context summary, suggested reply text, confidence, and ask Vincent to approve/edit/reject.\n- VIP contacts, including Dn Jeff: notify Telegram; outside quiet hours call Vincent if actionably important. During 1am-10am SGT, Telegram only unless life-and-death emergency.\n- Group events/meetups: extract what/when/where/who and notify Telegram.\n- Casual/social, forwarded content, religious shares not requiring reply, spam, broadcasts, normal receipts/read statuses: local log only unless there is a useful reply suggestion.\n- NEVER auto-reply directly on WhatsApp unless Vincent explicitly approves or future config marks that exact category pre-approved.\n\nDelivery rule:\n- For important/actionable/VIP/event/birthday/prayer/error OR suggested auto-reply: call send_message(target=\"telegram\", message=...) yourself.\n- For non-important items with no useful reply candidate: DO NOT call send_message. Keep final response as local-only log.\n", "timestamp": 1778497376.748115, "anchor": true}, {"id": 47201, "role": "assistant", "content": "", "timestamp": 1778497376.7648995, "tool_calls": [{"id": "call_jx0ISrom89SgZ70LZBlQ3GC2", "call_id": "call_jx0ISrom89SgZ70LZBlQ3GC2", "response_item_id": "fc_08fa660b5af7d54b016a01b74629808191b6ade16b07556349", "type": "function", "function": {"name": "read_file", "arguments": "{\"path\":\"/opt/data/agent-state/memory/whatsapp-vip-contacts.json\",\"limit\":500}"}}]}, {"id": 47203, "role": "assistant", "content": "", "timestamp": 1778497376.768603, "tool_calls": [{"id": "call_EkGzQf3hCEvTdMojyL4tvpPk", "call_id": "call_EkGzQf3hCEvTdMojyL4tvpPk", "response_item_id": "fc_04116c034b9a29dc016a01b750eccc81918c978c0fc0b3fe9a", "type": "function", "function": {"name": "terminal", "arguments": "{\"command\":\"python3 /opt/data/scripts/wa_recent_messages.py \\\"120363418526182134@g.us\\\" 30\",\"timeout\":60}"}}]}, {"id": 47205, "role": "assistant", "content": "", "timestamp": 1778497376.8870556, "tool_calls": [{"id": "call_pXLvzmgpFk9ovzZmDUXBrTDT", "call_id": "call_pXLvzmgpFk9ovzZmDUXBrTDT", "response_item_id": "fc_0a6311ab9d6286c7016a01b75bd6d48191be7e4b628b188956", "type": "function", "function": {"name": "send_message", "arguments": "{\"target\":\"telegram\",\"message\":\"WhatsApp family/group update from Shalom Evonne about Naomi’s school day tomorrow.\\n\\nNaomi timetable, Tuesday Week Even:\\nβ€’ Dismissal: 2:15 PM\\nβ€’ Main subjects: Science, English, Art, D&T, G2/G3 Humanities\\nβ€’ No PE reminder mentioned\\n\\nNo reply needed.\"}"}}]}], "bookend_end": [{"id": 47207, "role": "assistant", "content": "Local log: Processed incoming WhatsApp group message from Shalom Evonne. Classified as actionable family/school logistics, notified Vincent on Telegram with Naomi’s timetable and dismissal time.", "timestamp": 1778497376.8894646}], "messages_before": 0, "messages_after": 5}, {"session_id": "20260511_110208_bf5a92a2", "when": "May 11, 2026 at 11:02 AM", "source": "webhook", "model": "gpt-5.5", "title": "WhatsApp School Logistics Notification", "matched_role": "user", "match_message_id": 47190, "snippet": "...draft_text\": \"Thanks >>>Karen<<<, will miss you both β€” praying for a good time too πŸ™\",\n \"extras\": {\n \"group_jid\": \"120363351787895353@g.us\",\n \"sender_name\": \">>>Karen<<< >>>Tan<<<\",\n \"sender_jid\": \">>>6597630553<<<@s.whatsapp.net\",\n \"category\": \">>>prayer<<>>blessing<<<\",\n \"original_message\": \"Happy >>>cycling<<<, to those who...", "bookend_start": [], "messages": [{"id": 47190, "role": "user", "content": "[IMPORTANT: The user has invoked the \"whatsapp-evolution\" skill, indicating they want you to follow its instructions. The full skill content is loaded below.]\n\n---\nname: whatsapp-evolution\ndescription: WhatsApp messaging via Evolution API β€” send, receive, analyze, and auto-reply. Use when processing an incoming WhatsApp message delivered via the webhook, replying on Vincent's behalf, liaising with contacts, sending messages, or handling WhatsApp-related tasks. Covers intent analysis, auto-reply rules, escalation (VIP β†’ phone call), conversation context fetching, and draft approvals via the jimmy-wa-drafts sidecar.\n---\n\n# WhatsApp via Evolution API\n\n## Credentials\nStored in `/opt/data/secrets/.whatsapp-evolution-credentials`:\n- `EVOLUTION_API_URL`, `EVOLUTION_API_KEY`, `EVOLUTION_INSTANCE`\n\nLoad with: `export $(cat .whatsapp-evolution-credentials | xargs)`\n\n## Sending Messages (direct, no approval)\n\n```bash\nexport $(cat .whatsapp-evolution-credentials | xargs)\ncurl -s -X POST \"${EVOLUTION_API_URL}/message/sendText/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"number\": \"6591234567\", \"text\": \"Hello!\"}'\n```\n\n- Phone format: country code, no `+` (e.g. `6590054438`)\n- Groups: use JID `{groupid}@g.us`\n- For media/location/contacts β€” see [references/api.md](references/api.md)\n\nUse direct send only when Vincent has explicitly commanded/approved the exact send, or for pre-approved auto-replies/follow-ups. For any reply that needs Vincent's review, use the sidecar (next section).\n\n### Sending a local image/file as media\n\nFor user-approved media sends, base64-encode the local file and call `sendMedia`:\n\n```bash\nexport $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\npython3 - <<'PY'\nimport base64, json\npath = '/tmp/file.png'\npayload = {\n 'number': '6591234567',\n 'mediatype': 'image',\n 'mimetype': 'image/png',\n 'caption': 'Caption here',\n 'media': base64.b64encode(open(path, 'rb').read()).decode(),\n 'fileName': 'file.png',\n}\nopen('/tmp/wa_media_payload.json', 'w').write(json.dumps(payload))\nPY\ncurl -s --max-time 60 -X POST \"${EVOLUTION_API_URL}/message/sendMedia/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_media_payload.json\n```\n\nWhen sending QR tickets or passes, verify the crop/image contains the full QR code and the intended attendee name before sending; the QR screenshot is effectively the redeemable ticket.\n\n## Fetching Conversation Context\n\nBefore drafting or proposing any reply, always fetch recent chat history for context:\n\n```bash\npython3 /opt/data/scripts/wa_recent_messages.py \"6591234567@s.whatsapp.net\" 30\n```\n\nUse `limit=30` by default. For active group chats or ambiguous situations, fetch up to 50 messages. Read enough context to understand who is being discussed, current tone, previous commitments, and whether Vincent has already replied. The script returns newest messages first (descending order).\n\n## Draft Approvals β€” `jimmy-wa-drafts` sidecar\n\nWhen you need Vincent to approve a WhatsApp reply, POST the draft to the sidecar. It owns the entire approval flow: Telegram message with 4 inline buttons, Telegram WebApp for Edit/Context, sending to WhatsApp via Evolution API on Approve.\n\n### Posting a draft\n\nAlways include an `extras` object with the original WA context. The sidecar passes `extras` through verbatim if Vincent later clicks \"Context\" β€” without it, your redraft will be flying blind.\n\n```bash\ncat > /tmp/wa_draft.json <<'JSON'\n{\n \"contact\": \"120363351787895353@g.us\",\n \"contact_name\": \"Cycling group\",\n \"summary\": \"Karen sent a blessing over the bike ride; she's not joining\",\n \"draft_text\": \"Thanks Karen, will miss you both β€” praying for a good time too πŸ™\",\n \"extras\": {\n \"group_jid\": \"120363351787895353@g.us\",\n \"sender_name\": \"Karen Tan\",\n \"sender_jid\": \"6597630553@s.whatsapp.net\",\n \"category\": \"prayer/blessing\",\n \"original_message\": \"Happy cycling, to those who can make it… Dn Jeff and I had made plans for this PM so we won't be able to join y'all. Praying the Lord's favor, protection and good-weather blessings be upon y'all and your route!\",\n \"extracted\": {\n \"what\": \"Cycling from East Coast Park to Changi Village, then dinner\",\n \"when\": \"Today, 4:00 PM\",\n \"where\": \"East Coast Park to Changi Village\",\n \"who\": \"Group members who voted yes for cycling/dinner or dinner\"\n },\n \"confidence\": \"high\",\n \"why_appropriate\": \"Multiple group messages are blessings/wishes; Vincent has not replied; tone matches grace-based norm.\",\n \"wa_message_id\": \"ABC123XYZ\"\n }\n}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_draft.json\n```\n\nResponse: `{\"draft_id\":\"abc123def\",\"telegram_message_id\":42}`\n\nWhat to put in `extras` (use whatever makes sense for the message type):\n\n- **Always**: `original_message` (verbatim WA text), `sender_name`, `category` (e.g. `prayer`, `birthday`, `event`, `dm-reply`, `enquiry`)\n- **Group messages**: `group_jid`, `sender_jid`\n- **DMs**: `sender_jid`\n- **Strongly recommended (drives the rich Telegram render)**: `extracted` (dict of who/what/when/where), `confidence` (`high`/`medium`/`low`), `why_appropriate` (one-line reasoning shown to Vincent so he can decide)\n- **Optional**: `wa_message_id`, `tone_hint` (e.g. `warm-grace-based`), `sender_phone` (auto-derived from `sender_jid` if omitted)\n\nVincent receives a Telegram message in the `@jimmy_WA_bot` chat with 4 inline buttons. The sidecar handles every interaction; **you do not track replies, callbacks, or message IDs**. Just POST and move on.\n\n| Button | Behavior |\n|--------|----------|\n| **Approve** | Sidecar sends draft to WhatsApp |\n| **Edit** | Opens a Telegram WebApp with draft pre-filled; Vincent edits + sends; sidecar pushes to WhatsApp |\n| **Context** | Opens a Telegram WebApp with empty textarea + the original WA message visible; Vincent provides context. (When `HERMES_REDRAFT_WEBHOOK` is wired in the sidecar's env, this will POST back to Jimmy for a redraft. Until then, the context is just stored on the draft.) |\n| **Pass** | Sidecar deletes the draft |\n\n### JSON escaping when `draft_text` contains quotes\n\nSingle quotes (`It's`, `God's`) break shell interpolation. Write the JSON to a temp file first:\n\n```bash\ncat > /tmp/wa_draft.json <<'JSON'\n{\"contact\":\"6591234567@s.whatsapp.net\",\"contact_name\":\"John\",\"summary\":\"…\",\"draft_text\":\"It's nice to hear from you!\"}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_draft.json\n```\n\n### Networking\n\n`https://jimmy-draft.huper.asia` is the public HTTPS endpoint of the sidecar (TLS-terminated by Traefik on ubuntu-server). HTTPS is required because Hermes blocks plain-HTTP commands via tirith. The internal Docker DNS hostname http://jimmy-wa-v64mqe:8080 also works for non-shell calls but the agent is forbidden from using it.\n\n### Sidecar status & logs\n\n- Source: (auto-deploys on push)\n- Public URL: (`/health` for ping)\n- Container: `jimmy-wa-v64mqe.1.` on ubuntu-server\n- Bot: `@jimmy_WA_bot`\n\n## Handling a Redraft Trigger (`wa-redraft` webhook)\n\nWhen Vincent clicks the **Context** button on a draft, the sidecar:\n1. Edits the original Telegram message to \"πŸ”„ Adding context for X… agent is redrafting.\" (removes the old buttons so Vincent can't approve the now-stale text)\n2. POSTs to your `wa-redraft` webhook (`http://localhost:8644/webhooks/wa-redraft`) with the original draft fields plus Vincent's `context` instructions and the `extras` you stored when you first created the draft.\n\nWhen you receive a `wa-redraft` activation, the prompt template will lay out the fields. Your job:\n\n1. Read Vincent's `context` carefully β€” that's the steering signal. The `previous_draft` is just history, not necessarily a starting point.\n2. Re-fetch recent WA history if the original context isn't enough (`python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`). Use `extras.group_jid` for groups or `contact` for DMs.\n3. Write a NEW draft that addresses Vincent's instructions while staying faithful to the original WA context (in `extras.original_message`, `extras.extracted`).\n4. POST the new draft back to the sidecar **with the same `draft_id`** so the existing draft is updated, not duplicated:\n\n```bash\ncat > /tmp/wa_redraft.json <\",\n \"contact\": \"\",\n \"contact_name\": \"\",\n \"summary\": \"\",\n \"draft_text\": \"\",\n \"extras\": \n}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_redraft.json\n```\n\nThe sidecar will send a NEW Telegram message with 4 fresh buttons. Vincent can Approve, Edit, Context-again (loop), or Pass.\n\nDo **not** send to WhatsApp directly during a redraft β€” the approval flow is the sidecar's job.\n\n## Incoming Message Processing\n\n### Delivery Pipeline (current)\n1. Evolution API POSTs each `MESSAGES_UPSERT` webhook directly to Hermes at `http://localhost:8644/webhooks/whatsapp-incoming` (proxied via `jimmy.huper.asia/webhooks/whatsapp-incoming`)\n2. Hermes webhook route (`whatsapp-incoming`, dynamic subscription) validates and spawns an agent\n3. Agent loads `whatsapp-evolution` skill, follows the rules below\n4. Webhook subscription uses `deliver: log` because webhooks do not support `deliver: local`; the agent explicitly calls `send_message(target=\"telegram\", ...)` only for important/actionable/VIP/event/birthday/prayer/error items, OR posts a draft to the sidecar for approval items.\n\n**No more OpenClaw, no Python webhook server, no port 5001, no keepalive cron.** If messages aren't being processed, check: nginx proxy to port 8644, `hermes webhook list` for the subscription, Evolution API webhook URL setting.\n\n## Contact Rules & Escalation\n\nConfig: `memory/whatsapp-vip-contacts.json`\n\n### Priority Levels\n\n| Priority | Action | Example |\n|----------|--------|---------|\n| **VIP (call-always)** | **CALL Vincent immediately via Twilio without asking.** Do NOT ask for approval first. During quiet hours (1am-10am SGT) suppress call and use Telegram notification only, unless life-and-death emergency. | Dn Jeff (+6597630553) |\n| **Family** | Always notify Vincent on Telegram | ps112 group, family members |\n| **Auto-reply** | Reply based on context file, notify Vincent | Repair shops |\n| **Normal** | Analyze intent, reply if appropriate, notify if important | Everyone else |\n| **Broadcast/Spam** | Ignore silently | Forwarded job posts, chain messages |\n\n### Quiet Hours\n- 1am-10am SGT (17:00-02:00 UTC): avoid phone calls, including for Dn Jeff / VIP call-always contacts\n- **Exception:** life-and-death emergencies always call regardless of quiet hours\n- During quiet hours, use Telegram notification only and mention that a call was suppressed due to quiet hours\n\n## Making a Call via Twilio\n\nCredentials: `/opt/data/secrets/.twilio-credentials-new` (`TWILIO_ACCOUNT_SID`, `TWILIO_AUTH_TOKEN`, `TWILIO_PHONE_NUMBER`). From: `+16614809287`, Vincent: `+6590054438`.\n\nPreferred script: `/opt/data/scripts/call_vincent.py`, now using the Twilio Python SDK baked into the container.\n\n```bash\npython3 /opt/data/scripts/call_vincent.py \"Hello Vincent, this is Jimmy. Urgent summary here.\"\n```\n\n## Action Rules\n\n### Default: DO NOT auto-reply directly\nCollect intent and stay local unless something needs Vincent.\n\n**When you need Vincent to approve a reply, POST a draft to the sidecar (see \"Draft Approvals\" above).** The sidecar handles the Telegram notification with buttons, the WebApp for editing, and sending to WhatsApp on approve.\n\nDo NOT send WhatsApp replies directly unless Vincent explicitly approves them or the category is pre-approved.\n\n### Tone (when proposing or sending replies)\n- Casual, direct, no sign-offs (no \"Thanks & Regards\", no \"- Vincent\")\n- Ask questions instead of giving instructions\n- Keep it natural β€” like texting a friend\n- Christian and grace-based: warm, humble, encouraging, non-judgmental, peace-making\n- Never use em dashes in drafted replies or approval messages. Use commas, periods, colons, or parentheses instead.\n- For prayer/birthday contexts, use gentle faith-filled language; avoid sounding preachy, performative, or overly formal\n\n### What to Do With Each Message\n\n**DMs (personal chats):**\n- Analyze intent\n- Notify Vincent on Telegram (via `send_message(target=\"telegram\", ...)`) only if important/actionable/time-sensitive, requires a decision, comes from a VIP contact\n- If you have a useful suggested reply, POST it as a draft to the sidecar instead of a Telegram notification β€” Vincent gets the buttons inline\n- Casual/social/non-actionable DMs β†’ local log only, no Telegram notification, no draft\n- Do NOT reply unless Vincent says to\n- VIP contacts β†’ **call Vincent immediately without asking**. Do NOT post a draft. Just call. During quiet hours (1am-10am SGT), suppress the call and send Telegram notification only, unless life-and-death emergency.\n\n**Group chats β€” events/meetups:**\n- If someone posts about an event, gathering, meetup, dinner, etc:\n 1. Extract: what, when, where, who\n 2. Notify Vincent on Telegram with details (no draft needed unless you also want to suggest an RSVP reply)\n 3. Ask: \"Want me to add this to Google Calendar and set a reminder?\"\n\n**Group chats β€” prayer requests / praying for others:**\n- If someone asks for prayer, shares a burden, illness, grief, struggle, thanksgiving, or asks the group to pray:\n 1. Fetch recent group context first: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n 2. Identify who/what to pray for and any sensitive details\n 3. Draft a short, grace-based Christian reply/prayer\n 4. POST it to the sidecar with `summary` describing the request and `draft_text` being the prayer\n 5. Do NOT post to WhatsApp unless Vincent approves\n\n**Group chats β€” birthday greetings:**\n- If you see birthday wishes for someone in a group:\n 1. Fetch recent group context first: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n 2. Note who the birthday person is and relationship/context if inferable\n 3. Draft a warm, grace-based Christian birthday greeting\n 4. POST it to the sidecar\n 5. Do NOT post to WhatsApp unless Vincent approves\n\n**Group chats β€” general:**\n- Casual banter, memes, forwards, broadcast spam β†’ ignore silently\n- Religious content shares β†’ ignore unless specifically addressed to Vincent\n\n### When to Notify Vincent (Telegram) vs Post a Draft\n\n**Use `send_message(target=\"telegram\", ...)`** for plain notifications: VIP DMs, events/meetups summaries, errors in the pipeline, group context that doesn't need a reply.\n\n**Use the sidecar (POST `/drafts`)** whenever you have a suggested reply for Vincent to approve: prayer responses, birthday greetings, DM replies, group RSVPs.\n\nDo **not** notify or draft for casual/social/non-actionable DMs, read receipts/status updates, religious shares not addressed to Vincent, memes, forwarded/broadcast/spam, or general group chatter.\n\n## Intent Categories\n\nWhen analyzing incoming messages, classify as:\n\n1. **Urgent** β€” time-sensitive, emergencies β†’ call Vincent outside quiet hours; during quiet hours use Telegram only unless life-and-death\n2. **Important** β€” decisions, money, appointments, anything consequential β†’ call Vincent outside quiet hours; during quiet hours notify on Telegram only unless life-threatening\n3. **Request** β€” asking Vincent to do something β†’ **notify Vincent** (Telegram) or post a draft if you have a safe reply\n4. **Enquiry** β€” asking for info, pricing, availability β†’ auto-reply if context available, otherwise post a draft\n5. **Confirmation** β€” confirming plans, bookings β†’ acknowledge and notify Vincent\n6. **Casual/Social** β€” greetings, memes, forwards β†’ usually ignore or react\n7. **Broadcast/Spam** β€” forwarded job posts, chain messages β†’ ignore\n\n## Finding Group JIDs\n\nThe `group/fetchGroups` endpoint returns 404 on some Evolution API instances (including ours). To find a group JID when you don't have it:\n\n1. **Use `chat/findMessages` with an empty `where` filter** to pull recent messages across all chats, then extract unique `remoteJid` values containing `@g.us`.\n2. **Match by content**: Fetch messages from each group JID using `where.key.remoteJid` and match sender names or message text to identify the target group.\n3. **Known group JIDs** (store in `whatsapp-vip-contacts.json` as you discover them):\n - ps112 family group: `6591887488-1464061170@g.us`\n - Overcomers group: `6591444911-1614821287@g.us`\n - η₯žηš„εŽηΎŽ group: `6596756578-1405873840@g.us` (content-sharing, casual/broadcast, local log only)\n - Overcomers/TCians group: `120363230236878544@g.us` (church fellowship, PJ sends weekly attendance reports, routine attendance = local log only)\n\nExample search:\n```bash\ncurl -s -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"remoteJid\": \"6591444911-1614821287@g.us\"}}, \"limit\": 10, \"order\": \"desc\"}'\n```\n\n## Processing Image/Media Messages\n\nWhen an incoming WhatsApp message is an image (imageMessage), video, or other media with no text or only a caption:\n\n1. **Fetch conversation context** as usual with `wa_recent_messages.py`\n2. **Download the media** using the Evolution API:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"message\": {\"key\": {\"id\": \"MESSAGE_ID\", \"fromMe\": false, \"remoteJid\": \"GROUP_OR_CHAT_JID\"}}, \"convertToMp4\": false}' \\\n -o /tmp/wa_media_response.json\n ```\n Then decode: `python3 -c \"import json,base64; d=json.load(open('/tmp/wa_media_response.json')); open('/tmp/wa_media.jpg','wb').write(base64.b64decode(d['base64']))\"`\n3. **Analyze the image** using `vision_analyze` tool with `/tmp/wa_media.jpg` path and a context-aware question\n4. **Fallback if vision is unavailable**: Use `tesseract /tmp/wa_media.jpg stdout` for OCR, combined with pixel analysis (PIL/numpy) and conversation context to infer what the image contains\n5. **Album messages** (`albumMessage`): These bundle multiple images. Each image arrives as a separate `imageMessage` with a `messageAssociation.parentMessageKey` linking them to the album. Download each image separately. The `expectedImageCount` field tells how many images to expect\n6. **messageAssociation**: When images have `parentMessageKey` with `fromMe: true`, they are replying to a message Vincent sent. Use this for context\n\n### Audio / Voice Note Messages\nWhen an incoming WhatsApp message is `audioMessage` or a voice note (`ptt: true`), treat it like other media, but transcribe before intent classification when possible:\n\n1. **Fetch conversation context first**, especially in groups:\n ```bash\n python3 /opt/data/scripts/wa_recent_messages.py \"\" 50\n ```\n2. **Download the audio** with Evolution API, saving the response to a file:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n cat > /tmp/wa_audio_req.json <<'JSON'\n {\"message\":{\"key\":{\"id\":\"MESSAGE_ID\",\"fromMe\":false,\"remoteJid\":\"GROUP_OR_CHAT_JID\"}},\"convertToMp4\":false}\n JSON\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_audio_req.json -o /tmp/wa_audio_response.json\n python3 - <<'PY'\n import json, base64\n d=json.load(open('/tmp/wa_audio_response.json'))\n b=d.get('base64') or d.get('data',{}).get('base64')\n if ',' in b[:100]: b=b.split(',',1)[1]\n open('/tmp/wa_audio.ogg','wb').write(base64.b64decode(b))\n PY\n ```\n3. **Convert OGG/Opus to WAV**:\n ```bash\n ffmpeg -y -i /tmp/wa_audio.ogg -ar 16000 -ac 1 /tmp/wa_audio.wav\n ```\n4. **Transcribe fallback when Whisper is unavailable**: the container may not have `whisper`, `faster-whisper`, `torch`, or `speech_recognition` installed globally. A reliable lightweight fallback is a temp venv with SpeechRecognition and Google web speech:\n ```bash\n python3 -m venv /tmp/srvenv\n /tmp/srvenv/bin/pip install --quiet SpeechRecognition\n /tmp/srvenv/bin/python - <<'PY'\n import speech_recognition as sr\n r=sr.Recognizer()\n with sr.AudioFile('/tmp/wa_audio.wav') as source:\n audio=r.record(source)\n print(r.recognize_google(audio, language='en-SG'))\n PY\n ```\n5. **Check adjacent voice notes from the same sender**. WhatsApp voice notes are often sent in quick sequential chunks. Download and transcribe nearby `audioMessage` records from the same sender before deciding whether to notify Vincent.\n6. Use the transcript plus recent context to classify the message. For group meetup planning, extract what/when/where/who and notify Vincent if actionable, including transcript confidence.\n\n### Video Messages\n\nWhen an incoming WhatsApp message is `videoMessage` (with or without a caption):\n\n1. **Fetch conversation context** first, as with any media message: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n2. **Download the video** using Evolution API (same pattern as images/audio β€” save response to file, then decode):\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n cat > /tmp/wa_video_req.json <<'JSON'\n {\"message\":{\"key\":{\"id\":\"MESSAGE_ID\",\"fromMe\":false,\"remoteJid\":\"CONTACT_JID\"}},\"convertToMp4\":false}\n JSON\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_video_req.json -o /tmp/wa_video_response.json\n python3 - <<'PY'\n import json, base64\n d = json.load(open('/tmp/wa_video_response.json'))\n b = d.get('base64') or d.get('data', {}).get('base64', '')\n if ',' in b[:100]: b = b.split(',', 1)[1]\n open('/tmp/wa_video.mp4', 'wb').write(base64.b64decode(b))\n print(f\"Video saved: {len(base64.b64decode(b))} bytes\")\n PY\n ```\n3. **Extract representative frames** with ffmpeg (do NOT try to analyze video directly β€” vision tools work on images):\n ```bash\n # Frame at start\n ffmpeg -y -ss 0 -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame0.jpg\n # Frame at ~1/3 of the way (good for short videos)\n ffmpeg -y -ss 3 -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame3s.jpg\n # If video is longer (>15s), also extract at midpoint\n # ffprobe -v quiet -print_format json -show_format /tmp/wa_video.mp4 # get duration\n # ffmpeg -y -ss $HALF_DURATION -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame_mid.jpg\n ```\n4. **Analyze the frames** with `vision_analyze`. Ask specific questions about content: what objects/people/text are visible, what type of video this appears to be (screen recording, personal clip, forwarded meme, etc.), and any text or identifying details.\n5. **Use the caption** from `videoMessage.caption` if present (can be empty/null). Combine frame analysis + caption + conversation context to classify the message intent.\n6. **Transcription of audio in video** is generally not feasible with available tools. If the video likely contains important spoken content (e.g. voice memo style), note this limitation and flag it to Vincent.\n\n**Key points:**\n- Always use `-o` flag and file-based decode; videos can be 1-10MB and base64 responses will be even larger\n- `--max-time 25` on curl is essential for video downloads\n- Short casual clips (showing a key fob, a parking spot, quick view of surroundings) are usually casual/social β€” classify accordingly and don't over-escalate\n- If `vision_analyze` fails (400 on models that don't support images, or 429 rate limit), fall back to conversation context inference alone\n\n### Poll Messages\nWhen `pollUpdateMessage` appears in chat history (someone voted), the message contains a `pollCreationMessageKey` with the original poll's message `id`. To get the poll question and options, query:\n```bash\ncurl -s -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"id\": \"POLL_MESSAGE_ID\", \"remoteJid\": \"GROUP_JID\"}}, \"limit\": 1}'\n```\nThe response will have `pollCreationMessageV3` with `name` (the poll question) and `options` array. Always look up the poll when votes or poll references appear, since the poll text contains the event details.\n\n### `associatedChildMessage` (Motion Photos / Companion Media)\nWhatsApp motion photos and companion media arrive as `associatedChildMessage` type. These are short videos or supplementary images attached to a parent message via `messageAssociation.parentMessageKey`. The structure is:\n\n```\nmessage: {\n messageContextInfo: {\n messageAssociation: {\n associationType: 12,\n parentMessageKey: { id, fromMe, remoteJid }\n }\n },\n associatedChildMessage: {\n message: {\n videoMessage: { ... } // or imageMessage: { ... }\n }\n }\n}\n```\n\nKey points:\n- The `parentMessageKey.id` links back to the main message (the image/video they accompany)\n- When `parentMessageKey.fromMe: true`, the companion media is replying to a message Vincent sent\n- Captions may appear on the child's `videoMessage.caption` or `imageMessage.caption` inside the nested message, NOT at the top level\n- These often contain motion-photo short videos (3 seconds) or album companion images\n- Treat them as media attachments to the parent; don't process them as standalone messages requiring independent intent analysis\n\n### Sequential Image Sets (Non-Album)\nMultiple images sent rapidly by the same sender at similar timestamps may NOT be wrapped in an `albumMessage`. They appear as separate `imageMessage` entries. Always check for adjacent images from the same sender and download+OCR all of them together, as they often form a coherent multi-page set (e.g., route info, comparison tables, maps). Missing one can lose important context.\n\n### Key pitfalls\n- The Evolution API `getBase64FromMediaMessage` endpoint returns the full base64 payload (can be 100KB+ for images). Use `-o` flag to save to file, then parse with Python. Do NOT try to pipe it directly\n- `curl` with `--max-time 25` is essential; without it, large media responses can hang\n- The `albumMessage` type itself has no `fromMe` field; check each individual `imageMessage` instead\n- Image thumbnails in the webhook payload are truncated; always download the full image for analysis\n- **WhatsApp media URLs expire**: CDN download links (`mmg.whatsapp.net`) expire after roughly 24-48 hours. When `getBase64FromMediaMessage` returns a 400/404 error like \"Failed to fetch stream from https://mmg.whatsapp.net/...\", the media is no longer retrievable via the API. This means you CANNOT download adjacent/older voice notes, images, or documents for context if they're more than a day or two old. Fall back to conversation text context only. Fresh/incoming messages are almost always still downloadable.\n- **Voice note transcription reliability**: Short voice notes (<15s), especially in Singlish/Chinese-English mix common in Singapore contacts, often produce unreliable transcriptions via SpeechRecognition+Google. Always flag low-confidence transcriptions to Vincent rather than acting on potentially garbled text.\n- **Vision API failures**: `vision_analyze` can fail with 429 (rate limit) **or 400** (model doesn't support image input, e.g. glm-5.1 via ollama-cloud). Always have a fallback plan. When vision is unavailable, try tesseract OCR, then pixel analysis (PIL/numpy for brightness zones, text region detection via row-level dark-pixel density)\n- **OCR can fail on decorative/complex images**: Obituaries, memorial notices, religious flyers with ornate fonts, or dark-background images often defeat tesseract even with aggressive preprocessing (upscaling, thresholding, inverting, PSM mode variations). When OCR returns gibberish or empty, fall back to conversation context inference: use preceding/surrounding messages to determine what the image likely contains, then draft an appropriate response based on that context\n- **OCR on devotional/dark-background images**: These often defeat tesseract on full-image scans. Use a multi-strategy approach: (1) run tesseract on the full image first, (2) if text is sparse, crop into regions (top 15%, middle, bottom 15%), upscale 2-3x, enhance contrast (2-3x), and try both normal and inverted (`ImageOps.invert`) versions with `--psm 6`, (3) combine all partial OCR results. For Exodus/verse images, the title (e.g. \"WHEN MOSES' HANDS WERE\") is often in decorative font that OCR misses, while the verse body text is usually readable.\n- **Prioritize conversation context**: When processing group images, always fetch recent context FIRST (before attempting vision/OCR). If image analysis fails, the conversation history often makes the image's purpose clear enough to draft a meaningful reply\n- **DM conversation context is one-sided (fromMe only)**: The Evolution API `findMessages` endpoint often returns only Vincent's sent messages (`fromMe: true`) for DM chats, not the contact's incoming messages. This means `wa_recent_messages.py` can show only one side of the conversation. When processing a DM webhook, do NOT rely solely on fetched history for the contact's side. The webhook payload already contains the incoming message text. Use what the webhook gives you plus Vincent's side from the API. If you need the contact's recent messages, check if `fromMe: false` records exist; if none are returned, the API simply doesn't have them and you must work with what you have (webhook message + Vincent's replies for context).\n- **`findMessages` ignores `key.fromMe` filter**: Passing `\"key.fromMe\": false` in the `where` clause still returns all messages (including `fromMe: true`). To find messages from a specific sender, fetch results and filter client-side by checking `fromMe` in Python.\n- **`findMessages` caps at ~50 records per request**: Even with `limit: 200` or `limit: 500`, the API returns at most 50 records. When one party sends many consecutive messages, the other party's messages may not appear in the first 50 results. Use pagination or fetch multiple pages if you need older messages.\n- **Video frame extraction**: Use `ffmpeg -y -ss N -i video.mp4 -frames:v 1 -update 1 output.jpg` (the `-update 1` flag is needed; without it ffmpeg treats the output as an image sequence and fails). Extract frames at multiple timestamps (start, ~1/3, midpoint for longer videos) since a single frame may miss the key content.\n- **ffmpeg \"Invalid color space\" failure on WhatsApp videos**: Some WhatsApp H.264 videos have `color_space=reserved` and `color_primaries=reserved` metadata that ffmpeg cannot decode at all. Every codec/format/filter combination fails with \"Invalid color space\" / return code 234. This is NOT fixable with `-vf setparams`, `-color_trc`, `-colorspace`, `-color_primaries`, or `-vf format=` flags because the error occurs at the decoder level before any filter runs. **Do not waste time trying ffmpeg workarounds.** When this happens, fall back to: (1) conversation context inference (who sent it, what thread it's in, adjacent messages), (2) the jpegThumbnail embedded in the videoMessage (small but gives a rough idea; base64-encoded JPEG, typically 600-900 bytes, use `vision_analyze` on it if available), or (3) if all analysis fails, just classify based on surrounding text context.\n- **`wa_recent_messages.py` JSON output breaks `json.loads()`**: The script's stdout contains WhatsApp message content with control characters (emoji, CJK, deviceListMetadata binary blobs) that cause `json.loads()` to fail with `Invalid control character` or `Expecting ',' delimiter` errors. Even `strict=False` does not reliably fix it. **Reliable approach**: use `curl -o /tmp/wa_msgs.json` to save the Evolution API response directly to a file (the file write handles binary safely), then parse with `json.load(open('/tmp/wa_msgs.json'))` in Python. Example:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n curl -s --max-time 15 -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"remoteJid\": \"CONTACT_JID\"}}, \"limit\": 20}' -o /tmp/wa_msgs.json\n ```\n Then in Python: `data = json.load(open('/tmp/wa_msgs.json')); records = data['messages']['records']`\n\n## API Reference\n\nFor full API endpoints (media, contacts, location, buttons) β€” see [references/api.md](references/api.md).\n\n[Skill directory: /opt/data/skills/personal/whatsapp-evolution]\nResolve any relative paths in this skill (e.g. `scripts/foo.js`, `templates/config.yaml`) against that directory, then run them with the terminal tool using the absolute path.\n\n[This skill has supporting files:]\n- references/api.md -> /opt/data/skills/personal/whatsapp-evolution/references/api.md\n- scripts/process_queue.py -> /opt/data/skills/personal/whatsapp-evolution/scripts/process_queue.py\n- scripts/send_text.sh -> /opt/data/skills/personal/whatsapp-evolution/scripts/send_text.sh\n\nLoad any of these with skill_view(name=\"personal/whatsapp-evolution\", file_path=\"\"), or run scripts directly by absolute path (e.g. `node /opt/data/skills/personal/whatsapp-evolution/scripts/foo.js`).\n\nThe user has provided the following instruction alongside the skill invocation: Incoming WhatsApp webhook received.\n\nEvent: messages.upsert\nInstance: n8n-WA\nSender: Shalom Evonne\nSender JID: 120363418526182134@g.us\nFrom me: False\nMessage type: conversation\nMessage: {\n \"event\": \"messages.upsert\",\n \"instance\": \"n8n-WA\",\n \"data\": {\n \"key\": {\n \"remoteJid\": \"120363418526182134@g.us\",\n \"fromMe\": false,\n \"id\": \"3EB0186C2657AE3A20E157\",\n \"participant\": \"131503308673050@lid\",\n \"participantAlt\": \"6591887488@s.whatsapp.net\",\n \"addressingMode\": \"lid\"\n },\n \"pushName\": \"Shalom Evonne\",\n \"status\": \"DELIVERY_ACK\",\n \"message\": {\n \"conversation\": \"Good evening Shalom Family!\\n\\nTomorrow is Tuesday, Week Even\\n\\nNaomi Schedule:\\n8:00-8:25am - Morning Assembly\\n8:25-9:00am - Science\\n9:00-9:35am - Science\\n9:35-10:10am - Recess\\n10:10-10:45am - English Language\\n10:45-11:20am - English Language\\n11:20-11:55am - Art\\n11:55-12:30pm - Art\\n12:30-1:05pm - Design & Technology\\n1:05-1:40pm - Lunch\\n1:40-2:15pm - G2/G3 Class\\n\\nDismissal: 2:15pm\\n\\nPack your bag tonight and rest well! God bless you tomorrow!\"\n },\n \"contextInfo\": null,\n \"messageType\": \"conversation\",\n \"messageTimestamp\": 1778497327,\n \"instanceId\": \"b2bfaa30-d5f1-4b22-b979-791892619414\",\n \"source\": \"web\"\n },\n \"destination\": \"https://jimmy.huper.asia/webhooks/whatsapp-incoming\",\n \"date_time\": \"2026-05-11T08:02:08.187Z\",\n \"sender\": \"6590054438@s.whatsapp.net\",\n \"server_url\": \"https://evolution.huper.asia\",\n \"apikey\": \"5E4C532223A0-460B-A749-16C584F56E74\"\n}\n\nGoal: stay silent unless the WhatsApp message is important/actionable OR there is a safe/useful reply draft Vincent should approve. For a start, surface group prayer/praying-for-others messages and group birthday wishes with proposed replies for approval.\n\nFollow the whatsapp-evolution skill and /opt/data/agent-state/memory/whatsapp-vip-contacts.json:\n- If fromMe is true: ignore. Do NOT call send_message. Final response local log only.\n- Ignore status/update/read-receipt events or payloads with no real message content. Do NOT notify Telegram.\n- Analyze intent: Urgent/Important/Request/Enquiry/Confirmation/Prayer/Birthday/Casual/Broadcast-Spam.\n- Before drafting any reply or approval suggestion, fetch recent context: python3 /opt/data/scripts/wa_recent_messages.py \"120363418526182134@g.us\" 30. For active/ambiguous groups use 50.\n- All proposed replies must be Christian and grace-based: warm, humble, encouraging, non-judgmental, peace-making, not preachy.\n- Never use em dashes in drafted replies or approval messages. Use commas, periods, colons, or parentheses instead.\n- Group prayer/praying-for-others: if someone asks for prayer, shares illness/grief/burden/thanksgiving, or invites prayer, fetch context and send Telegram approval request with summary + suggested short prayer/reply + confidence + why appropriate.\n- Group birthday wishes: if someone is being wished happy birthday, fetch context and send Telegram approval request with birthday person + suggested warm grace-based greeting + confidence.\n- If you think you can auto-reply to any DM/group message: DO NOT send WhatsApp reply. Instead call send_message(target=\"telegram\", message=...) with sender/group, context summary, suggested reply text, confidence, and ask Vincent to approve/edit/reject.\n- VIP contacts, including Dn Jeff: notify Telegram; outside quiet hours call Vincent if actionably important. During 1am-10am SGT, Telegram only unless life-and-death emergency.\n- Group events/meetups: extract what/when/where/who and notify Telegram.\n- Casual/social, forwarded content, religious shares not requiring reply, spam, broadcasts, normal receipts/read statuses: local log only unless there is a useful reply suggestion.\n- NEVER auto-reply directly on WhatsApp unless Vincent explicitly approves or future config marks that exact category pre-approved.\n\nDelivery rule:\n- For important/actionable/VIP/event/birthday/prayer/error OR suggested auto-reply: call send_message(target=\"telegram\", message=...) yourself.\n- For non-important items with no useful reply candidate: DO NOT call send_message. Keep final response as local-only log.\n", "timestamp": 1778497354.3358994, "anchor": true}, {"id": 47191, "role": "assistant", "content": "", "timestamp": 1778497354.3511105, "tool_calls": [{"id": "call_3F0uCQgLLx8h5laxl9tnkpWf", "call_id": "call_3F0uCQgLLx8h5laxl9tnkpWf", "response_item_id": "fc_08f9826cf01c04b7016a01b7356c488191a801f543a673926b", "type": "function", "function": {"name": "read_file", "arguments": "{\"path\":\"/opt/data/agent-state/memory/whatsapp-vip-contacts.json\",\"limit\":500}"}}]}, {"id": 47193, "role": "assistant", "content": "", "timestamp": 1778497354.3553572, "tool_calls": [{"id": "call_aWjD4lSvGEdmXLdIujJdkjrP", "call_id": "call_aWjD4lSvGEdmXLdIujJdkjrP", "response_item_id": "fc_0bb669d2b998e96f016a01b73e998081918434983f37fd2f72", "type": "function", "function": {"name": "terminal", "arguments": "{\"command\":\"python3 /opt/data/scripts/wa_recent_messages.py \\\"120363418526182134@g.us\\\" 30\",\"timeout\":60}"}}]}, {"id": 47195, "role": "assistant", "content": "", "timestamp": 1778497354.455492, "tool_calls": [{"id": "call_50XD0MjRO2P49uOE7KV23Tnj", "call_id": "call_50XD0MjRO2P49uOE7KV23Tnj", "response_item_id": "fc_0aad50370b17473c016a01b7453a58819189988d177d1e1147", "type": "function", "function": {"name": "send_message", "arguments": "{\"target\":\"telegram\",\"message\":\"WhatsApp group update from Shalom Evonne in Shalom Family:\\n\\nNaomi’s school schedule for tomorrow, Tuesday Week Even.\\n\\nKey items:\\nβ€’ 8:00am Morning Assembly\\nβ€’ Science, English Language, Art, Design & Technology, G2/G3 Class\\nβ€’ Recess: 9:35am\\nβ€’ Lunch: 1:05pm\\nβ€’ Dismissal: 2:15pm\\n\\nReminder from Evonne: pack bag tonight and rest well.\\n\\nCategory: actionable family/school logistics. No reply needed.\"}"}}]}], "bookend_end": [{"id": 47197, "role": "assistant", "content": "Local log: processed incoming WhatsApp group message from Shalom Evonne. Classified as actionable family/school logistics, notified Vincent on Telegram.", "timestamp": 1778497354.4754677}], "messages_before": 0, "messages_after": 5}, {"session_id": "20260508_052645_d6fab425", "when": "May 08, 2026 at 05:26 AM", "source": "webhook", "model": "glm-5.1", "title": null, "matched_role": "user", "match_message_id": 41227, "snippet": "...draft_text\": \"Thanks >>>Karen<<<, will miss you both β€” praying for a good time too πŸ™\",\n \"extras\": {\n \"group_jid\": \"120363351787895353@g.us\",\n \"sender_name\": \">>>Karen<<< >>>Tan<<<\",\n \"sender_jid\": \">>>6597630553<<<@s.whatsapp.net\",\n \"category\": \">>>prayer<<>>blessing<<<\",\n \"original_message\": \"Happy >>>cycling<<<, to those who...", "bookend_start": [], "messages": [{"id": 41227, "role": "user", "content": "[SYSTEM: The user has invoked the \"whatsapp-evolution\" skill, indicating they want you to follow its instructions. The full skill content is loaded below.]\n\n---\nname: whatsapp-evolution\ndescription: WhatsApp messaging via Evolution API β€” send, receive, analyze, and auto-reply. Use when processing an incoming WhatsApp message delivered via the webhook, replying on Vincent's behalf, liaising with contacts, sending messages, or handling WhatsApp-related tasks. Covers intent analysis, auto-reply rules, escalation (VIP β†’ phone call), conversation context fetching, and draft approvals via the jimmy-wa-drafts sidecar.\n---\n\n# WhatsApp via Evolution API\n\n## Credentials\nStored in `/opt/data/secrets/.whatsapp-evolution-credentials`:\n- `EVOLUTION_API_URL`, `EVOLUTION_API_KEY`, `EVOLUTION_INSTANCE`\n\nLoad with: `export $(cat .whatsapp-evolution-credentials | xargs)`\n\n## Sending Messages (direct, no approval)\n\n```bash\nexport $(cat .whatsapp-evolution-credentials | xargs)\ncurl -s -X POST \"${EVOLUTION_API_URL}/message/sendText/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"number\": \"6591234567\", \"text\": \"Hello!\"}'\n```\n\n- Phone format: country code, no `+` (e.g. `6590054438`)\n- Groups: use JID `{groupid}@g.us`\n- For media/location/contacts β€” see [references/api.md](references/api.md)\n\nUse this only for pre-approved auto-replies or follow-ups Vincent has already authorized. For any reply that needs Vincent's review, use the sidecar (next section).\n\n## Fetching Conversation Context\n\nBefore drafting or proposing any reply, always fetch recent chat history for context:\n\n```bash\npython3 /opt/data/scripts/wa_recent_messages.py \"6591234567@s.whatsapp.net\" 30\n```\n\nUse `limit=30` by default. For active group chats or ambiguous situations, fetch up to 50 messages. Read enough context to understand who is being discussed, current tone, previous commitments, and whether Vincent has already replied. The script returns newest messages first (descending order).\n\n## Draft Approvals β€” `jimmy-wa-drafts` sidecar\n\nWhen you need Vincent to approve a WhatsApp reply, POST the draft to the sidecar. It owns the entire approval flow: Telegram message with 4 inline buttons, Telegram WebApp for Edit/Context, sending to WhatsApp via Evolution API on Approve.\n\n### Posting a draft\n\nAlways include an `extras` object with the original WA context. The sidecar passes `extras` through verbatim if Vincent later clicks \"Context\" β€” without it, your redraft will be flying blind.\n\n```bash\ncat > /tmp/wa_draft.json <<'JSON'\n{\n \"contact\": \"120363351787895353@g.us\",\n \"contact_name\": \"Cycling group\",\n \"summary\": \"Karen sent a blessing over the bike ride; she's not joining\",\n \"draft_text\": \"Thanks Karen, will miss you both β€” praying for a good time too πŸ™\",\n \"extras\": {\n \"group_jid\": \"120363351787895353@g.us\",\n \"sender_name\": \"Karen Tan\",\n \"sender_jid\": \"6597630553@s.whatsapp.net\",\n \"category\": \"prayer/blessing\",\n \"original_message\": \"Happy cycling, to those who can make it… Dn Jeff and I had made plans for this PM so we won't be able to join y'all. Praying the Lord's favor, protection and good-weather blessings be upon y'all and your route!\",\n \"extracted\": {\n \"what\": \"Cycling from East Coast Park to Changi Village, then dinner\",\n \"when\": \"Today, 4:00 PM\",\n \"where\": \"East Coast Park to Changi Village\",\n \"who\": \"Group members who voted yes for cycling/dinner or dinner\"\n },\n \"confidence\": \"high\",\n \"why_appropriate\": \"Multiple group messages are blessings/wishes; Vincent has not replied; tone matches grace-based norm.\",\n \"wa_message_id\": \"ABC123XYZ\"\n }\n}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_draft.json\n```\n\nResponse: `{\"draft_id\":\"abc123def\",\"telegram_message_id\":42}`\n\nWhat to put in `extras` (use whatever makes sense for the message type):\n\n- **Always**: `original_message` (verbatim WA text), `sender_name`, `category` (e.g. `prayer`, `birthday`, `event`, `dm-reply`, `enquiry`)\n- **Group messages**: `group_jid`, `sender_jid`\n- **DMs**: `sender_jid`\n- **Strongly recommended (drives the rich Telegram render)**: `extracted` (dict of who/what/when/where), `confidence` (`high`/`medium`/`low`), `why_appropriate` (one-line reasoning shown to Vincent so he can decide)\n- **Optional**: `wa_message_id`, `tone_hint` (e.g. `warm-grace-based`), `sender_phone` (auto-derived from `sender_jid` if omitted)\n\nVincent receives a Telegram message in the `@jimmy_WA_bot` chat with 4 inline buttons. The sidecar handles every interaction; **you do not track replies, callbacks, or message IDs**. Just POST and move on.\n\n| Button | Behavior |\n|--------|----------|\n| **Approve** | Sidecar sends draft to WhatsApp |\n| **Edit** | Opens a Telegram WebApp with draft pre-filled; Vincent edits + sends; sidecar pushes to WhatsApp |\n| **Context** | Opens a Telegram WebApp with empty textarea + the original WA message visible; Vincent provides context. (When `HERMES_REDRAFT_WEBHOOK` is wired in the sidecar's env, this will POST back to Jimmy for a redraft. Until then, the context is just stored on the draft.) |\n| **Pass** | Sidecar deletes the draft |\n\n### JSON escaping when `draft_text` contains quotes\n\nSingle quotes (`It's`, `God's`) break shell interpolation. Write the JSON to a temp file first:\n\n```bash\ncat > /tmp/wa_draft.json <<'JSON'\n{\"contact\":\"6591234567@s.whatsapp.net\",\"contact_name\":\"John\",\"summary\":\"…\",\"draft_text\":\"It's nice to hear from you!\"}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_draft.json\n```\n\n### Networking\n\n`https://jimmy-draft.huper.asia` is the public HTTPS endpoint of the sidecar (TLS-terminated by Traefik on ubuntu-server). HTTPS is required because Hermes blocks plain-HTTP commands via tirith. The internal Docker DNS hostname http://jimmy-wa-v64mqe:8080 also works for non-shell calls but the agent is forbidden from using it.\n\n### Sidecar status & logs\n\n- Source: (auto-deploys on push)\n- Public URL: (`/health` for ping)\n- Container: `jimmy-wa-v64mqe.1.` on ubuntu-server\n- Bot: `@jimmy_WA_bot`\n\n## Handling a Redraft Trigger (`wa-redraft` webhook)\n\nWhen Vincent clicks the **Context** button on a draft, the sidecar:\n1. Edits the original Telegram message to \"πŸ”„ Adding context for X… agent is redrafting.\" (removes the old buttons so Vincent can't approve the now-stale text)\n2. POSTs to your `wa-redraft` webhook (`http://localhost:8644/webhooks/wa-redraft`) with the original draft fields plus Vincent's `context` instructions and the `extras` you stored when you first created the draft.\n\nWhen you receive a `wa-redraft` activation, the prompt template will lay out the fields. Your job:\n\n1. Read Vincent's `context` carefully β€” that's the steering signal. The `previous_draft` is just history, not necessarily a starting point.\n2. Re-fetch recent WA history if the original context isn't enough (`python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`). Use `extras.group_jid` for groups or `contact` for DMs.\n3. Write a NEW draft that addresses Vincent's instructions while staying faithful to the original WA context (in `extras.original_message`, `extras.extracted`).\n4. POST the new draft back to the sidecar **with the same `draft_id`** so the existing draft is updated, not duplicated:\n\n```bash\ncat > /tmp/wa_redraft.json <\",\n \"contact\": \"\",\n \"contact_name\": \"\",\n \"summary\": \"\",\n \"draft_text\": \"\",\n \"extras\": \n}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_redraft.json\n```\n\nThe sidecar will send a NEW Telegram message with 4 fresh buttons. Vincent can Approve, Edit, Context-again (loop), or Pass.\n\nDo **not** send to WhatsApp directly during a redraft β€” the approval flow is the sidecar's job.\n\n## Incoming Message Processing\n\n### Delivery Pipeline (current)\n1. Evolution API POSTs each `MESSAGES_UPSERT` webhook directly to Hermes at `http://localhost:8644/webhooks/whatsapp-incoming` (proxied via `jimmy.huper.asia/webhooks/whatsapp-incoming`)\n2. Hermes webhook route (`whatsapp-incoming`, dynamic subscription) validates and spawns an agent\n3. Agent loads `whatsapp-evolution` skill, follows the rules below\n4. Webhook subscription uses `deliver: log` because webhooks do not support `deliver: local`; the agent explicitly calls `send_message(target=\"telegram\", ...)` only for important/actionable/VIP/event/birthday/prayer/error items, OR posts a draft to the sidecar for approval items.\n\n**No more OpenClaw, no Python webhook server, no port 5001, no keepalive cron.** If messages aren't being processed, check: nginx proxy to port 8644, `hermes webhook list` for the subscription, Evolution API webhook URL setting.\n\n## Contact Rules & Escalation\n\nConfig: `memory/whatsapp-vip-contacts.json`\n\n### Priority Levels\n\n| Priority | Action | Example |\n|----------|--------|---------|\n| **VIP (call-always)** | **CALL Vincent immediately via Twilio without asking.** Do NOT ask for approval first. During quiet hours (1am-10am SGT) suppress call and use Telegram notification only, unless life-and-death emergency. | Dn Jeff (+6597630553) |\n| **Family** | Always notify Vincent on Telegram | ps112 group, family members |\n| **Auto-reply** | Reply based on context file, notify Vincent | Repair shops |\n| **Normal** | Analyze intent, reply if appropriate, notify if important | Everyone else |\n| **Broadcast/Spam** | Ignore silently | Forwarded job posts, chain messages |\n\n### Quiet Hours\n- 1am-10am SGT (17:00-02:00 UTC): avoid phone calls, including for Dn Jeff / VIP call-always contacts\n- **Exception:** life-and-death emergencies always call regardless of quiet hours\n- During quiet hours, use Telegram notification only and mention that a call was suppressed due to quiet hours\n\n## Making a Call via Twilio\n\nCredentials: `/opt/data/secrets/.twilio-credentials-new` (`TWILIO_ACCOUNT_SID`, `TWILIO_AUTH_TOKEN`, `TWILIO_PHONE_NUMBER`). From: `+16614809287`, Vincent: `+6590054438`.\n\nPreferred script: `/opt/data/scripts/call_vincent.py`, now using the Twilio Python SDK baked into the container.\n\n```bash\npython3 /opt/data/scripts/call_vincent.py \"Hello Vincent, this is Jimmy. Urgent summary here.\"\n```\n\n## Action Rules\n\n### Default: DO NOT auto-reply directly\nCollect intent and stay local unless something needs Vincent.\n\n**When you need Vincent to approve a reply, POST a draft to the sidecar (see \"Draft Approvals\" above).** The sidecar handles the Telegram notification with buttons, the WebApp for editing, and sending to WhatsApp on approve.\n\nDo NOT send WhatsApp replies directly unless Vincent explicitly approves them or the category is pre-approved.\n\n### Tone (when proposing or sending replies)\n- Casual, direct, no sign-offs (no \"Thanks & Regards\", no \"- Vincent\")\n- Ask questions instead of giving instructions\n- Keep it natural β€” like texting a friend\n- Christian and grace-based: warm, humble, encouraging, non-judgmental, peace-making\n- Never use em dashes in drafted replies or approval messages. Use commas, periods, colons, or parentheses instead.\n- For prayer/birthday contexts, use gentle faith-filled language; avoid sounding preachy, performative, or overly formal\n\n### What to Do With Each Message\n\n**DMs (personal chats):**\n- Analyze intent\n- Notify Vincent on Telegram (via `send_message(target=\"telegram\", ...)`) only if important/actionable/time-sensitive, requires a decision, comes from a VIP contact\n- If you have a useful suggested reply, POST it as a draft to the sidecar instead of a Telegram notification β€” Vincent gets the buttons inline\n- Casual/social/non-actionable DMs β†’ local log only, no Telegram notification, no draft\n- Do NOT reply unless Vincent says to\n- VIP contacts β†’ **call Vincent immediately without asking**. Do NOT post a draft. Just call. During quiet hours (1am-10am SGT), suppress the call and send Telegram notification only, unless life-and-death emergency.\n\n**Group chats β€” events/meetups:**\n- If someone posts about an event, gathering, meetup, dinner, etc:\n 1. Extract: what, when, where, who\n 2. Notify Vincent on Telegram with details (no draft needed unless you also want to suggest an RSVP reply)\n 3. Ask: \"Want me to add this to Google Calendar and set a reminder?\"\n\n**Group chats β€” prayer requests / praying for others:**\n- If someone asks for prayer, shares a burden, illness, grief, struggle, thanksgiving, or asks the group to pray:\n 1. Fetch recent group context first: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n 2. Identify who/what to pray for and any sensitive details\n 3. Draft a short, grace-based Christian reply/prayer\n 4. POST it to the sidecar with `summary` describing the request and `draft_text` being the prayer\n 5. Do NOT post to WhatsApp unless Vincent approves\n\n**Group chats β€” birthday greetings:**\n- If you see birthday wishes for someone in a group:\n 1. Fetch recent group context first: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n 2. Note who the birthday person is and relationship/context if inferable\n 3. Draft a warm, grace-based Christian birthday greeting\n 4. POST it to the sidecar\n 5. Do NOT post to WhatsApp unless Vincent approves\n\n**Group chats β€” general:**\n- Casual banter, memes, forwards, broadcast spam β†’ ignore silently\n- Religious content shares β†’ ignore unless specifically addressed to Vincent\n\n### When to Notify Vincent (Telegram) vs Post a Draft\n\n**Use `send_message(target=\"telegram\", ...)`** for plain notifications: VIP DMs, events/meetups summaries, errors in the pipeline, group context that doesn't need a reply.\n\n**Use the sidecar (POST `/drafts`)** whenever you have a suggested reply for Vincent to approve: prayer responses, birthday greetings, DM replies, group RSVPs.\n\nDo **not** notify or draft for casual/social/non-actionable DMs, read receipts/status updates, religious shares not addressed to Vincent, memes, forwarded/broadcast/spam, or general group chatter.\n\n## Intent Categories\n\nWhen analyzing incoming messages, classify as:\n\n1. **Urgent** β€” time-sensitive, emergencies β†’ call Vincent outside quiet hours; during quiet hours use Telegram only unless life-and-death\n2. **Important** β€” decisions, money, appointments, anything consequential β†’ call Vincent outside quiet hours; during quiet hours notify on Telegram only unless life-threatening\n3. **Request** β€” asking Vincent to do something β†’ **notify Vincent** (Telegram) or post a draft if you have a safe reply\n4. **Enquiry** β€” asking for info, pricing, availability β†’ auto-reply if context available, otherwise post a draft\n5. **Confirmation** β€” confirming plans, bookings β†’ acknowledge and notify Vincent\n6. **Casual/Social** β€” greetings, memes, forwards β†’ usually ignore or react\n7. **Broadcast/Spam** β€” forwarded job posts, chain messages β†’ ignore\n\n## Finding Group JIDs\n\nThe `group/fetchGroups` endpoint returns 404 on some Evolution API instances (including ours). To find a group JID when you don't have it:\n\n1. **Use `chat/findMessages` with an empty `where` filter** to pull recent messages across all chats, then extract unique `remoteJid` values containing `@g.us`.\n2. **Match by content**: Fetch messages from each group JID using `where.key.remoteJid` and match sender names or message text to identify the target group.\n3. **Known group JIDs** (store in `whatsapp-vip-contacts.json` as you discover them):\n - ps112 family group: `6591887488-1464061170@g.us`\n - Overcomers group: `6591444911-1614821287@g.us`\n - η₯žηš„εŽηΎŽ group: `6596756578-1405873840@g.us` (content-sharing, casual/broadcast, local log only)\n - Overcomers/TCians group: `120363230236878544@g.us` (church fellowship, PJ sends weekly attendance reports, routine attendance = local log only)\n\nExample search:\n```bash\ncurl -s -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"remoteJid\": \"6591444911-1614821287@g.us\"}}, \"limit\": 10, \"order\": \"desc\"}'\n```\n\n## Processing Image/Media Messages\n\nWhen an incoming WhatsApp message is an image (imageMessage), video, or other media with no text or only a caption:\n\n1. **Fetch conversation context** as usual with `wa_recent_messages.py`\n2. **Download the media** using the Evolution API:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"message\": {\"key\": {\"id\": \"MESSAGE_ID\", \"fromMe\": false, \"remoteJid\": \"GROUP_OR_CHAT_JID\"}}, \"convertToMp4\": false}' \\\n -o /tmp/wa_media_response.json\n ```\n Then decode: `python3 -c \"import json,base64; d=json.load(open('/tmp/wa_media_response.json')); open('/tmp/wa_media.jpg','wb').write(base64.b64decode(d['base64']))\"`\n3. **Analyze the image** using `vision_analyze` tool with `/tmp/wa_media.jpg` path and a context-aware question\n4. **Fallback if vision is unavailable**: Use `tesseract /tmp/wa_media.jpg stdout` for OCR, combined with pixel analysis (PIL/numpy) and conversation context to infer what the image contains\n5. **Album messages** (`albumMessage`): These bundle multiple images. Each image arrives as a separate `imageMessage` with a `messageAssociation.parentMessageKey` linking them to the album. Download each image separately. The `expectedImageCount` field tells how many images to expect\n6. **messageAssociation**: When images have `parentMessageKey` with `fromMe: true`, they are replying to a message Vincent sent. Use this for context\n\n### Audio / Voice Note Messages\nWhen an incoming WhatsApp message is `audioMessage` or a voice note (`ptt: true`), treat it like other media, but transcribe before intent classification when possible:\n\n1. **Fetch conversation context first**, especially in groups:\n ```bash\n python3 /opt/data/scripts/wa_recent_messages.py \"\" 50\n ```\n2. **Download the audio** with Evolution API, saving the response to a file:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n cat > /tmp/wa_audio_req.json <<'JSON'\n {\"message\":{\"key\":{\"id\":\"MESSAGE_ID\",\"fromMe\":false,\"remoteJid\":\"GROUP_OR_CHAT_JID\"}},\"convertToMp4\":false}\n JSON\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_audio_req.json -o /tmp/wa_audio_response.json\n python3 - <<'PY'\n import json, base64\n d=json.load(open('/tmp/wa_audio_response.json'))\n b=d.get('base64') or d.get('data',{}).get('base64')\n if ',' in b[:100]: b=b.split(',',1)[1]\n open('/tmp/wa_audio.ogg','wb').write(base64.b64decode(b))\n PY\n ```\n3. **Convert OGG/Opus to WAV**:\n ```bash\n ffmpeg -y -i /tmp/wa_audio.ogg -ar 16000 -ac 1 /tmp/wa_audio.wav\n ```\n4. **Transcribe fallback when Whisper is unavailable**: the container may not have `whisper`, `faster-whisper`, `torch`, or `speech_recognition` installed globally. A reliable lightweight fallback is a temp venv with SpeechRecognition and Google web speech:\n ```bash\n python3 -m venv /tmp/srvenv\n /tmp/srvenv/bin/pip install --quiet SpeechRecognition\n /tmp/srvenv/bin/python - <<'PY'\n import speech_recognition as sr\n r=sr.Recognizer()\n with sr.AudioFile('/tmp/wa_audio.wav') as source:\n audio=r.record(source)\n print(r.recognize_google(audio, language='en-SG'))\n PY\n ```\n5. **Check adjacent voice notes from the same sender**. WhatsApp voice notes are often sent in quick sequential chunks. Download and transcribe nearby `audioMessage` records from the same sender before deciding whether to notify Vincent.\n6. Use the transcript plus recent context to classify the message. For group meetup planning, extract what/when/where/who and notify Vincent if actionable, including transcript confidence.\n\n### Video Messages\n\nWhen an incoming WhatsApp message is `videoMessage` (with or without a caption):\n\n1. **Fetch conversation context** first, as with any media message: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n2. **Download the video** using Evolution API (same pattern as images/audio β€” save response to file, then decode):\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n cat > /tmp/wa_video_req.json <<'JSON'\n {\"message\":{\"key\":{\"id\":\"MESSAGE_ID\",\"fromMe\":false,\"remoteJid\":\"CONTACT_JID\"}},\"convertToMp4\":false}\n JSON\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_video_req.json -o /tmp/wa_video_response.json\n python3 - <<'PY'\n import json, base64\n d = json.load(open('/tmp/wa_video_response.json'))\n b = d.get('base64') or d.get('data', {}).get('base64', '')\n if ',' in b[:100]: b = b.split(',', 1)[1]\n open('/tmp/wa_video.mp4', 'wb').write(base64.b64decode(b))\n print(f\"Video saved: {len(base64.b64decode(b))} bytes\")\n PY\n ```\n3. **Extract representative frames** with ffmpeg (do NOT try to analyze video directly β€” vision tools work on images):\n ```bash\n # Frame at start\n ffmpeg -y -ss 0 -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame0.jpg\n # Frame at ~1/3 of the way (good for short videos)\n ffmpeg -y -ss 3 -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame3s.jpg\n # If video is longer (>15s), also extract at midpoint\n # ffprobe -v quiet -print_format json -show_format /tmp/wa_video.mp4 # get duration\n # ffmpeg -y -ss $HALF_DURATION -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame_mid.jpg\n ```\n4. **Analyze the frames** with `vision_analyze`. Ask specific questions about content: what objects/people/text are visible, what type of video this appears to be (screen recording, personal clip, forwarded meme, etc.), and any text or identifying details.\n5. **Use the caption** from `videoMessage.caption` if present (can be empty/null). Combine frame analysis + caption + conversation context to classify the message intent.\n6. **Transcription of audio in video** is generally not feasible with available tools. If the video likely contains important spoken content (e.g. voice memo style), note this limitation and flag it to Vincent.\n\n**Key points:**\n- Always use `-o` flag and file-based decode; videos can be 1-10MB and base64 responses will be even larger\n- `--max-time 25` on curl is essential for video downloads\n- Short casual clips (showing a key fob, a parking spot, quick view of surroundings) are usually casual/social β€” classify accordingly and don't over-escalate\n- If `vision_analyze` fails (400 on models that don't support images, or 429 rate limit), fall back to conversation context inference alone\n\n### Poll Messages\nWhen `pollUpdateMessage` appears in chat history (someone voted), the message contains a `pollCreationMessageKey` with the original poll's message `id`. To get the poll question and options, query:\n```bash\ncurl -s -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"id\": \"POLL_MESSAGE_ID\", \"remoteJid\": \"GROUP_JID\"}}, \"limit\": 1}'\n```\nThe response will have `pollCreationMessageV3` with `name` (the poll question) and `options` array. Always look up the poll when votes or poll references appear, since the poll text contains the event details.\n\n### `associatedChildMessage` (Motion Photos / Companion Media)\nWhatsApp motion photos and companion media arrive as `associatedChildMessage` type. These are short videos or supplementary images attached to a parent message via `messageAssociation.parentMessageKey`. The structure is:\n\n```\nmessage: {\n messageContextInfo: {\n messageAssociation: {\n associationType: 12,\n parentMessageKey: { id, fromMe, remoteJid }\n }\n },\n associatedChildMessage: {\n message: {\n videoMessage: { ... } // or imageMessage: { ... }\n }\n }\n}\n```\n\nKey points:\n- The `parentMessageKey.id` links back to the main message (the image/video they accompany)\n- When `parentMessageKey.fromMe: true`, the companion media is replying to a message Vincent sent\n- Captions may appear on the child's `videoMessage.caption` or `imageMessage.caption` inside the nested message, NOT at the top level\n- These often contain motion-photo short videos (3 seconds) or album companion images\n- Treat them as media attachments to the parent; don't process them as standalone messages requiring independent intent analysis\n\n### Sequential Image Sets (Non-Album)\nMultiple images sent rapidly by the same sender at similar timestamps may NOT be wrapped in an `albumMessage`. They appear as separate `imageMessage` entries. Always check for adjacent images from the same sender and download+OCR all of them together, as they often form a coherent multi-page set (e.g., route info, comparison tables, maps). Missing one can lose important context.\n\n### Key pitfalls\n- The Evolution API `getBase64FromMediaMessage` endpoint returns the full base64 payload (can be 100KB+ for images). Use `-o` flag to save to file, then parse with Python. Do NOT try to pipe it directly\n- `curl` with `--max-time 25` is essential; without it, large media responses can hang\n- The `albumMessage` type itself has no `fromMe` field; check each individual `imageMessage` instead\n- Image thumbnails in the webhook payload are truncated; always download the full image for analysis\n- **WhatsApp media URLs expire**: CDN download links (`mmg.whatsapp.net`) expire after roughly 24-48 hours. When `getBase64FromMediaMessage` returns a 400/404 error like \"Failed to fetch stream from https://mmg.whatsapp.net/...\", the media is no longer retrievable via the API. This means you CANNOT download adjacent/older voice notes, images, or documents for context if they're more than a day or two old. Fall back to conversation text context only. Fresh/incoming messages are almost always still downloadable.\n- **Voice note transcription reliability**: Short voice notes (<15s), especially in Singlish/Chinese-English mix common in Singapore contacts, often produce unreliable transcriptions via SpeechRecognition+Google. Always flag low-confidence transcriptions to Vincent rather than acting on potentially garbled text.\n- **Vision API failures**: `vision_analyze` can fail with 429 (rate limit) **or 400** (model doesn't support image input, e.g. glm-5.1 via ollama-cloud). Always have a fallback plan. When vision is unavailable, try tesseract OCR, then pixel analysis (PIL/numpy for brightness zones, text region detection via row-level dark-pixel density)\n- **OCR can fail on decorative/complex images**: Obituaries, memorial notices, religious flyers with ornate fonts, or dark-background images often defeat tesseract even with aggressive preprocessing (upscaling, thresholding, inverting, PSM mode variations). When OCR returns gibberish or empty, fall back to conversation context inference: use preceding/surrounding messages to determine what the image likely contains, then draft an appropriate response based on that context\n- **OCR on devotional/dark-background images**: These often defeat tesseract on full-image scans. Use a multi-strategy approach: (1) run tesseract on the full image first, (2) if text is sparse, crop into regions (top 15%, middle, bottom 15%), upscale 2-3x, enhance contrast (2-3x), and try both normal and inverted (`ImageOps.invert`) versions with `--psm 6`, (3) combine all partial OCR results. For Exodus/verse images, the title (e.g. \"WHEN MOSES' HANDS WERE\") is often in decorative font that OCR misses, while the verse body text is usually readable.\n- **Prioritize conversation context**: When processing group images, always fetch recent context FIRST (before attempting vision/OCR). If image analysis fails, the conversation history often makes the image's purpose clear enough to draft a meaningful reply\n- **DM conversation context is one-sided (fromMe only)**: The Evolution API `findMessages` endpoint often returns only Vincent's sent messages (`fromMe: true`) for DM chats, not the contact's incoming messages. This means `wa_recent_messages.py` can show only one side of the conversation. When processing a DM webhook, do NOT rely solely on fetched history for the contact's side. The webhook payload already contains the incoming message text. Use what the webhook gives you plus Vincent's side from the API. If you need the contact's recent messages, check if `fromMe: false` records exist; if none are returned, the API simply doesn't have them and you must work with what you have (webhook message + Vincent's replies for context).\n- **`findMessages` ignores `key.fromMe` filter**: Passing `\"key.fromMe\": false` in the `where` clause still returns all messages (including `fromMe: true`). To find messages from a specific sender, fetch results and filter client-side by checking `fromMe` in Python.\n- **`findMessages` caps at ~50 records per request**: Even with `limit: 200` or `limit: 500`, the API returns at most 50 records. When one party sends many consecutive messages, the other party's messages may not appear in the first 50 results. Use pagination or fetch multiple pages if you need older messages.\n- **Video frame extraction**: Use `ffmpeg -y -ss N -i video.mp4 -frames:v 1 -update 1 output.jpg` (the `-update 1` flag is needed; without it ffmpeg treats the output as an image sequence and fails). Extract frames at multiple timestamps (start, ~1/3, midpoint for longer videos) since a single frame may miss the key content.\n- **ffmpeg \"Invalid color space\" failure on WhatsApp videos**: Some WhatsApp H.264 videos have `color_space=reserved` and `color_primaries=reserved` metadata that ffmpeg cannot decode at all. Every codec/format/filter combination fails with \"Invalid color space\" / return code 234. This is NOT fixable with `-vf setparams`, `-color_trc`, `-colorspace`, `-color_primaries`, or `-vf format=` flags because the error occurs at the decoder level before any filter runs. **Do not waste time trying ffmpeg workarounds.** When this happens, fall back to: (1) conversation context inference (who sent it, what thread it's in, adjacent messages), (2) the jpegThumbnail embedded in the videoMessage (small but gives a rough idea; base64-encoded JPEG, typically 600-900 bytes, use `vision_analyze` on it if available), or (3) if all analysis fails, just classify based on surrounding text context.\n- **`wa_recent_messages.py` JSON output breaks `json.loads()`**: The script's stdout contains WhatsApp message content with control characters (emoji, CJK, deviceListMetadata binary blobs) that cause `json.loads()` to fail with `Invalid control character` or `Expecting ',' delimiter` errors. Even `strict=False` does not reliably fix it. **Reliable approach**: use `curl -o /tmp/wa_msgs.json` to save the Evolution API response directly to a file (the file write handles binary safely), then parse with `json.load(open('/tmp/wa_msgs.json'))` in Python. Example:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n curl -s --max-time 15 -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"remoteJid\": \"CONTACT_JID\"}}, \"limit\": 20}' -o /tmp/wa_msgs.json\n ```\n Then in Python: `data = json.load(open('/tmp/wa_msgs.json')); records = data['messages']['records']`\n\n## API Reference\n\nFor full API endpoints (media, contacts, location, buttons) β€” see [references/api.md](references/api.md).\n\n[Skill directory: /opt/data/skills/personal/whatsapp-evolution]\nResolve any relative paths in this skill (e.g. `scripts/foo.js`, `templates/config.yaml`) against that directory, then run them with the terminal tool using the absolute path.\n\n[This skill has supporting files:]\n- references/api.md -> /opt/data/skills/personal/whatsapp-evolution/references/api.md\n- scripts/process_queue.py -> /opt/data/skills/personal/whatsapp-evolution/scripts/process_queue.py\n- scripts/send_text.sh -> /opt/data/skills/personal/whatsapp-evolution/scripts/send_text.sh\n\nLoad any of these with skill_view(name=\"personal/whatsapp-evolution\", file_path=\"\"), or run scripts directly by absolute path (e.g. `node /opt/data/skills/personal/whatsapp-evolution/scripts/foo.js`).\n\nThe user has provided the following instruction alongside the skill invocation: Incoming WhatsApp webhook received.\n\nEvent: messages.upsert\nInstance: n8n-WA\nSender: Jeffry Lim\nSender JID: 120363351787895353@g.us\nFrom me: False\nMessage type: conversation\nMessage: {\n \"event\": \"messages.upsert\",\n \"instance\": \"n8n-WA\",\n \"data\": {\n \"key\": {\n \"remoteJid\": \"120363351787895353@g.us\",\n \"fromMe\": false,\n \"id\": \"3ACEEB2D5C1BF9BBA5C3\",\n \"participant\": \"107889763065951@lid\",\n \"participantAlt\": \"6597630553@s.whatsapp.net\",\n \"addressingMode\": \"lid\"\n },\n \"pushName\": \"Jeffry Lim\",\n \"status\": \"DELIVERY_ACK\",\n \"message\": {\n \"senderKeyDistributionMessage\": {\n \"groupId\": \"120363351787895353@g.us\",\n \"axolotlSenderKeyDistributionMessage\": {\n \"0\": 51,\n \"1\": 8,\n \"2\": 170,\n \"3\": 133,\n \"4\": 246,\n \"5\": 194,\n \"6\": 1,\n \"7\": 16,\n \"8\": 0,\n \"9\": 26,\n \"10\": 32,\n \"11\": 62,\n \"12\": 159,\n \"13\": 98,\n \"14\": 71,\n \"15\": 62,\n \"16\": 11,\n \"17\": 236,\n \"18\": 130,\n \"19\": 189,\n \"20\": 94,\n \"21\": 81,\n \"22\": 233,\n \"23\": 209,\n \"24\": 109,\n \"25\": 160,\n \"26\": 36,\n \"27\": 201,\n \"28\": 45,\n \"29\": 49,\n \"30\": 144,\n \"31\": 245,\n \"32\": 98,\n \"33\": 15,\n \"34\": 24,\n \"35\": 160,\n \"36\": 220,\n \"37\": 119,\n \"38\": 185,\n \"39\": 59,\n \"40\": 19,\n \"41\": 185,\n \"42\": 119,\n \"43\": 34,\n \"44\": 33,\n \"45\": 5,\n \"46\": 160,\n \"47\": 44,\n \"48\": 38,\n \"49\": 51,\n \"50\": 15,\n \"51\": 228,\n \"52\": 250,\n \"53\": 157,\n \"54\": 246,\n \"55\": 246,\n \"56\": 108,\n \"57\": 90,\n \"58\": 127,\n \"59\": 144,\n \"60\": 90,\n \"61\": 160,\n \"62\": 98,\n \"63\": 180,\n \"64\": 68,\n \"65\": 237,\n \"66\": 106,\n \"67\": 164,\n \"68\": 92,\n \"69\": 107,\n \"70\": 78,\n \"71\": 82,\n \"72\": 29,\n \"73\": 155,\n \"74\": 139,\n \"75\": 223,\n \"76\": 168,\n \"77\": 14\n }\n },\n \"messageContextInfo\": {\n \"threadId\": [],\n \"messageSecret\": {\n \"0\": 177,\n \"1\": 76,\n \"2\": 93,\n \"3\": 219,\n \"4\": 100,\n \"5\": 175,\n \"6\": 143,\n \"7\": 105,\n \"8\": 233,\n \"9\": 211,\n \"10\": 73,\n \"11\": 236,\n \"12\": 47,\n \"13\": 211,\n \"14\": 233,\n \"15\": 171,\n \"16\": 142,\n \"17\": 192,\n \"18\": 110,\n \"19\": 155,\n \"20\": 72,\n \"21\": 166,\n \"22\": 214,\n \"23\": 26,\n \"24\": 52,\n \"25\": 253,\n \"26\": 111,\n \"27\": 158,\n \"28\": 222,\n \"29\": 114,\n \"30\": 206,\n \"31\": 102\n }\n },\n \"conversation\": \"Hi all our next G3 LG will be on Saturday 23 May 2026. Come and receive the much more.\\n\\n\\ud83d\\uddd3\\ufe0f Date: Saturday 23 May 2026\\n\\u23f0 Time: 2.30 pm\\n\\ud83d\\udccd Venue: our home 18 Bedok Walk S(469119)\"\n },\n \"messageType\": \"conversation\",\n \"messageTimestamp\": 1778218004,\n \"instanceId\": \"b2bfaa30-d5f1-4b22-b979-791892619414\",\n \"source\": \"ios\"\n },\n \"destination\": \"https://jimmy.huper.asia/webhooks/whatsapp-incoming\",\n \"date_time\": \"2026-05-08T02:26:44.901Z\",\n \"sender\": \"6590054438@s.whatsapp.net\",\n \"server_url\": \"https://evolution.huper.asia\",\n \"apikey\": \"5E4C532223A0-460B-A749-16C584F56E74\"\n}\n\nGoal: stay silent unless the WhatsApp message is important/actionable OR there is a safe/useful reply draft Vincent should approve. For a start, surface group prayer/praying-for-others messages and group birthday wishes with proposed replies for approval.\n\nFollow the whatsapp-evolution skill and /opt/data/agent-state/memory/whatsapp-vip-contacts.json:\n- If fromMe is true: ignore. Do NOT call send_message. Final response local log only.\n- Ignore status/update/read-receipt events or payloads with no real message content. Do NOT notify Telegram.\n- Analyze intent: Urgent/Important/Request/Enquiry/Confirmation/Prayer/Birthday/Casual/Broadcast-Spam.\n- Before drafting any reply or approval suggestion, fetch recent context: python3 /opt/data/scripts/wa_recent_messages.py \"120363351787895353@g.us\" 30. For active/ambiguous groups use 50.\n- All proposed replies must be Christian and grace-based: warm, humble, encouraging, non-judgmental, peace-making, not preachy.\n- Never use em dashes in drafted replies or approval messages. Use commas, periods, colons, or parentheses instead.\n- Group prayer/praying-for-others: if someone asks for prayer, shares illness/grief/burden/thanksgiving, or invites prayer, fetch context and send Telegram approval request with summary + suggested short prayer/reply + confidence + why appropriate.\n- Group birthday wishes: if someone is being wished happy birthday, fetch context and send Telegram approval request with birthday person + suggested warm grace-based greeting + confidence.\n- If you think you can auto-reply to any DM/group message: DO NOT send WhatsApp reply. Instead call send_message(target=\"telegram\", message=...) with sender/group, context summary, suggested reply text, confidence, and ask Vincent to approve/edit/reject.\n- VIP contacts, including Dn Jeff: notify Telegram; outside quiet hours call Vincent if actionably important. During 1am-10am SGT, Telegram only unless life-and-death emergency.\n- Group events/meetups: extract what/when/where/who and notify Telegram.\n- Casual/social, forwarded content, religious shares not requiring reply, spam, broadcasts, normal receipts/read statuses: local log only unless there is a useful reply suggestion.\n- NEVER auto-reply directly on WhatsApp unless Vincent explicitly approves or future config marks that exact category pre-approved.\n\nDelivery rule:\n- For important/actionable/VIP/event/birthday/prayer/error OR suggested auto-reply: call send_message(target=\"telegram\", message=...) yourself.\n- For non-important items with no useful reply candidate: DO NOT call send_message. Keep final response as local-only log.\n", "timestamp": 1778218226.7954414, "anchor": true}, {"id": 41228, "role": "assistant", "content": "", "timestamp": 1778218226.7989028, "tool_calls": [{"id": "call_l6xuwnqp", "call_id": "call_l6xuwnqp", "response_item_id": "fc_l6xuwnqp", "type": "function", "function": {"name": "execute_code", "arguments": "{\"code\":\"import subprocess, json\\n\\n# Fetch recent group context\\nresult = subprocess.run(\\n ['python3', '/opt/data/scripts/wa_recent_messages.py', '120363351787895353@g.us', '30'],\\n capture_output=True, text=True, timeout=30\\n)\\nprint(\\\"STDOUT:\\\", result.stdout[:3000])\\nprint(\\\"STDERR:\\\", result.stderr[:500] if result.stderr else \\\"none\\\")\"}"}}, {"id": "call_kq0jq7tm", "call_id": "call_kq0jq7tm", "response_item_id": "fc_kq0jq7tm", "type": "function", "function": {"name": "read_file", "arguments": "{\"path\":\"/opt/data/agent-state/memory/whatsapp-vip-contacts.json\"}"}}]}, {"id": 41231, "role": "assistant", "content": "", "timestamp": 1778218226.8012145, "tool_calls": [{"id": "call_0atiindt", "call_id": "call_0atiindt", "response_item_id": "fc_0atiindt", "type": "function", "function": {"name": "execute_code", "arguments": "{\"code\":\"from datetime import datetime, timezone, timedelta\\n\\n# Current time check\\nnow_utc = datetime.now(timezone.utc)\\nnow_sgt = now_utc + timedelta(hours=8)\\nprint(f\\\"Current SGT time: {now_sgt.strftime('%Y-%m-%d %H:%M:%S')}\\\")\\n\\n# Quiet hours: 1am-10am SGT\\nsgt_hour = now_sgt.hour\\nprint(f\\\"SGT hour: {sgt_hour}\\\")\\nprint(f\\\"In quiet hours: {1 \\u003c= sgt_hour \\u003c 10}\\\")\"}"}}]}], "bookend_end": [{"id": 41251, "role": "assistant", "content": "Processed the incoming WhatsApp message. Here's the summary:\n\n**Message:** Dn Jeffry Lim (VIP, +6597630553) announced the next **G3 Life Group** meeting in group chat `120363351787895353@g.us`:\n\n- **What:** G3 LG gathering (\"Come and receive the much more\")\n- **When:** Saturday 23 May 2026, 2:30 PM\n- **Where:** Dn Jeff's home, 18 Bedok Walk S(469119)\n\n**Actions taken:**\n1. βœ… Notified Vincent on Telegram with the event details\n2. βœ… Added G3 LG group to `whatsapp-vip-contacts.json` as a known group\n3. No call needed β€” this is a standard group event announcement, not an urgent DM from Dn Jeff\n4. No draft reply posted β€” this is an announcement, not a conversation requiring a response", "timestamp": 1778218226.8122857}, {"id": 41252, "role": "user", "content": "[System: Your previous response was truncated by the output length limit. Continue exactly where you left off. Do not restart or repeat prior text. Finish the answer directly.]", "timestamp": 1778218226.8145766}, {"id": 41253, "role": "assistant", "content": "The task is fully complete. No further action needed.", "timestamp": 1778218226.8151088}], "messages_before": 0, "messages_after": 5}, {"session_id": "20260507_110155_9df3257c", "when": "May 07, 2026 at 11:01 AM", "source": "webhook", "model": "gpt-5.5", "title": null, "matched_role": "user", "match_message_id": 39378, "snippet": "...draft_text\": \"Thanks >>>Karen<<<, will miss you both β€” praying for a good time too πŸ™\",\n \"extras\": {\n \"group_jid\": \"120363351787895353@g.us\",\n \"sender_name\": \">>>Karen<<< >>>Tan<<<\",\n \"sender_jid\": \">>>6597630553<<<@s.whatsapp.net\",\n \"category\": \">>>prayer<<>>blessing<<<\",\n \"original_message\": \"Happy >>>cycling<<<, to those who...", "bookend_start": [], "messages": [{"id": 39378, "role": "user", "content": "[SYSTEM: The user has invoked the \"whatsapp-evolution\" skill, indicating they want you to follow its instructions. The full skill content is loaded below.]\n\n---\nname: whatsapp-evolution\ndescription: WhatsApp messaging via Evolution API β€” send, receive, analyze, and auto-reply. Use when processing an incoming WhatsApp message delivered via the webhook, replying on Vincent's behalf, liaising with contacts, sending messages, or handling WhatsApp-related tasks. Covers intent analysis, auto-reply rules, escalation (VIP β†’ phone call), conversation context fetching, and draft approvals via the jimmy-wa-drafts sidecar.\n---\n\n# WhatsApp via Evolution API\n\n## Credentials\nStored in `/opt/data/secrets/.whatsapp-evolution-credentials`:\n- `EVOLUTION_API_URL`, `EVOLUTION_API_KEY`, `EVOLUTION_INSTANCE`\n\nLoad with: `export $(cat .whatsapp-evolution-credentials | xargs)`\n\n## Sending Messages (direct, no approval)\n\n```bash\nexport $(cat .whatsapp-evolution-credentials | xargs)\ncurl -s -X POST \"${EVOLUTION_API_URL}/message/sendText/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"number\": \"6591234567\", \"text\": \"Hello!\"}'\n```\n\n- Phone format: country code, no `+` (e.g. `6590054438`)\n- Groups: use JID `{groupid}@g.us`\n- For media/location/contacts β€” see [references/api.md](references/api.md)\n\nUse this only for pre-approved auto-replies or follow-ups Vincent has already authorized. For any reply that needs Vincent's review, use the sidecar (next section).\n\n## Fetching Conversation Context\n\nBefore drafting or proposing any reply, always fetch recent chat history for context:\n\n```bash\npython3 /opt/data/scripts/wa_recent_messages.py \"6591234567@s.whatsapp.net\" 30\n```\n\nUse `limit=30` by default. For active group chats or ambiguous situations, fetch up to 50 messages. Read enough context to understand who is being discussed, current tone, previous commitments, and whether Vincent has already replied. The script returns newest messages first (descending order).\n\n## Draft Approvals β€” `jimmy-wa-drafts` sidecar\n\nWhen you need Vincent to approve a WhatsApp reply, POST the draft to the sidecar. It owns the entire approval flow: Telegram message with 4 inline buttons, Telegram WebApp for Edit/Context, sending to WhatsApp via Evolution API on Approve.\n\n### Posting a draft\n\nAlways include an `extras` object with the original WA context. The sidecar passes `extras` through verbatim if Vincent later clicks \"Context\" β€” without it, your redraft will be flying blind.\n\n```bash\ncat > /tmp/wa_draft.json <<'JSON'\n{\n \"contact\": \"120363351787895353@g.us\",\n \"contact_name\": \"Cycling group\",\n \"summary\": \"Karen sent a blessing over the bike ride; she's not joining\",\n \"draft_text\": \"Thanks Karen, will miss you both β€” praying for a good time too πŸ™\",\n \"extras\": {\n \"group_jid\": \"120363351787895353@g.us\",\n \"sender_name\": \"Karen Tan\",\n \"sender_jid\": \"6597630553@s.whatsapp.net\",\n \"category\": \"prayer/blessing\",\n \"original_message\": \"Happy cycling, to those who can make it… Dn Jeff and I had made plans for this PM so we won't be able to join y'all. Praying the Lord's favor, protection and good-weather blessings be upon y'all and your route!\",\n \"extracted\": {\n \"what\": \"Cycling from East Coast Park to Changi Village, then dinner\",\n \"when\": \"Today, 4:00 PM\",\n \"where\": \"East Coast Park to Changi Village\",\n \"who\": \"Group members who voted yes for cycling/dinner or dinner\"\n },\n \"confidence\": \"high\",\n \"why_appropriate\": \"Multiple group messages are blessings/wishes; Vincent has not replied; tone matches grace-based norm.\",\n \"wa_message_id\": \"ABC123XYZ\"\n }\n}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_draft.json\n```\n\nResponse: `{\"draft_id\":\"abc123def\",\"telegram_message_id\":42}`\n\nWhat to put in `extras` (use whatever makes sense for the message type):\n\n- **Always**: `original_message` (verbatim WA text), `sender_name`, `category` (e.g. `prayer`, `birthday`, `event`, `dm-reply`, `enquiry`)\n- **Group messages**: `group_jid`, `sender_jid`\n- **DMs**: `sender_jid`\n- **Strongly recommended (drives the rich Telegram render)**: `extracted` (dict of who/what/when/where), `confidence` (`high`/`medium`/`low`), `why_appropriate` (one-line reasoning shown to Vincent so he can decide)\n- **Optional**: `wa_message_id`, `tone_hint` (e.g. `warm-grace-based`), `sender_phone` (auto-derived from `sender_jid` if omitted)\n\nVincent receives a Telegram message in the `@jimmy_WA_bot` chat with 4 inline buttons. The sidecar handles every interaction; **you do not track replies, callbacks, or message IDs**. Just POST and move on.\n\n| Button | Behavior |\n|--------|----------|\n| **Approve** | Sidecar sends draft to WhatsApp |\n| **Edit** | Opens a Telegram WebApp with draft pre-filled; Vincent edits + sends; sidecar pushes to WhatsApp |\n| **Context** | Opens a Telegram WebApp with empty textarea + the original WA message visible; Vincent provides context. (When `HERMES_REDRAFT_WEBHOOK` is wired in the sidecar's env, this will POST back to Jimmy for a redraft. Until then, the context is just stored on the draft.) |\n| **Pass** | Sidecar deletes the draft |\n\n### JSON escaping when `draft_text` contains quotes\n\nSingle quotes (`It's`, `God's`) break shell interpolation. Write the JSON to a temp file first:\n\n```bash\ncat > /tmp/wa_draft.json <<'JSON'\n{\"contact\":\"6591234567@s.whatsapp.net\",\"contact_name\":\"John\",\"summary\":\"…\",\"draft_text\":\"It's nice to hear from you!\"}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_draft.json\n```\n\n### Networking\n\n`https://jimmy-draft.huper.asia` is the public HTTPS endpoint of the sidecar (TLS-terminated by Traefik on ubuntu-server). HTTPS is required because Hermes blocks plain-HTTP commands via tirith. The internal Docker DNS hostname http://jimmy-wa-v64mqe:8080 also works for non-shell calls but the agent is forbidden from using it.\n\n### Sidecar status & logs\n\n- Source: (auto-deploys on push)\n- Public URL: (`/health` for ping)\n- Container: `jimmy-wa-v64mqe.1.` on ubuntu-server\n- Bot: `@jimmy_WA_bot`\n\n## Handling a Redraft Trigger (`wa-redraft` webhook)\n\nWhen Vincent clicks the **Context** button on a draft, the sidecar:\n1. Edits the original Telegram message to \"πŸ”„ Adding context for X… agent is redrafting.\" (removes the old buttons so Vincent can't approve the now-stale text)\n2. POSTs to your `wa-redraft` webhook (`http://localhost:8644/webhooks/wa-redraft`) with the original draft fields plus Vincent's `context` instructions and the `extras` you stored when you first created the draft.\n\nWhen you receive a `wa-redraft` activation, the prompt template will lay out the fields. Your job:\n\n1. Read Vincent's `context` carefully β€” that's the steering signal. The `previous_draft` is just history, not necessarily a starting point.\n2. Re-fetch recent WA history if the original context isn't enough (`python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`). Use `extras.group_jid` for groups or `contact` for DMs.\n3. Write a NEW draft that addresses Vincent's instructions while staying faithful to the original WA context (in `extras.original_message`, `extras.extracted`).\n4. POST the new draft back to the sidecar **with the same `draft_id`** so the existing draft is updated, not duplicated:\n\n```bash\ncat > /tmp/wa_redraft.json <\",\n \"contact\": \"\",\n \"contact_name\": \"\",\n \"summary\": \"\",\n \"draft_text\": \"\",\n \"extras\": \n}\nJSON\ncurl -s -X POST https://jimmy-draft.huper.asia/drafts \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/wa_redraft.json\n```\n\nThe sidecar will send a NEW Telegram message with 4 fresh buttons. Vincent can Approve, Edit, Context-again (loop), or Pass.\n\nDo **not** send to WhatsApp directly during a redraft β€” the approval flow is the sidecar's job.\n\n## Incoming Message Processing\n\n### Delivery Pipeline (current)\n1. Evolution API POSTs each `MESSAGES_UPSERT` webhook directly to Hermes at `http://localhost:8644/webhooks/whatsapp-incoming` (proxied via `jimmy.huper.asia/webhooks/whatsapp-incoming`)\n2. Hermes webhook route (`whatsapp-incoming`, dynamic subscription) validates and spawns an agent\n3. Agent loads `whatsapp-evolution` skill, follows the rules below\n4. Webhook subscription uses `deliver: log` because webhooks do not support `deliver: local`; the agent explicitly calls `send_message(target=\"telegram\", ...)` only for important/actionable/VIP/event/birthday/prayer/error items, OR posts a draft to the sidecar for approval items.\n\n**No more OpenClaw, no Python webhook server, no port 5001, no keepalive cron.** If messages aren't being processed, check: nginx proxy to port 8644, `hermes webhook list` for the subscription, Evolution API webhook URL setting.\n\n## Contact Rules & Escalation\n\nConfig: `memory/whatsapp-vip-contacts.json`\n\n### Priority Levels\n\n| Priority | Action | Example |\n|----------|--------|---------|\n| **VIP (call-always)** | **CALL Vincent immediately via Twilio without asking.** Do NOT ask for approval first. During quiet hours (1am-10am SGT) suppress call and use Telegram notification only, unless life-and-death emergency. | Dn Jeff (+6597630553) |\n| **Family** | Always notify Vincent on Telegram | ps112 group, family members |\n| **Auto-reply** | Reply based on context file, notify Vincent | Repair shops |\n| **Normal** | Analyze intent, reply if appropriate, notify if important | Everyone else |\n| **Broadcast/Spam** | Ignore silently | Forwarded job posts, chain messages |\n\n### Quiet Hours\n- 1am-10am SGT (17:00-02:00 UTC): avoid phone calls, including for Dn Jeff / VIP call-always contacts\n- **Exception:** life-and-death emergencies always call regardless of quiet hours\n- During quiet hours, use Telegram notification only and mention that a call was suppressed due to quiet hours\n\n## Making a Call via Twilio\n\nCredentials: `/opt/data/secrets/.twilio-credentials-new` (`TWILIO_ACCOUNT_SID`, `TWILIO_AUTH_TOKEN`, `TWILIO_PHONE_NUMBER`). From: `+16614809287`, Vincent: `+6590054438`.\n\nPreferred script: `/opt/data/scripts/call_vincent.py`, now using the Twilio Python SDK baked into the container.\n\n```bash\npython3 /opt/data/scripts/call_vincent.py \"Hello Vincent, this is Jimmy. Urgent summary here.\"\n```\n\n## Action Rules\n\n### Default: DO NOT auto-reply directly\nCollect intent and stay local unless something needs Vincent.\n\n**When you need Vincent to approve a reply, POST a draft to the sidecar (see \"Draft Approvals\" above).** The sidecar handles the Telegram notification with buttons, the WebApp for editing, and sending to WhatsApp on approve.\n\nDo NOT send WhatsApp replies directly unless Vincent explicitly approves them or the category is pre-approved.\n\n### Tone (when proposing or sending replies)\n- Casual, direct, no sign-offs (no \"Thanks & Regards\", no \"- Vincent\")\n- Ask questions instead of giving instructions\n- Keep it natural β€” like texting a friend\n- Christian and grace-based: warm, humble, encouraging, non-judgmental, peace-making\n- Never use em dashes in drafted replies or approval messages. Use commas, periods, colons, or parentheses instead.\n- For prayer/birthday contexts, use gentle faith-filled language; avoid sounding preachy, performative, or overly formal\n\n### What to Do With Each Message\n\n**DMs (personal chats):**\n- Analyze intent\n- Notify Vincent on Telegram (via `send_message(target=\"telegram\", ...)`) only if important/actionable/time-sensitive, requires a decision, comes from a VIP contact\n- If you have a useful suggested reply, POST it as a draft to the sidecar instead of a Telegram notification β€” Vincent gets the buttons inline\n- Casual/social/non-actionable DMs β†’ local log only, no Telegram notification, no draft\n- Do NOT reply unless Vincent says to\n- VIP contacts β†’ **call Vincent immediately without asking**. Do NOT post a draft. Just call. During quiet hours (1am-10am SGT), suppress the call and send Telegram notification only, unless life-and-death emergency.\n\n**Group chats β€” events/meetups:**\n- If someone posts about an event, gathering, meetup, dinner, etc:\n 1. Extract: what, when, where, who\n 2. Notify Vincent on Telegram with details (no draft needed unless you also want to suggest an RSVP reply)\n 3. Ask: \"Want me to add this to Google Calendar and set a reminder?\"\n\n**Group chats β€” prayer requests / praying for others:**\n- If someone asks for prayer, shares a burden, illness, grief, struggle, thanksgiving, or asks the group to pray:\n 1. Fetch recent group context first: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n 2. Identify who/what to pray for and any sensitive details\n 3. Draft a short, grace-based Christian reply/prayer\n 4. POST it to the sidecar with `summary` describing the request and `draft_text` being the prayer\n 5. Do NOT post to WhatsApp unless Vincent approves\n\n**Group chats β€” birthday greetings:**\n- If you see birthday wishes for someone in a group:\n 1. Fetch recent group context first: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n 2. Note who the birthday person is and relationship/context if inferable\n 3. Draft a warm, grace-based Christian birthday greeting\n 4. POST it to the sidecar\n 5. Do NOT post to WhatsApp unless Vincent approves\n\n**Group chats β€” general:**\n- Casual banter, memes, forwards, broadcast spam β†’ ignore silently\n- Religious content shares β†’ ignore unless specifically addressed to Vincent\n\n### When to Notify Vincent (Telegram) vs Post a Draft\n\n**Use `send_message(target=\"telegram\", ...)`** for plain notifications: VIP DMs, events/meetups summaries, errors in the pipeline, group context that doesn't need a reply.\n\n**Use the sidecar (POST `/drafts`)** whenever you have a suggested reply for Vincent to approve: prayer responses, birthday greetings, DM replies, group RSVPs.\n\nDo **not** notify or draft for casual/social/non-actionable DMs, read receipts/status updates, religious shares not addressed to Vincent, memes, forwarded/broadcast/spam, or general group chatter.\n\n## Intent Categories\n\nWhen analyzing incoming messages, classify as:\n\n1. **Urgent** β€” time-sensitive, emergencies β†’ call Vincent outside quiet hours; during quiet hours use Telegram only unless life-and-death\n2. **Important** β€” decisions, money, appointments, anything consequential β†’ call Vincent outside quiet hours; during quiet hours notify on Telegram only unless life-threatening\n3. **Request** β€” asking Vincent to do something β†’ **notify Vincent** (Telegram) or post a draft if you have a safe reply\n4. **Enquiry** β€” asking for info, pricing, availability β†’ auto-reply if context available, otherwise post a draft\n5. **Confirmation** β€” confirming plans, bookings β†’ acknowledge and notify Vincent\n6. **Casual/Social** β€” greetings, memes, forwards β†’ usually ignore or react\n7. **Broadcast/Spam** β€” forwarded job posts, chain messages β†’ ignore\n\n## Finding Group JIDs\n\nThe `group/fetchGroups` endpoint returns 404 on some Evolution API instances (including ours). To find a group JID when you don't have it:\n\n1. **Use `chat/findMessages` with an empty `where` filter** to pull recent messages across all chats, then extract unique `remoteJid` values containing `@g.us`.\n2. **Match by content**: Fetch messages from each group JID using `where.key.remoteJid` and match sender names or message text to identify the target group.\n3. **Known group JIDs** (store in `whatsapp-vip-contacts.json` as you discover them):\n - ps112 family group: `6591887488-1464061170@g.us`\n - Overcomers group: `6591444911-1614821287@g.us`\n - η₯žηš„εŽηΎŽ group: `6596756578-1405873840@g.us` (content-sharing, casual/broadcast, local log only)\n - Overcomers/TCians group: `120363230236878544@g.us` (church fellowship, PJ sends weekly attendance reports, routine attendance = local log only)\n\nExample search:\n```bash\ncurl -s -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"remoteJid\": \"6591444911-1614821287@g.us\"}}, \"limit\": 10, \"order\": \"desc\"}'\n```\n\n## Processing Image/Media Messages\n\nWhen an incoming WhatsApp message is an image (imageMessage), video, or other media with no text or only a caption:\n\n1. **Fetch conversation context** as usual with `wa_recent_messages.py`\n2. **Download the media** using the Evolution API:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"message\": {\"key\": {\"id\": \"MESSAGE_ID\", \"fromMe\": false, \"remoteJid\": \"GROUP_OR_CHAT_JID\"}}, \"convertToMp4\": false}' \\\n -o /tmp/wa_media_response.json\n ```\n Then decode: `python3 -c \"import json,base64; d=json.load(open('/tmp/wa_media_response.json')); open('/tmp/wa_media.jpg','wb').write(base64.b64decode(d['base64']))\"`\n3. **Analyze the image** using `vision_analyze` tool with `/tmp/wa_media.jpg` path and a context-aware question\n4. **Fallback if vision is unavailable**: Use `tesseract /tmp/wa_media.jpg stdout` for OCR, combined with pixel analysis (PIL/numpy) and conversation context to infer what the image contains\n5. **Album messages** (`albumMessage`): These bundle multiple images. Each image arrives as a separate `imageMessage` with a `messageAssociation.parentMessageKey` linking them to the album. Download each image separately. The `expectedImageCount` field tells how many images to expect\n6. **messageAssociation**: When images have `parentMessageKey` with `fromMe: true`, they are replying to a message Vincent sent. Use this for context\n\n### Audio / Voice Note Messages\nWhen an incoming WhatsApp message is `audioMessage` or a voice note (`ptt: true`), treat it like other media, but transcribe before intent classification when possible:\n\n1. **Fetch conversation context first**, especially in groups:\n ```bash\n python3 /opt/data/scripts/wa_recent_messages.py \"\" 50\n ```\n2. **Download the audio** with Evolution API, saving the response to a file:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n cat > /tmp/wa_audio_req.json <<'JSON'\n {\"message\":{\"key\":{\"id\":\"MESSAGE_ID\",\"fromMe\":false,\"remoteJid\":\"GROUP_OR_CHAT_JID\"}},\"convertToMp4\":false}\n JSON\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_audio_req.json -o /tmp/wa_audio_response.json\n python3 - <<'PY'\n import json, base64\n d=json.load(open('/tmp/wa_audio_response.json'))\n b=d.get('base64') or d.get('data',{}).get('base64')\n if ',' in b[:100]: b=b.split(',',1)[1]\n open('/tmp/wa_audio.ogg','wb').write(base64.b64decode(b))\n PY\n ```\n3. **Convert OGG/Opus to WAV**:\n ```bash\n ffmpeg -y -i /tmp/wa_audio.ogg -ar 16000 -ac 1 /tmp/wa_audio.wav\n ```\n4. **Transcribe fallback when Whisper is unavailable**: the container may not have `whisper`, `faster-whisper`, `torch`, or `speech_recognition` installed globally. A reliable lightweight fallback is a temp venv with SpeechRecognition and Google web speech:\n ```bash\n python3 -m venv /tmp/srvenv\n /tmp/srvenv/bin/pip install --quiet SpeechRecognition\n /tmp/srvenv/bin/python - <<'PY'\n import speech_recognition as sr\n r=sr.Recognizer()\n with sr.AudioFile('/tmp/wa_audio.wav') as source:\n audio=r.record(source)\n print(r.recognize_google(audio, language='en-SG'))\n PY\n ```\n5. **Check adjacent voice notes from the same sender**. WhatsApp voice notes are often sent in quick sequential chunks. Download and transcribe nearby `audioMessage` records from the same sender before deciding whether to notify Vincent.\n6. Use the transcript plus recent context to classify the message. For group meetup planning, extract what/when/where/who and notify Vincent if actionable, including transcript confidence.\n\n### Video Messages\n\nWhen an incoming WhatsApp message is `videoMessage` (with or without a caption):\n\n1. **Fetch conversation context** first, as with any media message: `python3 /opt/data/scripts/wa_recent_messages.py \"\" 30`\n2. **Download the video** using Evolution API (same pattern as images/audio β€” save response to file, then decode):\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n cat > /tmp/wa_video_req.json <<'JSON'\n {\"message\":{\"key\":{\"id\":\"MESSAGE_ID\",\"fromMe\":false,\"remoteJid\":\"CONTACT_JID\"}},\"convertToMp4\":false}\n JSON\n curl -s --max-time 25 -X POST \"${EVOLUTION_API_URL}/chat/getBase64FromMediaMessage/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d @/tmp/wa_video_req.json -o /tmp/wa_video_response.json\n python3 - <<'PY'\n import json, base64\n d = json.load(open('/tmp/wa_video_response.json'))\n b = d.get('base64') or d.get('data', {}).get('base64', '')\n if ',' in b[:100]: b = b.split(',', 1)[1]\n open('/tmp/wa_video.mp4', 'wb').write(base64.b64decode(b))\n print(f\"Video saved: {len(base64.b64decode(b))} bytes\")\n PY\n ```\n3. **Extract representative frames** with ffmpeg (do NOT try to analyze video directly β€” vision tools work on images):\n ```bash\n # Frame at start\n ffmpeg -y -ss 0 -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame0.jpg\n # Frame at ~1/3 of the way (good for short videos)\n ffmpeg -y -ss 3 -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame3s.jpg\n # If video is longer (>15s), also extract at midpoint\n # ffprobe -v quiet -print_format json -show_format /tmp/wa_video.mp4 # get duration\n # ffmpeg -y -ss $HALF_DURATION -i /tmp/wa_video.mp4 -frames:v 1 -update 1 /tmp/wa_video_frame_mid.jpg\n ```\n4. **Analyze the frames** with `vision_analyze`. Ask specific questions about content: what objects/people/text are visible, what type of video this appears to be (screen recording, personal clip, forwarded meme, etc.), and any text or identifying details.\n5. **Use the caption** from `videoMessage.caption` if present (can be empty/null). Combine frame analysis + caption + conversation context to classify the message intent.\n6. **Transcription of audio in video** is generally not feasible with available tools. If the video likely contains important spoken content (e.g. voice memo style), note this limitation and flag it to Vincent.\n\n**Key points:**\n- Always use `-o` flag and file-based decode; videos can be 1-10MB and base64 responses will be even larger\n- `--max-time 25` on curl is essential for video downloads\n- Short casual clips (showing a key fob, a parking spot, quick view of surroundings) are usually casual/social β€” classify accordingly and don't over-escalate\n- If `vision_analyze` fails (400 on models that don't support images, or 429 rate limit), fall back to conversation context inference alone\n\n### Poll Messages\nWhen `pollUpdateMessage` appears in chat history (someone voted), the message contains a `pollCreationMessageKey` with the original poll's message `id`. To get the poll question and options, query:\n```bash\ncurl -s -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"id\": \"POLL_MESSAGE_ID\", \"remoteJid\": \"GROUP_JID\"}}, \"limit\": 1}'\n```\nThe response will have `pollCreationMessageV3` with `name` (the poll question) and `options` array. Always look up the poll when votes or poll references appear, since the poll text contains the event details.\n\n### `associatedChildMessage` (Motion Photos / Companion Media)\nWhatsApp motion photos and companion media arrive as `associatedChildMessage` type. These are short videos or supplementary images attached to a parent message via `messageAssociation.parentMessageKey`. The structure is:\n\n```\nmessage: {\n messageContextInfo: {\n messageAssociation: {\n associationType: 12,\n parentMessageKey: { id, fromMe, remoteJid }\n }\n },\n associatedChildMessage: {\n message: {\n videoMessage: { ... } // or imageMessage: { ... }\n }\n }\n}\n```\n\nKey points:\n- The `parentMessageKey.id` links back to the main message (the image/video they accompany)\n- When `parentMessageKey.fromMe: true`, the companion media is replying to a message Vincent sent\n- Captions may appear on the child's `videoMessage.caption` or `imageMessage.caption` inside the nested message, NOT at the top level\n- These often contain motion-photo short videos (3 seconds) or album companion images\n- Treat them as media attachments to the parent; don't process them as standalone messages requiring independent intent analysis\n\n### Sequential Image Sets (Non-Album)\nMultiple images sent rapidly by the same sender at similar timestamps may NOT be wrapped in an `albumMessage`. They appear as separate `imageMessage` entries. Always check for adjacent images from the same sender and download+OCR all of them together, as they often form a coherent multi-page set (e.g., route info, comparison tables, maps). Missing one can lose important context.\n\n### Key pitfalls\n- The Evolution API `getBase64FromMediaMessage` endpoint returns the full base64 payload (can be 100KB+ for images). Use `-o` flag to save to file, then parse with Python. Do NOT try to pipe it directly\n- `curl` with `--max-time 25` is essential; without it, large media responses can hang\n- The `albumMessage` type itself has no `fromMe` field; check each individual `imageMessage` instead\n- Image thumbnails in the webhook payload are truncated; always download the full image for analysis\n- **WhatsApp media URLs expire**: CDN download links (`mmg.whatsapp.net`) expire after roughly 24-48 hours. When `getBase64FromMediaMessage` returns a 400/404 error like \"Failed to fetch stream from https://mmg.whatsapp.net/...\", the media is no longer retrievable via the API. This means you CANNOT download adjacent/older voice notes, images, or documents for context if they're more than a day or two old. Fall back to conversation text context only. Fresh/incoming messages are almost always still downloadable.\n- **Voice note transcription reliability**: Short voice notes (<15s), especially in Singlish/Chinese-English mix common in Singapore contacts, often produce unreliable transcriptions via SpeechRecognition+Google. Always flag low-confidence transcriptions to Vincent rather than acting on potentially garbled text.\n- **Vision API failures**: `vision_analyze` can fail with 429 (rate limit) **or 400** (model doesn't support image input, e.g. glm-5.1 via ollama-cloud). Always have a fallback plan. When vision is unavailable, try tesseract OCR, then pixel analysis (PIL/numpy for brightness zones, text region detection via row-level dark-pixel density)\n- **OCR can fail on decorative/complex images**: Obituaries, memorial notices, religious flyers with ornate fonts, or dark-background images often defeat tesseract even with aggressive preprocessing (upscaling, thresholding, inverting, PSM mode variations). When OCR returns gibberish or empty, fall back to conversation context inference: use preceding/surrounding messages to determine what the image likely contains, then draft an appropriate response based on that context\n- **OCR on devotional/dark-background images**: These often defeat tesseract on full-image scans. Use a multi-strategy approach: (1) run tesseract on the full image first, (2) if text is sparse, crop into regions (top 15%, middle, bottom 15%), upscale 2-3x, enhance contrast (2-3x), and try both normal and inverted (`ImageOps.invert`) versions with `--psm 6`, (3) combine all partial OCR results. For Exodus/verse images, the title (e.g. \"WHEN MOSES' HANDS WERE\") is often in decorative font that OCR misses, while the verse body text is usually readable.\n- **Prioritize conversation context**: When processing group images, always fetch recent context FIRST (before attempting vision/OCR). If image analysis fails, the conversation history often makes the image's purpose clear enough to draft a meaningful reply\n- **DM conversation context is one-sided (fromMe only)**: The Evolution API `findMessages` endpoint often returns only Vincent's sent messages (`fromMe: true`) for DM chats, not the contact's incoming messages. This means `wa_recent_messages.py` can show only one side of the conversation. When processing a DM webhook, do NOT rely solely on fetched history for the contact's side. The webhook payload already contains the incoming message text. Use what the webhook gives you plus Vincent's side from the API. If you need the contact's recent messages, check if `fromMe: false` records exist; if none are returned, the API simply doesn't have them and you must work with what you have (webhook message + Vincent's replies for context).\n- **`findMessages` ignores `key.fromMe` filter**: Passing `\"key.fromMe\": false` in the `where` clause still returns all messages (including `fromMe: true`). To find messages from a specific sender, fetch results and filter client-side by checking `fromMe` in Python.\n- **`findMessages` caps at ~50 records per request**: Even with `limit: 200` or `limit: 500`, the API returns at most 50 records. When one party sends many consecutive messages, the other party's messages may not appear in the first 50 results. Use pagination or fetch multiple pages if you need older messages.\n- **Video frame extraction**: Use `ffmpeg -y -ss N -i video.mp4 -frames:v 1 -update 1 output.jpg` (the `-update 1` flag is needed; without it ffmpeg treats the output as an image sequence and fails). Extract frames at multiple timestamps (start, ~1/3, midpoint for longer videos) since a single frame may miss the key content.\n- **ffmpeg \"Invalid color space\" failure on WhatsApp videos**: Some WhatsApp H.264 videos have `color_space=reserved` and `color_primaries=reserved` metadata that ffmpeg cannot decode at all. Every codec/format/filter combination fails with \"Invalid color space\" / return code 234. This is NOT fixable with `-vf setparams`, `-color_trc`, `-colorspace`, `-color_primaries`, or `-vf format=` flags because the error occurs at the decoder level before any filter runs. **Do not waste time trying ffmpeg workarounds.** When this happens, fall back to: (1) conversation context inference (who sent it, what thread it's in, adjacent messages), (2) the jpegThumbnail embedded in the videoMessage (small but gives a rough idea; base64-encoded JPEG, typically 600-900 bytes, use `vision_analyze` on it if available), or (3) if all analysis fails, just classify based on surrounding text context.\n- **`wa_recent_messages.py` JSON output breaks `json.loads()`**: The script's stdout contains WhatsApp message content with control characters (emoji, CJK, deviceListMetadata binary blobs) that cause `json.loads()` to fail with `Invalid control character` or `Expecting ',' delimiter` errors. Even `strict=False` does not reliably fix it. **Reliable approach**: use `curl -o /tmp/wa_msgs.json` to save the Evolution API response directly to a file (the file write handles binary safely), then parse with `json.load(open('/tmp/wa_msgs.json'))` in Python. Example:\n ```bash\n export $(cat /opt/data/secrets/.whatsapp-evolution-credentials | xargs)\n curl -s --max-time 15 -X POST \"${EVOLUTION_API_URL}/chat/findMessages/${EVOLUTION_INSTANCE}\" \\\n -H \"apikey: ${EVOLUTION_API_KEY}\" -H \"Content-Type: application/json\" \\\n -d '{\"where\": {\"key\": {\"remoteJid\": \"CONTACT_JID\"}}, \"limit\": 20}' -o /tmp/wa_msgs.json\n ```\n Then in Python: `data = json.load(open('/tmp/wa_msgs.json')); records = data['messages']['records']`\n\n## API Reference\n\nFor full API endpoints (media, contacts, location, buttons) β€” see [references/api.md](references/api.md).\n\n[Skill directory: /opt/data/skills/personal/whatsapp-evolution]\nResolve any relative paths in this skill (e.g. `scripts/foo.js`, `templates/config.yaml`) against that directory, then run them with the terminal tool using the absolute path.\n\n[This skill has supporting files:]\n- references/api.md -> /opt/data/skills/personal/whatsapp-evolution/references/api.md\n- scripts/process_queue.py -> /opt/data/skills/personal/whatsapp-evolution/scripts/process_queue.py\n- scripts/send_text.sh -> /opt/data/skills/personal/whatsapp-evolution/scripts/send_text.sh\n\nLoad any of these with skill_view(name=\"personal/whatsapp-evolution\", file_path=\"\"), or run scripts directly by absolute path (e.g. `node /opt/data/skills/personal/whatsapp-evolution/scripts/foo.js`).\n\nThe user has provided the following instruction alongside the skill invocation: Incoming WhatsApp webhook received.\n\nEvent: messages.upsert\nInstance: n8n-WA\nSender: Shalom Evonne\nSender JID: 120363418526182134@g.us\nFrom me: False\nMessage type: conversation\nMessage: {\n \"event\": \"messages.upsert\",\n \"instance\": \"n8n-WA\",\n \"data\": {\n \"key\": {\n \"remoteJid\": \"120363418526182134@g.us\",\n \"fromMe\": false,\n \"id\": \"3EB03FA0A1AB1D8F20A5D6\",\n \"participant\": \"131503308673050@lid\",\n \"participantAlt\": \"6591887488@s.whatsapp.net\",\n \"addressingMode\": \"lid\"\n },\n \"pushName\": \"Shalom Evonne\",\n \"status\": \"DELIVERY_ACK\",\n \"message\": {\n \"conversation\": \"Good evening Shalom Family!\\n\\nTomorrow is Friday, Week Odd\\n\\nNaomis Schedule:\\n8:00 - 8:25 Morning Assembly\\n8:25 - 9:00 CCE\\n9:00 - 9:35 EL\\n9:35 - 10:10 Recess\\n10:10 - 10:45 Recess\\n10:45 - 11:20 Music\\n11:20 - 11:55 Needlework\\n11:55 - 12:30 Geog (G2/G3)\\n\\nDismissal: 12:30 PM\\n\\nHave a great evening and rest well! God bless you all!\\n\\nWith love,\\nShalom Evonne\"\n },\n \"contextInfo\": null,\n \"messageType\": \"conversation\",\n \"messageTimestamp\": 1778151714,\n \"instanceId\": \"b2bfaa30-d5f1-4b22-b979-791892619414\",\n \"source\": \"web\"\n },\n \"destination\": \"https://jimmy.huper.asia/webhooks/whatsapp-incoming\",\n \"date_time\": \"2026-05-07T08:01:55.063Z\",\n \"sender\": \"6590054438@s.whatsapp.net\",\n \"server_url\": \"https://evolution.huper.asia\",\n \"apikey\": \"5E4C532223A0-460B-A749-16C584F56E74\"\n}\n\nGoal: stay silent unless the WhatsApp message is important/actionable OR there is a safe/useful reply draft Vincent should approve. For a start, surface group prayer/praying-for-others messages and group birthday wishes with proposed replies for approval.\n\nFollow the whatsapp-evolution skill and /opt/data/agent-state/memory/whatsapp-vip-contacts.json:\n- If fromMe is true: ignore. Do NOT call send_message. Final response local log only.\n- Ignore status/update/read-receipt events or payloads with no real message content. Do NOT notify Telegram.\n- Analyze intent: Urgent/Important/Request/Enquiry/Confirmation/Prayer/Birthday/Casual/Broadcast-Spam.\n- Before drafting any reply or approval suggestion, fetch recent context: python3 /opt/data/scripts/wa_recent_messages.py \"120363418526182134@g.us\" 30. For active/ambiguous groups use 50.\n- All proposed replies must be Christian and grace-based: warm, humble, encouraging, non-judgmental, peace-making, not preachy.\n- Never use em dashes in drafted replies or approval messages. Use commas, periods, colons, or parentheses instead.\n- Group prayer/praying-for-others: if someone asks for prayer, shares illness/grief/burden/thanksgiving, or invites prayer, fetch context and send Telegram approval request with summary + suggested short prayer/reply + confidence + why appropriate.\n- Group birthday wishes: if someone is being wished happy birthday, fetch context and send Telegram approval request with birthday person + suggested warm grace-based greeting + confidence.\n- If you think you can auto-reply to any DM/group message: DO NOT send WhatsApp reply. Instead call send_message(target=\"telegram\", message=...) with sender/group, context summary, suggested reply text, confidence, and ask Vincent to approve/edit/reject.\n- VIP contacts, including Dn Jeff: notify Telegram; outside quiet hours call Vincent if actionably important. During 1am-10am SGT, Telegram only unless life-and-death emergency.\n- Group events/meetups: extract what/when/where/who and notify Telegram.\n- Casual/social, forwarded content, religious shares not requiring reply, spam, broadcasts, normal receipts/read statuses: local log only unless there is a useful reply suggestion.\n- NEVER auto-reply directly on WhatsApp unless Vincent explicitly approves or future config marks that exact category pre-approved.\n\nDelivery rule:\n- For important/actionable/VIP/event/birthday/prayer/error OR suggested auto-reply: call send_message(target=\"telegram\", message=...) yourself.\n- For non-important items with no useful reply candidate: DO NOT call send_message. Keep final response as local-only log.\n", "timestamp": 1778151740.5611923, "anchor": true}, {"id": 39379, "role": "assistant", "content": "", "timestamp": 1778151740.564187, "tool_calls": [{"id": "call_NuQNladvCuJD1Pl5tgdSaGjy", "call_id": "call_NuQNladvCuJD1Pl5tgdSaGjy", "response_item_id": "fc_0ffdb037b55d5b410169fc712cf7148191b230dcdc26c6688a", "type": "function", "function": {"name": "read_file", "arguments": "{\"path\":\"/opt/data/agent-state/memory/whatsapp-vip-contacts.json\",\"limit\":500}"}}, {"id": "call_Jit2CnZyV7idcbSqmmZo7ZHb", "call_id": "call_Jit2CnZyV7idcbSqmmZo7ZHb", "response_item_id": "fc_0ffdb037b55d5b410169fc712cf7248191bda0a5fe3d7fe2f9", "type": "function", "function": {"name": "terminal", "arguments": "{\"command\":\"date -u && TZ=Asia/Singapore date\",\"timeout\":10}"}}]}, {"id": 39382, "role": "assistant", "content": "", "timestamp": 1778151740.5663753, "tool_calls": [{"id": "call_CN5zEBrBuKTedWhJM0Uh23Ul", "call_id": "call_CN5zEBrBuKTedWhJM0Uh23Ul", "response_item_id": "fc_035ba1ba88a31b020169fc713675fc8191a33800e23a7ec14f", "type": "function", "function": {"name": "send_message", "arguments": "{\"target\":\"telegram\",\"message\":\"WhatsApp group update from Shalom Evonne\\n\\nCategory: Important family/school logistics\\nGroup JID: 120363418526182134@g.us\\n\\nShe shared Naomi’s schedule for tomorrow, Friday Week Odd.\\n\\nKey point:\\nβ€’ Dismissal: 12:30 PM\\n\\nSchedule:\\nβ€’ 8:00 to 8:25 Morning Assembly\\nβ€’ 8:25 to 9:00 CCE\\nβ€’ 9:00 to 9:35 EL\\nβ€’ 9:35 to 10:10 Recess\\nβ€’ 10:10 to 10:45 Recess\\nβ€’ 10:45 to 11:20 Music\\nβ€’ 11:20 to 11:55 Needlework\\nβ€’ 11:55 to 12:30 Geog (G2/G3)\\n\\nNo reply needed.\"}"}}]}], "bookend_end": [{"id": 39384, "role": "assistant", "content": "Local log: WhatsApp group update from Shalom Evonne classified as important family/school logistics. Telegram notification sent.", "timestamp": 1778151740.5673707}], "messages_before": 0, "messages_after": 5}], "count": 10, "sessions_searched": 10}