{"id":"uE18dLsrw_kKYMv1vVn2b","title":"Decision test: 5 real decisions for hs-decide-1-flash /v1/systemone","language":"bash","tags":[],"content":"#!/usr/bin/env bash\n# Decision test: 5 real decisions against hs-decide-1-flash via /v1/systemone\n# Same payload shape as the contract probe, but with real content instead of dummy text.\nKEY=\"sk-local\"\nBASE=\"http://192.168.50.35:8080\"\n\npost() {  # $1 = test name; JSON body comes in on stdin\n  echo \"== $1 ==\"\n  curl -s -m 60 \"$BASE/v1/systemone\" \\\n    -H \"Authorization: Bearer $KEY\" \\\n    -H \"Content-Type: application/json\" \\\n    -d @-\n  echo; echo\n}\n\npost \"1. provider choice (OpenRouter vs local 3080)\" <<'EOF'\n{\"model\":\"hs-decide-1-flash\",\"state\":{\"query\":\"Where should the Hindsight decision model run, OpenRouter or the local 3080 GPU?\",\"documents\":{\"OPENROUTER\":\"OpenRouter adds 300-800ms network latency per decision and costs per token, but needs no local VRAM.\",\"LOCAL_GPU\":\"The decision model now runs on the local 3080: decisions under 100ms, no per-token cost, but it shares VRAM with the reranker.\"}},\"questions\":{\"provider\":{\"type\":\"choice\",\"instructions\":\"Which document states the current placement and its latency tradeoff?\",\"criteria\":{\"OPENROUTER\":null,\"LOCAL_GPU\":null}},\"moved\":{\"type\":\"noul\",\"instructions\":\"Did the decision model move off OpenRouter?\"},\"OPENROUTER\":{\"type\":\"score\",\"instructions\":\"How well does document `documents.OPENROUTER` answer `query`?\",\"criteria\":[\"about something else\",\"same topic, no help\",\"helps\",\"answers\"]}}}\nEOF\n\npost \"2. recall timeout (30s vs 90s)\" <<'EOF'\n{\"model\":\"hs-decide-1-flash\",\"state\":{\"query\":\"What recall client timeout should be set while debugging the rerank stage?\",\"documents\":{\"THIRTY\":\"Keep the timeout at 30 seconds; measured recall was 26.9s so 30s is enough.\",\"NINETY\":\"Raise the timeout to 90 seconds; measured recall is 26.9s and parallel recalls run slower, so 30s is knife-edge.\"}},\"questions\":{\"timeout\":{\"type\":\"choice\",\"instructions\":\"Which document states the recommended timeout while debugging?\",\"criteria\":{\"THIRTY\":null,\"NINETY\":null}},\"changed\":{\"type\":\"noul\",\"instructions\":\"Did the recommended timeout change from 30 seconds?\"},\"NINETY\":{\"type\":\"score\",\"instructions\":\"How well does document `documents.NINETY` answer `query`?\",\"criteria\":[\"about something else\",\"same topic, no help\",\"helps\",\"answers\"]}}}\nEOF\n\npost \"3. rerank root cause (oversized pool vs network)\" <<'EOF'\n{\"model\":\"hs-decide-1-flash\",\"state\":{\"query\":\"Why was the rerank stage timing out on the Hindsight server?\",\"documents\":{\"POOL\":\"The cross-encoder reranked an oversized pool of 1,602 candidates in fp32 on CPU, taking roughly 27 seconds per request.\",\"NET\":\"A network partition between the API container and Postgres made the rerank stage time out.\"}},\"questions\":{\"cause\":{\"type\":\"choice\",\"instructions\":\"Which document gives the actual root cause of the rerank timeouts?\",\"criteria\":{\"POOL\":null,\"NET\":null}},\"network\":{\"type\":\"noul\",\"instructions\":\"Was the rerank timeout caused by a network partition?\"},\"POOL\":{\"type\":\"score\",\"instructions\":\"How well does document `documents.POOL` answer `query`?\",\"criteria\":[\"about something else\",\"same topic, no help\",\"helps\",\"answers\"]}}}\nEOF\n\npost \"4. memory strategy (fix-in-place vs Mem0)\" <<'EOF'\n{\"model\":\"hs-decide-1-flash\",\"state\":{\"query\":\"Should the memory layer be switched from Hindsight to Mem0?\",\"documents\":{\"KEEP\":\"Mem0 was the only alternative worth a ~30-minute side-by-side test, expected to lose on recall quality; fix-in-place on Hindsight is the right call.\",\"SWITCH\":\"Mem0 won the side-by-side test and should replace Hindsight as the memory layer.\"}},\"questions\":{\"strategy\":{\"type\":\"choice\",\"instructions\":\"Which document reflects the conclusion actually reached?\",\"criteria\":{\"KEEP\":null,\"SWITCH\":null}},\"mem0\":{\"type\":\"noul\",\"instructions\":\"Did Mem0 win the side-by-side test?\"},\"KEEP\":{\"type\":\"score\",\"instructions\":\"How well does document `documents.KEEP` answer `query`?\",\"criteria\":[\"about something else\",\"same topic, no help\",\"helps\",\"answers\"]}}}\nEOF\n\npost \"5. serving stack (GGUF vs transformers package)\" <<'EOF'\n{\"model\":\"hs-decide-1-flash\",\"state\":{\"query\":\"How should hs-decide-1-flash be served locally?\",\"documents\":{\"GGUF\":\"The model was converted to GGUF and served with stock llama.cpp, but the server rejected it: the systemone prompt template is compiled into the official serving stack, so a bare GGUF cannot serve decisions.\",\"SERVE\":\"Serve it with the transformers serving package: hindsight-decision-serve-transformers hindsight-ai/hs-decide-1-flash --port 8000 --api-key sk-local; the 0.8B model runs fine on CPU.\"}},\"questions\":{\"stack\":{\"type\":\"choice\",\"instructions\":\"Which document describes a serving path that actually works?\",\"criteria\":{\"GGUF\":null,\"SERVE\":null}},\"gguf_ok\":{\"type\":\"noul\",\"instructions\":\"Did stock llama.cpp serve the GGUF successfully?\"},\"SERVE\":{\"type\":\"score\",\"instructions\":\"How well does document `documents.SERVE` answer `query`?\",\"criteria\":[\"about something else\",\"same topic, no help\",\"helps\",\"answers\"]}}}\nEOF\n\necho \"== done ==\"\n","created_at":"2026-11-13T20:48:21.959Z"}