The first architecture decision is often phrased as a model choice. A better starting point is the moment the user needs help: what information is available, how quickly the response is needed, and what happens if it is wrong.
Map the information boundary
List the inputs before selecting a model: images, audio, documents, customer records and operational context. Identify which information may leave the device and which requires a controlled environment. Local inference can reduce the amount of data sent to a hosted service, but it does not automatically make the application private. Logging, backups, analytics and synchronisation still need deliberate design.
Test quality on a real task
Build a small evaluation set from representative examples you are authorised to use. Include incomplete notes, poor lighting, ambiguous language and cases where the right answer is to ask for clarification. Compare the outputs people actually need, such as a structured observation with a source reference, rather than judging how fluent the response sounds.
A local model that reliably extracts a few fields may be a better fit than a broader model for that particular step. A more demanding interpretation task may need a hosted model, a specialist model or human review. Measure the difference instead of assuming one architecture will cover every task.
Budget for the whole interaction
Measure capture, preprocessing, inference, retrieval and presentation together. On a phone, sustained use also brings battery, memory and thermal constraints. On a hosted service, the experience depends on network conditions, request limits and failure handling. Test in the setting where the work happens, including lost connectivity.
Use a hybrid design deliberately
One workable pattern is to capture and organise information locally, then offer an explicit escalation for a task that needs more capability. Make that boundary understandable to the user. Record which source material informed the output and avoid silently changing where sensitive content is processed.
What a first prototype should answer
- Can the system complete one useful task on representative inputs?
- What does it do when evidence is missing or ambiguous?
- Which data leaves the device, and why?
- Can the user review and correct the output?
- Does the end-to-end experience improve on the current process?
My work on Manex brings practical experience in language understanding and semantic retrieval to these decisions. For a client engagement, the output of feasibility work should be an evidence-backed architecture recommendation and a small, testable next step.