Diego, a16z
Design For Robust Intelligence Not Determinism
Test AI-powered components for semantic stability across varied inputs, not exact output matching, so you can stop crutching on example queries.
Diego says
I don't aim for strict determinism inside AI-powered software, I aim for robustness, similar intelligence every time. The real test is whether a human in that exact situation would find the response understandable, that's what a developer can program around. The highest honor of reliability is when people can program against the system without writing example queries anymore.
Try this
Test the system for understandable, humanlike responses instead of exact-match determinism.
