Pick the right era of software for the problem
Karpathy describes three kinds of software. Software 1.0 is the code you write for the computer. Software 2.0 is neural networks, in particular their weights. Software 3.0 is prompts, which are programs that program the LLM and are written in English. He calls these three completely different programming paradigms, each with slight pros and cons, and says it is a good idea to be fluent in all of them.