When to Code vs. When to use LLMs: Striking the Right Balance for Efficiency

I see a lot of content that is trying to solve problems with Large Language Models first and coding second (if at all).

As an AI Engineer, you need to know when to use a model (be it more traditional ML or LLM) to solve a problem and when it’s best to use code or a combination of the two.

For example, I ran across code recently where the developers had been scraping articles from Google RSS feeds but only wanted new articles about US companies.

They were taking those search results and looping over every article, sending each one to OpenAI to determine if it’s relevant or not.

There are a few ways to optimize this for efficiency and cost that they didn’t think about.

1. The articles from the RSS feed had a publication date and a domain of the source (i.e. forbes.com). You could immediately filter out any articles with a publication date older than a couple of days from the search results.
   

2. You could maintain a list of news sources that aren’t ever going to be relevant, say like a uknews.co.uk (not sure if that’s real, don’t click it). It’s obviously not relevant, so filter that out too.
   

3. You could even further filter on keywords, but that has a chance of potentially filtering out articles they might care about.
   

4. After all of those filters, you could check the remaining articles against your database and see what has and hasn’t been scraped previously.
   

Then it’s finally time to use your LLM to determine which article titles + sources are relevant using a well-tuned and well-tested system prompt with many examples using JSON output. Depending on how many articles are left and your token limits, you can include many or even all of the remaining articles into that user prompt, compared to 1 request/prompt for each and every article in the original list.

While this example was specific to scraping, this is applicable to all sorts of use cases.

Whenever you pull in a model like an LLM, I just ask that you think first, “Could I solve this problem with an LLM?” You might be surprised what’s possible with regex and ingenuity.

You might not need that LLM every time. And if you do, you can probably reduce your dependency on them and have a more efficient and cheaper overall solution.

Work smarter, not harder.