Nolen began their writing career in 2019, with three years dedicated to editing the Creative section at MakeUseOf. Their expertise lies at the crossroads of technology and creativity, covering areas like photography, video editing, and graphic design.
Outside of work, you'll often find Nolen diving into a good book, writing their own stories, or playing video games.
Searching with Google used to be the answer to everything. But these days, the answer sits somewhere below five sponsored results and an AI summary that's probably not accurate. Getting to the answer takes way more scrolling than the question actually deserves, and given user behavior, this usually means just settling for the easiest one, which might not be adequate.
Local LLMs have gotten good at answering questions without sending anything to big tech companies, and I also recently gave one filesystem access which ended up replacing Windows Search for the most part. But a model that fits on an 8GB GPU is usually a small one, and it only knows what it's trained on, which rules out recent and current events. So, I gave mine web access.
The problem with Google Search
Finding the answer feels like a treasure hunt now
Google search has been getting worse for years, and there's actual research behind that. A year-long study out of Germany tracked 7,392 product review searches across Google, Bing, and DuckDuckGo and found the top results were mostly pages built around affiliate links, and a good amount of it was just outright SEO spam. To be fair, the researchers said Google improved a bit over the year they watched it, though the average user probably wouldn't have noticed, and it was still riddled with spam at the end of the day.
And then there are the AI overviews; I have a love-hate relationship with them. They're insanely convenient, but at what cost? AI summaries often present personal opinions from forums as fact, so you have to verify all the information yourself anyway, and at that point you might as well keep scrolling like we used to before AI summaries. Plus, Google now runs ads inside the AI summaries too.
So it's not that Google search is completely useless, the vibe has simply changed and not for the better. Google decides what I see before I've even read anything. Most of the top of a results page is stuff Google wants in front of me, and not necessarily what I'm looking for.
Giving my local LLM the web
My local model can look things up without asking Google
A local model only knows what was in its training data. This is actually perfectly fine when you just want something explained or if the information you need to rely on is a few years old already. But anything recent will end up being a guess, for example, when I asked Gemma E4B what the date was, it confidently said 2016.
The fix is giving the model a way to look things up, and I ended up doing that in Jan, simply because it's the least annoying option. Setting up simply involves switching on a toggle in the settings and installing the Jan browser extension, rather than editing a config file. By default it searches through Exa on a free endpoint with no API key, which also means my questions weren't going through Google.
Search and fetch do different jobs - search gets the model a list of results with titles and short snippets, and fetch is the part that opens one of those pages and reads the actual text, otherwise the model is mostly working off snippets. Jan’s built-in web search gives the model both, so it can run a search and then pick a result to read on its own, and fetched pages get cut down to a set length before they reach the model so a single page doesn’t fill up its context.
LM Studio can do all of this as well, and I already have Brave Search set up in it, which is worth doing if you want an engine that runs its own index instead of borrowing from Google or Bing. It involves signing up for a Brave Search API and then editing the mcp.json file.
Letting a tiny model run my searches
My model now goes looking before it answers
On an 8GB card, picking your models will come down to whether it bothers to look things up at all. The Qwen family is very strong at tool calling, so I recommend picking something from there - my pick is Qwen 3.5 4B Q4_K_M for lighter chats and the 9B also at Q4_K_M for heavier work. Unlike Gemma, it uses search without having to be asked.
Once it does search, Jan makes the process pretty easy to follow. Qwen fetches the pages it wants to read and a little progress bar shows it working through them, which usually only takes seconds to complete. Jan also tells you which tool the model used, so you can tell whether it actually read a page or just went off the search snippets.
Searching didn't make Qwen 100% reliable though. When I asked what Obsidian released this month, it pulled ten sources and then dated the update October 2035. The heading under that said 2045, and the version number it gave me doesn't seem to exist anywhere. Most of those ten were third-party plugin changelogs, with only one coming from Obsidian itself, so I think it read a pile of loosely related pages and blended them together. But in its defense, my prompt was pretty vague; with Qwen you have to give more specifics to reel it in. Also, this is pretty much what the average AI summary looks like, anyway.
Jan lists the sources under every answer and you can open each one, so I'd go straight to the official page and check. The way it implements this feature is very user friendly - all your sources will be neatly stacked at the bottom.
Be mindful of context, though. One of my searches used 70% of my context window, and that was after I set Qwen to 30k context length! So if you're working with a smaller GPU, it's going to be about finding a balance between model, quant, context length, and the way you formulate your prompt. It might sound like hurdles, but it actually makes you more mindful about what to search for, which I feel is a lost art since AI entered the search engine game.
A week of asking instead of searching
I still open Google, but just for different queries, such as going to an account login or finding hyper-specific real-time information like which restaurants are open near me right now. But questions start in my local LLM now, typed out properly. Now I just want to get this same setup on my phone, but that's a whole other story.
