LLMs, Agents, and Tools: What Is Actually Doing the Work?

When I started planning my local agentic coding environment, I had a few questions in mind:

• What concepts don't I know but need to understand first?
• What do I actually need to install? Models? Software? Plugins?
• How do all these pieces connect?
• And, probably most importantly, who is doing what?

Before setting up anything, I needed to identify the pieces, understand their responsibilities, and then figure out how to connect them into the environment I wanted to build.

A quick search brought up a collection of terms I had heard before but didn't fully understand: LLMs, parameters, quantization, context windows, tokens, inference, model runtimes, agents, and tools.

Some of these concepts are closely related, but they aren't interchangeable. And if I'm going to experiment with different models and tools later, I need at least a basic understanding of what I'm actually changing.

So let's start with the most obvious piece: the model itself.

The LLM: the reasoning engine

LLM stands for Large Language Model. In simple terms, this is the AI model that receives information and generates a response.

For a coding model, that could be something like:

Explain this Spring Boot service and suggest an improvement.

The model receives the prompt and whatever code or additional information I provide, processes it, and generates an answer.

I find it useful to think of the model as the reasoning engine, or simply the brain of the system.

I also needed to understand a few characteristics of the model itself.

What does 7B, 14B, or 30B actually mean?

When looking for local models, names and descriptions quickly start showing numbers such as:

7B
14B
30B
70B

The B means billions of parameters.

Parameters are the numerical values learned by the model during training. A 7B model contains roughly seven billion of them; a 30B model contains roughly thirty billion.

I don't find it particularly useful to think about the individual numbers. What's more useful is to think of parameter count as a rough indication of the model's capacity.

A larger model can potentially represent more complex patterns, but that doesn't mean:

30B = 4x smarter than 7B

Model quality isn't that simple. Architecture, training data, fine-tuning, specialization, and several other factors matter. A newer or better-trained smaller model can outperform a larger one at a particular task.

For my experiment, the practical lesson is simpler:

More parameters can provide more capability, but they also require more resources.

And resources become very important when the AI server happens to be the same 16 GB Mac I'm using to run IntelliJ, a browser, databases, and everything else.

Quantization: making large models smaller

Parameter count led me directly to another term I kept seeing: quantization.

Model versions often have names containing things such as:

Q4
Q5
Q6
Q8
FP16

Quantization is essentially a way of representing the model's parameters using less numerical precision.

Imagine that a value inside the model was represented as:

0.183729481

A lower-precision representation might store something closer to:

0.18

The second representation loses some precision, but requires less space.

Apply that idea across billions of parameters and the memory savings become significant. That's what makes models that would otherwise be too large potentially usable on consumer hardware.

A useful way for me to separate these two concepts is:

Parameters describe roughly how large the model is. Quantization describes how precisely those parameters are represented.

Or, with a slightly less technical analogy:

Parameters = how big the library is. Quantization = how compressed the copies of the books are.

A 30B Q4 model still has roughly 30 billion parameters, but those parameters are stored much more compactly than they would be at higher precision.

That compression can come with some loss in quality, especially when it becomes aggressive. But without it, many larger models simply wouldn't fit on my computer at all.

So the goal isn't necessarily to run the model with the highest theoretical quality.

It's to find the best useful model that fits comfortably inside the hardware I actually have.

Context window: what the model can work with

Another number that appears frequently when comparing models is the context window:

32K
64K
128K
256K

The context window defines how much information the model can work with during an interaction.

For coding, that information might eventually include:

My request
+
Project instructions
+
Relevant source files
+
Previous conversation
+
Tool results
+
Compiler or test output

A useful analogy is to think of context as the model's working memory for the current task.

This becomes particularly interesting for software development because a model doesn't automatically know my entire repository. Somehow, the relevant parts of the project need to reach the model as context.

A larger context window means more information can potentially fit, but it doesn't mean dumping an entire large project into every request is a good idea. More information isn't automatically better information.

I'll explore context much more later because it becomes one of the most important parts of working with coding agents. For now, the important thing is simply understanding that the model can only reason about the project information available to it at that moment.

Tokens: how that information is processed

Then there are tokens.

We normally think in terms of words, sentences, or lines of code. Language models process text as smaller units called tokens.

Very conceptually:

Tokens matter for several reasons. They consume space inside the context window, affect how much information a model can process, influence inference performance, and become part of the cost calculation when using many cloud AI services.

Locally, I don't receive a bill because I processed another thousand tokens, which is one of the reasons this environment is interesting for learning. But tokens aren't suddenly free from a computational perspective. My computer still has to process them.

So even in a zero-dollar local setup, token efficiency still matters.

Inference: actually running the model

One more word I needed to understand was inference.

Fortunately, this one is simpler than it sounds.

Inference is the process of taking a trained model, giving it some input, and running it to generate an output.

Training is what created the model.

Inference is what I'm doing when I use it.

For a cloud AI service, that inference happens on somebody else's hardware.

For the environment I'm trying to build, it happens on my Mac.

And that brings hardware directly into the picture.

Why my hardware suddenly matters

My Mac has an Apple M2 processor and 16 GB of unified memory.

The important part for this experiment is the 16 GB.

The model needs memory while it's running, but it isn't the only thing using that memory. macOS needs some. IntelliJ needs some. My project needs some. A local database might need some. Chrome will happily take whatever is left and probably ask for more.

So the actual environment looks more like:

16 GB unified memory
        │
        ├── macOS
        ├── IDE
        ├── Browser
        ├── Project
        ├── Local services
        └── AI model

If the model comfortably fits alongside everything else, great.

If it consumes most or all of the available memory, macOS may start moving memory contents to disk, a process called swapping. That can make the entire machine dramatically slower. The original experiment notes made this hardware constraint explicit: memory, rather than available disk space, was expected to be the practical limit when choosing models.

This is why model size and quantization aren't just interesting specifications on a download page.

They directly affect whether I can actually use the model while continuing to use my computer as a development machine.

At this point, I understand what the model is and some of the main characteristics that affect whether I can run one locally.

But I still need something to actually run it.

The local runtime and management tool

A language model isn't simply an application I open and use by itself.

At a high level, something needs to:

That's where a model runtime enters the environment.

For this experiment, the main candidate is Ollama.

Ollama provides a relatively simple way to download and manage local models, load and execute them, switch between models, and expose them through a local API (Application Programming Interface) so other software can communicate with them.

The important distinction is:

Ollama is not the AI model.

Conceptually:

Ollama's job is to make running and communicating with those models practical.

This is useful for the experiment because I don't want to spend my time building infrastructure just to execute every model I want to test. Ideally, I want to keep the runtime relatively stable and swap models underneath it.

That gives me a common way to compare different local models later.

It also means I need to keep the responsibilities separate when evaluating the results. If Model A produces terrible Java code, that doesn't necessarily mean Ollama is bad. Ollama ran the model; the model generated the response.

The same distinction will matter as more pieces enter the environment.

The API connects the runtime to other software

Ollama also provides another important piece: a local API.

An API is simply a defined way for one application to communicate with another. In this case, another application can send a request to Ollama asking it to use a particular model.

Because Ollama is running locally, this communication can happen through localhost, which essentially means this same computer.

Now I have a model and something capable of running it.

There's still one major piece missing.

The agent layer

What I ultimately want isn't just a local chatbot.

I want to give the system a development task and eventually allow it to interact with my project. That means something needs to connect the model to the development environment.

This is the agent layer.

For now, I only need a high-level definition:

The agent is the software layer that coordinates the model with the information and tools needed to work on a task.

Those tools could eventually provide capabilities such as:

Read files
Edit files
Run terminal commands
Run tests
Inspect Git

The model itself doesn't inherently perform those actions. It provides the reasoning that helps determine what should happen, while the agent provides the workflow around it and access to the tools that can interact with the project.

Exactly how that works is important enough that I'll explore it separately. There are questions around how an agent chooses actions, how it observes results, how much autonomy it has, and what happens when something fails.

For this step, I just need to know where the agent belongs.

Putting the local environment together

Now the pieces finally start forming the environment I originally wanted to build:

Each piece has a different responsibility:

Piece	Responsibility
LLM / Model	Provides the reasoning and generates output
Model runtime	Loads and executes the model
Ollama	The local runtime and model-management tool I'm using for this experiment
Tools	Provide ways to interact with files, terminal, tests, Git, and other parts of the environment
Agent	Connects the task, model, and available tools into a working system

And now I can finally answer the question I started with: what happens when I eventually send a prompt to my local coding agent?

In simplified form:

I give the agent a task
        ↓
Agent prepares the request
        ↓
Ollama runs the local model
        ↓
Model reasons about the request
        ↓
Response returns through Ollama
        ↓
Agent interprets the response
        ↓
Agent can use its available tools
        ↓
Project

There will eventually be more back-and-forth than this diagram shows, but that's a topic for the next step.

For now, I have the mental model I was looking for:

The model provides the reasoning. The runtime executes the model. The tools provide access to the development environment. The agent connects everything together.

Now I know what the pieces are.

The next question is much more interesting: what can the agent actually do with them?