Auto-fine tuning Agentic Workflow with Qodo CLI

Running agentic AI workflows with the Qodo Command CLI is a powerful way to automate tasks but what’s even cooler is that you can also automate the whole fine-tuning of the agentic workflow process itself USING THE AGENT!

In a previous article I showed you how I use the Qodo Command CLI to automate the OpenSSF Scorecard security issues in my GitHub repositories. That was cool, but my agent instructions didn’t all work out great on the first try. In this article, I will show you how I can use the Qodo Command CLI to automate the fine-tuning of the agentic workflow itself.

Agentic Workflow Problems

Agentic workflows, and in general, “agentic” AI carries a lot in this term. To me (and by Anthropic’s definition) an agent isn’t just a program running through pre-defined code paths and invoking LLMs to do things. Instead, an AI Agent is a workflow (or system in a broader sense) that is characterized by agency, autonomy and reasoning. Meaning, an AI agent can make plan out what it needs to do, roll out its decisions, adapt to new information, and optimize its actions based on feedback.

The adapt to new information is a key part because this is how the agent actually fulfills its autonomy.

In the first time I instructed the agent to run the OpenSSF Scorecard security issues, I gave it sparse instructions that made sense to me but didn’t work out as I expected. What I mean by that is that I didn’t read the full capabilities of the GitHub MCP Server and for one example, it didn’t easily support some features needed by the agent to carry out. So what happened is the following log of actions:

• The agent would try to run a tool from the the GitHub MCP Server
• The tool would fail
• The agent will reason about it and say “let me try running this GitHub configuration with the gh tool at my disposal”
• The agent would run the gh tool and it would work well

So essentially, the agent was able to adapt to the new information that the GitHub MCP Server didn’t support the tool it was trying to run, and it used the gh tool instead.

If I left the agent configuration (the instructions) as is, then in the next time I run it, on the same or a different repository, it would likely run into the same problems (although, there’s some non-deterministic factor that it would not). So I needed to fine-tune the agent instructions to make sure it doesn’t run into the same problems again.

Fine-tuning Qodo’s Agentic AI Workflow

What would a human do in this situation?

I figured what I’d do is take a look at the running log of all the actions the agent took, find the tool executions that failed and then update the agent instructions to give it specific instructions to run some action via whatever path it succeeded to do before.

Well, of course I’m not going to do all of that manually, right?

Instead, I copied all of that log of prior agentic workflow and kicked off a new agentic workflow, a simpler one that reads the log file, reasons about which actions should be better communicated, and then update the agent.yaml file with the new instructions that would make it succeed the next time it runs. Self-healing agentic workflows! Feels like we’re in the future ;-)

Here is what it looks like:

And it identified a bunch of issues and rectified them:

And then goes on to update the agent.yaml file with the new instructions:

If you review the git diff above for changes, you can see how the instructions are now driving the agent to use the gh tool where the GitHub MCP Server fails, or update the commands allow-list configuration for the shell tool to include commands that the agent needs to run and had trouble with before.

Takeaways on Auto-healing Agentic Workflows

This was an interesting experience to get the agent to read its own log file and reason about the changes to make that improve it, then autonomously apply them to its own agent instructions.

Some manual work was still required from me, as I was executing the agent after its first pass, to give it the logs to process, and then re-run the agent with the updated instructions. It would make a futuristic outlook to allow the agent to keep a log as it is running and then update instructions on the fly. This is somewhat similar to how Memory and Task Manager extensions for Cursor and others work but trying this out is a task for another day ;-)