Forem: Jon McLaren

Current Date and Time for agents

Jon McLaren — Fri, 24 Jan 2025 03:00:50 +0000

This is a submission for the Agent.ai Challenge: Full-Stack Agent (See Details)

What I Built

In building agents I came across an annoying problem. LLM's are often terrible when it comes to the concept of time. They're often bad at:

Sorting items by date
Consistently understanding relative words like Yesterday, Tomorrow, Next week, etc.
Understanding the concept of what is in the past and what is in the future - date and time wise, it's not going to tell your fortune.

Some LLM's behind the scenes are provided the current date and time in a system prompt. That's great, that doesn't always apply to every LLM though. The understanding of relationships is also interesting and I think the problem is that "Tomorrow" "yesterday" are variables that humans understand as being relative, but would appear in written language of training data that is already "old" or in the past from the current time and date.

Additionally LLM's just like how they're not natively good at math, have a hard time with dates and times because probably because they're basic math. It's likely partially an issue of tokens can be broken up multiple different ways for a date/time and the LLM isn't seeing the actual words and numbers, it's seeing basically id numbers that represent these concepts.

So the strategy is that I take away some of the "math" and give the LLM a little extra understanding of time so that it can work with time the way humans do.

Demo

This agent is fully designed to be used by your agent. All you need to do to use it within your agent builder, click add action, go to "advanced", choose invoke agent. Enter datetime. 1 optional parameter will appear for setting a time_zone (defaults to UTC if time zone is not provided).
You can go directly to the agent and click the dropdown to see the available time zones, but basically it's all standard time zones. For example for New York you could enter:
America/New York

Set the output variable to something like current_date_and_time, then when you use an LLM action, you can insert that variable somewhere in your prompt.

That's it! Now ask it to do something it could only do if it understood dates and times and it should perform better.

In order to make trying it easier I built a demo agent that does the above

Ask it questions that would require understanding of date and time such as reordering tasks, use relative words, try asking it what month it will be x months from now, etc, regardless of the LLM model you use it will perform better than without the agent.

Check out the actual current Date and Time agent

Why does this meet the criteria for full stack

This is a utility for other agents to use. Behind the scenes what this agent does is hits a serverless function to grab the current date and time and provide just enough information to better enable LLM's to properly handle dates and time.

I added this agent entry in-case the fact that it's an agent invoked by my Agent Inspector Agent - would disqualify the Agent Inspector from the Full Stack competition.

Agent.ai Experience

I've enjoyed building on Agent AI, it's a cool tool that is among the pioneers of supporting agents working together.

Agent Inspector (Debug your agent and LLM actions) - Assembly of Agents

Jon McLaren — Wed, 22 Jan 2025 20:17:32 +0000

This is a submission for the Agent.ai Challenge: Assembly of Agents (See Details)

What I Built

Agent Inspector is a must-have tool for those building and iterating on agents on Agent.ai. Agent Inspector provides you information beyond what Agent.AI's built in debugger shows. It tells you what format the output of an agent should be, validates if the agent is in-fact returning that output, assesses toxicity of the agent's output. Provides a clear overall Pass/Fail result. If there's opportunity to improve the prompt, or provide more data to the agent, it makes suggestions.

The agent has become useful enough that I use every time I create a new agent or am iterating on an existing one.

Without giving away all the secrets publicly here are some of the things that have gone into it:

LLM actions with specific models chosen for special skills they have.
Invoking agents. I created another utility agent that simply grabs the current time and date and returns it in a useful way for agents. The current date and time agent uses a serverless function to grab the date and time and return a JSON response providing that information in a way that is easily understood by LLMs to better handle time related requests. This modular way of building allowed me to both help others build agents that need to utilize time and date, as well as power the functionality for this agent. The Debug Agent is also intended to be used by other agents - that is the primary way to use it.
To build the Agent Inspector and ensure quality in all of it's tests I actually had to build an agent for testing it.
Multiple prompt engineering techniques.
The agent does hit a serverless function within the previously mentioned current date and time agent I created that it invokes.
Working around some issues found in the if statement action and the inability to export multiple variables to other agents - This agent actually has a companion JSON variant that returns the data in JSON so you can take automated action based on the test results.

Tests performed by the Agent Inspector:

Expected Data type of output based on prompt
Validation that the output is actually matching the prompt with confidence score.
Relevancy of output to prompt provided - with confidence score.
Likelihood of hallucination - validates LLM has all needed information to provide answer. - with confidence score.
Toxicity - is the response using offensive or harmful language. (This takes into account the prompt and does not simply flag simply using 1 "bad" word as toxic, so if the prompt is talking about a subject in an academic sense it is not going to say the response is toxic unless it goes too far.)
Fluffy/substance test of text answer - Checks the response for fluffy content that doesn't have much substance. Think of it like this: if it's supposed to be a blog post - is this something someone's going to in turn just pass to an LLM to summarize because it's not concise and meaty enough?
Suggested prompt revisions - based on all of the tests the agent suggests improvements to encourage better results from your LLM action or agent.
Execution timing - know how long your agent takes to execute from start to finish. Provides some best practices and norms.

Why it meets this criteria

This agent relies on another agent for a step in it's process.

Demo

Watch a video showing Agent Inspector as well as instructions to set it up
Try the agent yourself

Agent.ai Experience

Overall I've enjoyed the experience, and want to push the platform to do things it was never designed to do. I'm providing feedback on a lot of areas where the platform can be improved - understand that my feedback is coming from a place of appreciating it for what it is, and just wanting to see it further thrive. Being a developer I'm always going to want more, but there are limitations I ran into that prevent me from being able to deliver even more with this agent and other agents.

Feedback for the Agent.ai team.

Agent Inspector (Debug your agents) - Productivity Pro

Jon McLaren — Wed, 22 Jan 2025 20:16:44 +0000

This is a submission for the Agent.ai Challenge: Productivity-Pro Agent (See Details)

What I Built

The agent has become useful enough that I use every time I create a new agent or am iterating on an existing one.

Without giving away all the secrets publicly here are some of the things that have gone into it:

LLM actions with specific models chosen for special skills they have.
Invoking agents. I created another utility agent that simply grabs the current time and date and returns it in a useful way for agents. The current date and time agent uses a serverless function to grab the date and time and return a JSON response providing that information in a way that is easily understood by LLMs to better handle time related requests. This modular way of building allowed me to both help others build agents that need to utilize time and date, as well as power the functionality for this agent. The Debug Agent is also intended to be used by other agents - that is the primary way to use it.
To build the Agent Inspector and ensure quality in all of it's tests I actually had to build an agent for testing it.
Multiple prompt engineering techniques.
The agent does hit a serverless function within the previously mentioned current date and time agent I created that it invokes.
Working around some issues found in the if statement action and the inability to export multiple variables to other agents - This agent actually has a companion JSON variant that returns the data in JSON so you can take automated action based on the test results.

Tests performed by the Agent Inspector:

Expected Data type of output based on prompt
Validation that the output is actually matching the prompt with confidence score.
Relevancy of output to prompt provided - with confidence score.
Likelihood of hallucination - validates LLM has all needed information to provide answer. - with confidence score.
Toxicity - is the response using offensive or harmful language. (This takes into account the prompt and does not simply flag simply using 1 "bad" word as toxic, so if the prompt is talking about a subject in an academic sense it is not going to say the response is toxic unless it goes too far.)
Fluffy/substance test of text answer - Checks the response for fluffy content that doesn't have much substance. Think of it like this: if it's supposed to be a blog post - is this something someone's going to in turn just pass to an LLM to summarize because it's not concise and meaty enough?
Suggested prompt revisions - based on all of the tests the agent suggests improvements to encourage better results from your LLM action or agent.
Execution timing - know how long your agent takes to execute from start to finish. Provides some best practices and norms.

Why this agent meets the criteria for this challenge

This agent directly provides meaningful insights and validates LLM and agent output enabling agent builders to move significantly faster to produce reliable quality agents. It's designed to be part of the agent building workflow and can even be used for automating actions within your agent - for example if there is toxic content in an LLM action response, you can choose to have your agent do something other than return that directly to the user.

Demo

Watch a video showing Agent Inspector as well as instructions to set it up
Try the agent yourself

Agent.ai Experience

Feedback for the Agent.ai team.

Agent Inspector (debug your agents and LLM actions) - Full stack

Jon McLaren — Wed, 22 Jan 2025 20:14:20 +0000

This is a submission for the Agent.ai Challenge: Full-Stack Agent (See Details)

What I Built

The agent has become useful enough that I use every time I create a new agent or am iterating on an existing one.

Without giving away all the secrets publicly here are some of the things that have gone into it:

LLM actions with specific models chosen for special skills they have.
Invoking agents. I created another utility agent that simply grabs the current time and date and returns it in a useful way for agents. The current date and time agent uses a serverless function to grab the date and time and return a JSON response providing that information in a way that is easily understood by LLMs to better handle time related requests. This modular way of building allowed me to both help others build agents that need to utilize time and date, as well as power the functionality for this agent. The Debug Agent is also intended to be used by other agents - that is the primary way to use it.
To build the Agent Inspector and ensure quality in all of it's tests I actually had to build an agent for testing it.
Multiple prompt engineering techniques.
The agent does hit a serverless function within the previously mentioned current date and time agent I created that it invokes.
Working around some issues found in the if statement action and the inability to export multiple variables to other agents - This agent actually has a companion JSON variant that returns the data in JSON so you can take automated action based on the test results.

Tests performed by the Agent Inspector:

Expected Data type of output based on prompt
Validation that the output is actually matching the prompt with confidence score.
Relevancy of output to prompt provided - with confidence score.
Likelihood of hallucination - validates LLM has all needed information to provide answer. - with confidence score.
Toxicity - is the response using offensive or harmful language. (This takes into account the prompt and does not simply flag simply using 1 "bad" word as toxic, so if the prompt is talking about a subject in an academic sense it is not going to say the response is toxic unless it goes too far.)
Fluffy/substance test of text answer - Checks the response for fluffy content that doesn't have much substance. Think of it like this: if it's supposed to be a blog post - is this something someone's going to in turn just pass to an LLM to summarize because it's not concise and meaty enough?
Suggested prompt revisions - based on all of the tests the agent suggests improvements to encourage better results from your LLM action or agent.
Execution timing - know how long your agent takes to execute from start to finish. Provides some best practices and norms.

Why this agent meets the criteria

This agent invokes another agent I created datetime which is designed to swiftly get the current date and time for a specific timezone, it returns that data in a format that agents can understand. That agent uses a serverless function. I wanted to open that functionality up to others to use so that is why I'm invoking the agent within Agent Inspector rather than using the serverless function action directly, keeping my code modular and reusable.

Additionally I built Agent Inspector with a companion JSON response version. The purpose of that agent is that it responds with the same report but in JSON and doesn't display it to the user. Instead you can then take that JSON either store it in a database to keep a log of data you can use to improve your agent, or you could have your agent actually take action immediately with the results of the report. For example if it was going to provide a toxic response to the user the agent can be set up to see the report, and take an action other than returning directly the toxic response.

Originally the JSON functionality was built into Agent Inspector itself but I found that the If statement capability and invoking agents capability within Agent.AI to have integration issues preventing that from properly working. So the solution was to provide this second agent for folks wanting the JSON.

Demo

Watch a video showing Agent Inspector as well as instructions to set it up

Try the agent yourself

Agent.ai Experience

Feedback for the Agent.ai team.