In this chapter we're going to implement the AI search tool allowing the AI to use the previously created knowledge base and embeddings and provide our users with answers. Let's go ahead and develop the search tool. Inside of packages, backend, convex, system, AI, tools, create a new file called search. Now let's go ahead and import our AI model. In my case that's going to be OpenAI from AISDK OpenAI.
In your case it is whatever you've been using so far. Then let's go ahead and let's import create tool from convex dev agent. Now let's go ahead and import generate text from AI package, let's import z from Zod, let's import internal from generated API, let's import support agent from agents support agent, let's import rag from rag. Let's export const search to be a tool. So create tool and let's give it a description.
So in the description you would provide something relevant for the AI to know when to use this tool. Search the knowledge base for relevant information to help answer user questions. As for the arguments, let's go ahead and use Zod to define this object. So we're going to accept a query which is a type of string and let's add the scribe here, the search query to find relevant information. So the way this tool will be used is the AI will detect the user query and then it's going to pass it here as the argument as in the user query.
And then we're going to use our embeddings and the rag component to search for that. So let's go ahead and mark this as a sink. Let's add arguments here. And now let's first check if there is no context that thread ID. Let's return missing thread ID.
We can't continue further. Now let's go ahead and let's find the conversation. Const conversation and let's go ahead and reuse our internal conversation query. If you Remember in the system we have conversations right in the system folder and in here we have get by thread ID. So let's reuse it here.
Await context run query and let's go ahead and use internal system conversations get by thread ID and pass in thread ID context thread ID. So the reason we have to use the internal query and with the reason we need to use context run query is very simply because context.database is not available here. The reason it's not available is because create tool is an abstraction over action so the context property does not have the database here. We are basically calling the third-party tool here, OpenAI, so we don't have direct access to the database in here. Now let's check if we don't have the conversation and let's simply return conversation not found.
So We're not throwing errors here because this is a tool. Instead we're just returning messages like this. Then let's grab the organization ID here, conversation.organizationID. And I think this should always exist, right? Yeah, so no need to check anything here.
And then let's do const searchResult and let's do await rag.search. Let's give it a namespace organization ID. Let's pass in the query and let's limit this to 5. The query will be arguments.query. And that's how you use the rag component to search through the embeddings based on the query and the namespace.
And now let's go ahead And let's standardize what we found. So context text will be found results in search results. My apologies, search result dot entries dot map for each entry return entry dot title or null filter for each title check if it's not null did I do this correctly oops t isn't null And then join by adding a comma and space like this. And add here is the context Go ahead and add backwards slash n, backwards slash n. So we have new line search result dot text.
So this isn't exactly for the human to read. This is just formatting the search result from the React component in a way that will be easier to pass further now to AI. So let's now generate the response. So the AI can respond, okay, here's what I found. Or basically, right now we can only return things like this but this isn't super readable imagine if you ask what is your most popular plan and then we just returned with found results in here and it just gave you a million results because the embeddings kind of match.
We don't want that. We just want to kind of summarize what we found and then we want to use AI again using generate text, passing the messages in here. The first one, role system and the content. Now inside of the content here, let's go ahead and simply say, you interpret knowledge, Base search results and provide helpful, accurate answers to user questions. Then let's add another message here.
This one coming from the user and the content will be user asked arguments dot query. So that's what the user asked. And then let's add backwards slash and backwards slash and search results content text Like this. And let's add the model and basically use your model here. Open ai.chat or gemini.chat and then you can just use whatever you've been using so far.
I will use 4.0.mini. And let's see, this is not context text, this is context text. Like this. And once this is generated let's do await support agent save message context thread ID context thread ID message role assistant content response text. Like that.
And let's return response text. Perfect. So what's going on with this tool? So basically the first thing we do is this tool will be called with the user query. So the user will ask what is your most popular pricing plan?
And then we're going to store that in this query here. And once we find the matching conversation by the thread ID, we're going to pass that query into our rag search. And that will return the embeddings or the entries from our knowledge base, which match that question. And it will return a text like this. And we're kind of going to standardize it so it looks just a little bit better.
But this is not exactly readable format. This is huge information right now. So we then pass that into your AI system, which goal is to interpret knowledge-based search results, which is this. These are the search results and provide helpful and accurate answers to user questions. And then we simply repeat.
So the user asked this and this and these are the search results using whatever model you've been using and then we simply store that response.txt from the AI above inside of the message. So now let's go ahead inside of our agents, support agent, My apologies, this will be inside of public messages. And in here, in the create action, where we already have escalate conversation, let's do search. From system AI tools, search. And now, if we've done everything correctly, we should have a working example.
So how do we exactly test this? Let's go ahead and first load our dashboard. Let's go inside of the knowledge base here. And let's also prepare our widget with a working organization ID. I would highly recommend that you use the exact organization ID that you're logged in with right So you can use the clerk dashboard to obtain organization IDs.
So you can test that very easily by just adding a new question here. Let me start a new chat. Test, test, test. Okay, so I am in the correct organization. My organization is test22.
So dashboardclerk.com, this is how you will do it. You will go inside of organizations, find test22, copy the organization ID and in your widget, make sure you're using that organization ID in the URL. Perfect. So let's go ahead now and let's try adding something to our knowledge base. So if you remember in the previous chapter, I believe I gave you knowledge base.
So I would suggest choosing something super simple that you can test. Let's see. We have pricing plans here. So that would be interesting to look at. But we can also do like frequently asked questions because I think this is the simplest possible thing.
So let me download knowledge base frequently asked questions here. If you already have it, great. So I'm just going to go ahead and add this again. Frequently asked questions. I will upload this.
And there we go. I have it right here. So if we've done this correctly, we should now be able to ask it something about frequently asked questions. Let's just open this file again just so we know what's inside. For example, if I ask how do I reset my password, I should get steps like this.
Or if I ask it what are the pricing plans it should return me with these pricing plans. Let's try something. Let's go inside of the widget. Let me refresh And let me ask what are your pricing plans? And let's see if it will use the tool or not.
Given that it's taking a while to answer I'm assuming it's using the tool and there we go. We offer three subscription tiers for our pricing plans. The starter plan, professional plan and the enterprise plan and it uses the salesexample.com. Let's see if this is made up or actual information. So $29.99 a month.
Let's see if that was correct or not. $29.99 a month. I think that is correct, right? And starter is $9.99 a month. Exactly.
Perfect. And saleasexample.com. Let's ask it something else. Do they offer refunds? So we should expect a 30-day money-back guarantee.
Let's go ahead and ask that. Do you offer refunds? Let's see. We should get back a return about a 30-day money back guarantee. Amazing, amazing job.
So this is how the embedding function works. Now we have the search tool to actually test it. But still, in my opinion, this wasn't fully perfect because I could still ask it things like, can I get married? Like something completely random. And you can see, it just tells you, yes, you can get married.
If you have specific questions about the marriage process feel free to ask right. So something weird it will still answer you the same way a normal chatbot would answer you. So the way I fixed this was simply by making prompts a bit stricter. So let's go ahead and do that now. I have added inside of my echo assets folder constants.ts in which I have all of the prompts that I have used.
So let's go ahead and add them and let's improve our app now. So inside of packages, backend, convex, system, AI, in here add a new file constants.ts and paste everything inside. So you should have support agent prompt, search interpreter prompt, operator message enhancement prompt. So three of them. So let's start by modifying the support agent prompt.
Go inside of your support agent and change the instructions to be support agent prompt from the constants. So let's take a look at it here. So we have some normal things here. You're a friendly, knowledgeable AI support assistant, but be careful here. So make sure that you name these tools the same way I named them, right?
So if you are calling these tools something else, then replace them. So what am I talking about? I'm talking about messages.ts in the public folder. When you create a new message here, we're passing some actions. So escalate conversation, resolve conversation and search.
So yeah, perhaps I should call this then search and I should call this escalate conversation and I should call this resolve conversation. Even though I'm pretty sure I can understand what I mean or maybe simpler leave the prompt as it is and then simply call this tool, escalate conversation. Call this tool again, resolve conversation. Call this tool and just add search. That's even simpler.
Or you can rename this tool if you want to. And then all of this will make sense. And in here, I give it some examples like blah, blah, blah. But basically what I'm telling it here is that it shouldn't make up information and that it shouldn't serve as anything other than a customer support agent tailored to its knowledge base. So critical rules never provide generic advice, only info from the search results.
Always search first. And if you are unsure, offer human support. Basically, don't guess, right? Because why should your AI be able to answer a question like, can I get married, right? That's kind of weird.
Because you're not building a general purpose customer support. You're building a very tailored customer support. So that's why I added this prompt to kind of improve it here. And let's see where else should we use it. So we just added the support agent prompt.
Now let's do the search interpreter prompt here. So this will actually be used instead of the search.ts instead of the tools folder which we just developed here. And in here I basically used this super simple prompt which is good enough but let's actually add search interpreter prompt here. Just so in here it also doesn't hallucinate. Basically, if it found information, return the information.
But don't return, For example, some generic information. In here, you can see I give it examples. Good response is specific information. Good response is also partial information and then offer to connect to a human to provide more details. Bad response is making things up.
So when it cannot find an information and then it responds, typically you would go to settings and look for a password option. That's wrong, right? Why should the AI tell you that? AI shouldn't guess about your product. It either knows the answer from the embeddings or it has no idea what it's talking about.
And in here I also have operator message enhancement prompt and I use this inside of messages in the private folder even though I'm not too sure it's needed. But yeah it's basically for the enhanced response action. When the operator clicks to enhance their message. Even though I think this is more than enough, if you want to, you can use the operator message enhancement prompt, because that's what I use when I develop this, but maybe it's a little bit of an overkill. But yeah, I'm using Markdown because it works well with OpenAI.
I think it works well with all other Gemini models and Anthropic models, but not. So let's Just for fun, try now as well. So I'm gonna go ahead and leave this chat here. I will start a new one and I'm gonna go ahead and ask it, can I get married? And let's see if we improve this or not.
And there we go, exactly what I wanted to say. I don't have specific information about that in our knowledge base. Exactly what I wanted to do. So what is the price of pepperoni pizza? Things like that.
So if we ask it something stupid it should always tell us that it can't find the information about that. Would you like me to connect with the human support agent? Yes please. And you will see conversation escalated to a human operator and that is also reflected in here. There we go.
It is escalated. And then we can chime in and say, hey, we really don't have pizzas. And if you want, you can enhance that here and send it back. So yes, Now it will understand, yes, they actually don't offer pizzas and you can get married using their app. But if you ask something specific like, what is your most expensive pricing plan?
It should be able to find that because it has the frequently asked questions embeddings. So let's see, will it answer us that or not? So I suggest that you test this by adding simple text file. Oh, it can't answer this because we switched to escalated mode, my apologies. So let me start a new chat here.
So what is your most expensive pricing plan? Let's see. Are we maybe too strict with our... Looks like we are not too strict. So the most expensive pricing plan is the enterprise plan which has custom pricing.
This is correct. So it correctly extracted from our pricing plan. So if yours for whatever reason is very stubborn and keeps saying, I don't have the information, I don't have the information. It could be that the prompt is a little bit too strict. So you might have to dumb it down a bit.
I basically, yeah, my goal was here, this. If it's not in the search results, you don't know it. Offer human help instead. But sometimes that can be a bit too strict and then AI is afraid to say anything other than recommend a human, which is technically a good thing, but it kind of defeats the purpose of having an AI helper. So I recommend that you test with like simple AI tools and then maybe try some more interesting ones like adding a PDF file or something, but try to keep them small in size.
It's gonna be much easier for you to test things. Excellent, amazing, amazing job. So what if this is not working for you? How can you debug and see what's going on? Well, in here, what I wanted to show you is the convex developer playground.
So if you go inside of agents and click on the playground here, they actually developed a UI where you can talk with the agents without having to build anything yourself. So let's add this to our backend here. Pnpm f backend add convex dev agent playground. Basically this right here. Then let's build playground.ts file in the convex folder.
Packages backend convex playground.ts so on the same level as auth and convex config. And in here I think we can copy all of these things. Let's paste them. Let me just see. It cannot find convex dev agent playground.
Did I add it correctly in my back end? I did. It should be available here. Let me just see. Did I do something incorrectly maybe?
No, It should be here. Maybe I just have to restart TypeScript server or refresh. Yeah, I just had to restart the server. And now we have to actually use the system here. AI agents support agent.
That's the only agent we have support agent. And then simply add support agent. Just like that. There we go. Once you have no errors in your playground here, Let's go ahead and continue further.
So in your project repo, issue yourself an API key. So let's go ahead and just copy this part because we're going to be using PNPM for this. So let's close this and let's have a new terminal open here but keep this running. I'm pretty sure you have to have this running. Make sure your backend is convex functions ready and then go inside of your packages.
Go inside of backend here and go ahead and run pnpm tlx convex run component agent api keys issues and in here I think you can just put test here and now you have this API key so just copy this API key here and then run convex dev agent playground so pnpmdlx what was it again Convex Dev Agent Playground inside of the backend folder here. Actually, I'm not sure if you even need to be inside of the backend folder for this. And now in here, go ahead and paste that API key you just copied and click submit here. And in here, you can select the existing user that you have and you will see all of their conversations. And this makes it easier for you to test things, right?
Because you can see that it uses the search tool, you can see the query that was passed, most expensive pricing plan, and you can see the return value which happened. So you can see exactly, for example, if I go up here, what is the price of pepperoni pizza? It used the search tool and the query was price of pepperoni pizza. And we couldn't find any information about that, right? So that's why this playground tool is quite useful because it's easier to debug.
If you are convinced that something should work, but it doesn't, open it in a playground and then try and see why it doesn't work. And you also have the entire system prompt in here so you can change it in here directly to see what works better. So if yours is very stubborn, try maybe changing the prompt from here and then try to get it to answer something. Perfect. So I would recommend that you develop, that you research the playground even more because it's super cool and it can definitely help you in assisting you to improve your prompts and AI agents.
And I think that marks the end of this chapter. So let's go ahead and see. We developed the search tool and we improved our prompts our is now able to tell us exactly what we found in the knowledge base. Let's go ahead and just shut down this like that. Let me just test one more thing.
So right now it's able to tell us everything about our pricing plans, right? But what if I go in here and what if I delete this? So now If I go ahead and try again and ask what is your most popular pricing plan. Let's see will it still be able to give us that and as you can see now it has no idea what is our pricing plan because we just remove that embedding and we remove that knowledge base. So all of our previous work was 100 percent correct.
Amazing amazing job. I think this is super impressive what you've just developed. So 22 AI search tool. Let's go ahead and merge all of that. 22, AI search tool.
Let's commit. Let me change to a new branch. 22 AI search tool. And let me publish this branch. And let's go ahead and let's review using CodeRabbit.
I'm pretty sure there are no security issues here whatsoever, but maybe we will get some interesting comment about our prompt or something we can improve. And here we have the summary. We introduced the Playground API for front-end access to AI agent features, including agent management, thread creation, messaging, and prompt context retrieval. So this is the last thing we did, a simple playground for you to test your AI agents. We also added a new knowledge base search tool enabling AI driven search and contextual responses within conversations exactly what we did.
In here we have a diagram explaining how all of this works. So let's see, the user asks a question, the support agent receives the question, and it invokes the search tool with query because of the prompt that we used. Now inside of that tool, what we do is we first fetch the conversation by thread ID provided from the context. We then return back the conversation or error. If the conversation exists we have the namespace organization ID.
Using the namespace we can search the rag component and we can return the file results from that context and then we simply interpret the answer using another AI model and we save that message in a thread and finally respond with that answer. Amazing. In here the only comments are to throw errors instead of returning strings which I'm pretty sure we cannot do inside of tools. I think we should return strings as we are doing. So yes I think This is perfectly fine the way we did it.
Great! Amazing, amazing job. Let's go ahead and just go back inside of our main branch. Click on synchronize changes right here. Okay.
And once the changes have been synchronized, always check the graph to confirm that you've just merged 22 back inside of me. And I believe that marks the end of this chapter. Amazing, amazing job and see you in the next one. Thank you.