I think we are all aware of the importance of reviewing AI generated content. Various AI chatbots talk about it like ChatGPT writes: "ChatGPT can make mistakes. Check important info." and Gemini Chat writes: "Gemini can make mistakes, so double-check it.". So I think it's something we are generally aware of. It looks like we all use AI generated responses in one way or the other, whether to write code, review documents, or to write articles.
As good as AI is, it can sometimes generate content that is inaccurate or lacks an understanding of your problem. We see this sometimes in responses that seem outdated or just does not fit our problem.
The causes can be numerous but some are: Out of Date knowledge where the AI model is not aware of or updated about the topic and when we don't provide it enough information about what we want it to do.
I was trying to set up custom authentication (with Better Auth) using firebase for an application I was building. Better Auth does not support Firebase or Firestore and I think it's logical given that Firebase already comes with authentication but I was trying to build my own authentication on top of Firebase using Firestore. Better Auth does provide a MongoDB adapter which is a NoSQL database similar to Firestore. So I decided to just view the code for MongoDB's adapter and convert it to work with Firestore.
As I have come to accept and adopt, I decided to use AI and for this, I decided to use Gemini (with Google AI Studio). I passed in my query and also the code from Better Auth for the MongoDB adapter and asked it to change it to support Firestore. It basically created an adapter and after fixing a few typescript errors, I was able to plug it into my Better Auth integration in my code.
This worked out well. All was good. Until I tested, and it basically broke down. The issue I was experiencing was that after the user logs in, they get redirected to the dashboard but Better Auth returned an empty session. I checked on Firestore and realized that there was a collection or data for the users and accounts but none for the user login sessions. Upon further investigation, it looked like Better Auth initially created the user login session but deleted it shortly after.
As I have come to accept and adopt, once more, I sent this error to Gemini. Gemini reviewed, thought about it and sent me a response that the issue was caused by me not supplying a schema to Better Auth config. It actually sent a schema for me to add to Better Auth initially when I asked for the code for the Firestore adapter but I just never considered it because I thought it was not needed for Firestore.
Perhaps I will have to show you the conversation for you to confirm but it was over several conversations where Gemini was still adamant that because Firestore is a custom adapter, it requires me to pass in a schema to the better auth config. To which I explained that, since I was using a similar implementation to MongoDB, it does not require a schema since Better Auth does not require a schema in the MongoDB adapter documentation. It went on and on about how this is different because the MongoDB adapter is official and so somewhere in the code of Better Auth, they are checking for MongoDB and auto injecting the schema.
I asked it to check once more and verify from the codebase because it was just assuming that there was some code in the Better Auth library doing the check but there was actually none. It went on again about there being some hidden implementation and even sent a schema for me to use.
I explained that the concept of schema is only needed by SQL like database adapters that needed to migrate the schema to their database and it had nothing to do with how the library itself operates since the library just saves the data and the schema ensures that the tables are created but in the case of Firestore, there's no need for a schema.
I even checked the Better Auth config and there was no such option as schema that could be passed and it sent another code saying, oh, it should rather be passed as models and it even showed me some made up reference to Better Auth library code that was showing models as a parameter that can be passed but upon checking the library, there was no such parameter in the documentation.
In the end, I went to ChatGPT, sent it context from my error logs and the Better Auth adapter code. It also went on a very weird journey of talking about the `sessionToken` being passed from Better Auth and firestore saving this as an id instead which was totally false. I shared all my adapter code but I realized that it was just outdated in terms of its understanding of Better Auth which was not surprising because I am on the FREE plan of ChatGPT and the supported model is not as advanced as the paid one. It did suggest some useful tips to debug like logging and trying to see if there's any error when creating or getting the data from firestore. I did check that and there were no errors as a result of those operations but it did give me some clues as to what could be the issue.
I paused ChatGPT and did some debugging myself on the actual code and it turns out, when I enabled session cookie cache, the issue was somehow resolved and the user was able to login. I then sent this information back to Gemini asking it to once more take another look at the problem without focusing on the schema. It reasoned and finally, it was able to identify the root cause. Which was that Firestore saves dates as Timestamps and Better Auth was trying to use those dates to do comparisons. This comparison failed so Better Auth then assumed the session was invalid and deleted it. So the user was logged in, the session was created and then immediately after, deleted, logging them out again.
The solution was to make sure that when data is being retrieved for Better Auth in the Firestore Adapater, all fields that are an instance of the Firestore Timestamps are converted to Javascript Date objects.
This turned out into quite a long essay on the conversation but I hope it helped me to drive or explain my point. AI is very good in generating content but it can sometimes hallucinate and imagine answers that don't even fit the problem like passing the schema or models to the Better Auth config.
So it's very critical that we review whatever responses we get from AI to make sure that it's valid. It might take you sometime to get very good results but it's better than going on a journey that is not worth it. Imagine I agreed and went on a journey to change the schema? I would have spent several hours trying to update the modelNames of the various Better Auth config options without avail or solving the problem.
Let's take a bit of time to make sure we review whatever the AI generates.
Custom built AI agents using tools like Crew AI, Google ADK, Google Agentspace, Llama Index and other AI agent building tools can go a long way to help us in terms of reviewing and validating AI generated response. You can think of it as a way to validate AI generated content by providing tools or instructions for the AI models or LLMs to validate the work that they have done. You can use these tools to build very good AI integrations that involve steps such as content generation, content review and content editing allowing you to generate more rich responses.
I assume to some extent, "vibe coding" tools like Firebase Studio and Claude Code help to perform such actions and they also have access to the full codebase. And who knows, perhaps if I had fed Gemini the whole codebase, it could have found the issue earlier but to some extent, it had the code for the adapter already, it just couldn't figure out that it was due to Better Auth processing the response from Firestore.
I'll end by saying, "Review AI generated content".
