A - Some answers, some more questions, and some satisfaction.

At least that's what I got when I burned 181.4 million tokens over the span of 10 days and some more following that while doing research with the help of AI tools...

So I want to talk about this experience. This will require multiple parts, will try to cover this in four parts, the what, the why, the how and the result.

Let's start with the 'what'

Starting with some fancy sounding shite!

I burned a total of 181.4 million tokens over the span of 10 days. (it is not much, I know, but this entire research exercise was not sponsored and was paid by myself.) So I burned 181.4 m tokens, out of which almost 64m-ish tokens were spent on processes which were discarded. Spent around $80-$100. Made around 1750+ API calls. Tested 7 prompt frameworks on 17 questions. Evaluated and scored all the 408 answers / responses given by the frameworks and scored them for 5 aspects. Used LLM as a judge, found out it was utterly useless and then almost scrapped it. Used a dataset with 1.5 lakh entries, realised the limit of my budget, my time and usage limits of my AI tools... and then reduced the dataset. Used 2 different Gemini Pro, Claude and Perplexity subscriptions to beat the biases in AI tools. Spent over $40 to realise that what I was doing had basic flaws and the script needed to be rewritten. Spent $40 more to do it correctly. With the help of Claude wrote a custom Python script for doing this research, later realised that there isn't anything like this out there and so made a custom tool out of it. Will be launching that tool soon so keep an eye out for my next posts.

Gotta choose them all!! Selecting frameworks for research.

Doing some initial discovery

So I started this, the way I would. I already knew a couple of frameworks, from doing research as mentiond in my previous blog. Got to know few more from YouTub, AI influencers and also some frameworks which I discovered myself (as I have written it in my first blog - check it here ). But I wanted to find out more of these frameworks. So I did some manual digging and first found out what the different frameworks out there were, which ones were well-known, which claimed to perform better. I did a little bit of reading and exploration.

I also did deep research on multiple factors via NotebookLM in 2 rounds, first giving results on around 80+ sources across two or three different questions I asked, and the second time it gathered around 50-something sources. I asked NotebookLM to do deep research on prompting frameworks: what types of frameworks are there, how they are used, different types of frameworks, etc. Whatever NotebookLM gathered, I took all that data and uploaded it to Claude as well as Gemini. Then I prompted these models to go through this data and find out how many total frameworks can be derived from these, which are the most used, which are claimed to be the best? Claude did the analysis, so did Gemini. I also asked NotebookLM to create a report for me explaining this particular thing, and it created one as well.

Narrowing down to make decision

Overall, there were many frameworks. In the analysis done by Claude the number came out to be around 22 or 23 frameworks. But when I went through the results of this analysis, I removed some frameworks like meta-prompting or one-shot prompting. Because I knew there were much better-structured frameworks out there. After fine-tuning, what remained was something like 12. Out of those, I again did the analysis and narrowed it down to 5 frameworks which I thought were worth spending my energy and resources on. In addition to these 5, I added 1 framework which I had discovered earlier by asking Gemini itself about which is the best framework for it. And I added another framework of my own making. A simple question everyone will have, 'why I did not use all 12 frameworks?' The answer to this we will see in the next blogs "the why" and "the how"

So starting with which frameworks I compared, here are the frameworks I compared.

All 7 frameworks

Picking the test

Next, I had the frameworks but I needed something to test these frameworks with, which meant I needed a task or tasks to give to these frameworks and then judge the results. As I am a UX designer, researcher and have been working in the field for almost 15 years, I decided to take something close to home and decided to test the frameworks for UX research. But research on what was the next question.

I wanted an easily available data set which would be open to the public and still be near to the actual UX research. I found many such data sets and I chose one of them. It was data showing app reviews for an OTT platform.

After the data, it was time to finalize the tasks. Created basic questions and used Claude to improve those. After multiple rounds of refinement, landed on the final 17 questions.

The 17 questions in 5 categories I used to test the frameworks.

What came out of this ?

Now .....Around millions plus tokens, some thousands of rupees, many hours. I also burned millionish tokens and AI credits for a day in under 2 mins (could have been faster if the internet and my setup would have allowed), I did all this when I compared multiple frameworks of prompts to see which gives the best result in a UX research use case, but also in the process I got insights on things like,

• What results do they give when put against each other for answering the same question on the same data set?
• Does the default framework give better answers? Is it good enough for most of the things?
• Is there any benefit of using the frameworks or can we just use defaults?
• Do AI models satisfice instead of going for absolute perfection?
• Which is the most effective framework in terms of tokens used?
• Or do all frameworks use the same amount of tokens?
• Which framework gives the most value (tokens spent vs results)
• Can frameworks help us reduce hallucination??
• Which are the most token-consuming frameworks?
• What happens when we give a persona to a framework?
• Is LLM an effective judge for checking the results as compared to a human evaluator?
• Do frameworks adhere to our instructions?
• Is it worth doing the UX research with the help of AI tools?
• If we are doing UX research with AI tools then how can it be done effectively?

---