--- title: Comparing the Quality of Responses of AI Models date: 2026-04-09T08:05:00-04:00 author: cc_admin canonical_url: "https://caffeinecreations.ca/blog/comparing-the-quality-of-responses-of-ai-models/" section: Blog --- ![Apples to oranges](https://caffeinecreations.ca/uploads/blog/_1920x660_crop_center-center_none_ns/apples-to-oranges.jpg) - [AI](https://caffeinecreations.ca/blog/category/ai/) # Comparing the Quality of Responses of AI Models Most AI comparison posts test models on generic questions and call it a day. That's not very useful if you're a developer working in a specific CMS. I wanted to see how these models perform when the questions actually require domain knowledge. I ran two Craft CMS prompts across five models — *Claude Opus 4.6*, *Claude Sonnet 4.6*, *ChatGPT* (free tier), *Ollama Gemma4:26b*, and *Ollama Llama3.2* — scoring each on quality, readability, and more. Here's what I found. **A note on the Ollama models:** Both Gemma4:26b and Llama3.2 run locally on my machine. Speed will vary depending on your hardware — RAM, CPU, and whether you have GPU acceleration available all play a role. My results reflect my setup, so take the speed numbers as relative rather than absolute. I chose these prompts specifically to test each model's Craft CMS knowledge. One covering project setup fundamentals, the other a real-world troubleshooting scenario. Any model can talk about project structure or performance, the question is whether it knows how Craft actually handles it. #### The First Prompt > What are the best practices for structuring a Craft CMS project? Include recommendations for template organization, Matrix field strategy, plugin selection criteria, and local development setup. **Note:** speed is from time prompt is entered until the first word of the response appears. **Scoring:** Quality and Readability are each scored out of 5. 0">AI ModelWord CountTimeQualityReadabilityStrengthsWeaknessesClaude Opus 4.66815 seconds55Easy to understand explanations. Excellent plugin selection criteria including examples of popular plugins.matrix field strategy - recommended a switch for every block rather than a single include calling the handle for each block type (my personal preference is the include over the switch). No summary like Gemma providedClaude Sonnet 4.67542 seconds55Template organization - shows a directory tree - easy to follow. Good best practices recommendations. Good matrix strategy. Good advice on choosing plugins. Module first recommendation well explained.noneChatGPT (Free)3701 second3.55recommend directory tree structure for templates. Best practices. Good advice on choosing plugins but shallowuse module use case not clear. Advice is a little on the simple side and lacking depth.Gemma4:26b621100 seconds55Recommends components. Excellent plugin selection criteria. Best practices. Good advice on choosing plugins. Good summary table at end of filetoo much effort on local development.llama3.2:latest4047 seconds35Template advice mostly solidSome unnecessary and irrelevant bits including advice to participate in the community and setting up version control. suggested using grid fields and more #### The Second Prompt > My Craft CMS site is running slow. What are the most common performance bottlenecks and how would you diagnose and fix them? 0">AI ModelWord CountTimeQualityReadabilityStrengthsWeaknessesClaude Opus 4.65174 seconds35N+1 (but doesn't call it that), eager loading. Caching, Debug toolbar, Quick wins sectionNo mention of Blitz or other caching plugin. Only one example template code. Answer feels like it is lacking anything helpful. Follow-up prompts likely will draw out more information.Claude Sonnet 4.64843 seconds55Mentions N+1, eager loading, and caching. Debug toolbar. Mentions Blitz plugin, Redis, and native {% cache %},NoneChatGPT (Free)7402 seconds55Mentions N+1, eager loading and caching. Recommends debug toolbar. Provides template examples. Recommends Blitz for caching. Cloudflare and cache headers. all advice is easy to read and apply to a project. Recommendations for highest ROI to achieve improvements quicklyNoneGemma4:26b58892 seconds55Extremely detailed. Mentions N+1 and eager loading. Caching strategies such as Redis/Memcached or native {% cache %} tags. Good summary checklist at bottom of file. No mention of Blitz or other cache pluginllama3.2:latest4117 seconds3.55Good organization of information and recommendations.Mentions inefficient db design. despite db design is out of control of the developer - created by Craft CMS. No mention of Blitz or other cache plugin. ### Conclusion After running both prompts across all five models, *Claude Sonnet 4.6* came out on top. It consistently gave accurate, Craft-specific answers with good structure. For my workflow it's also the practical choice as it uses fewer tokens than Opus, which is important when you're on the free or Pro plan rather than Max. That said, a few other models impressed me: *Llama3.2* punched above its weight given its size, and *Gemma4:26b* produced solid responses. The 90+ second wait time is the only real knock against it, though that's a hardware problem as much as a model problem. *ChatGPT* surprised me on Prompt 2, producing one of the more thorough and actionable responses of the group. This was a limited test with two prompts, and I plan to revisit it with a prompt focused on building out a template to see if the rankings hold. If you want to run the same prompts yourself, I've included them above. It's worth seeing how the results compare on your own setup. Want to read the full responses and draw your own conclusions? I've zipped up all ten markdown files — [download here](https://caffeinecreations.ca/uploads/downloads/compare-ai-model-prompts.zip). ## More articles [![Running a Local LLM on Windows with Ollama Thumbnail](https://caffeinecreations.ca/uploads/blog/_680x320_crop_center-center_65_none_ns/Ollama.jpg)### Running a Local LLM on Windows with Ollama](https://caffeinecreations.ca/blog/running-a-local-llm-on-windows-with-ollama/) [![Displaying Exif Data in Craft CMS Templates Thumbnail](https://caffeinecreations.ca/uploads/blog/_680x320_crop_center-center_65_none_ns/exif.jpg)### Displaying Exif Data in Craft CMS Templates](https://caffeinecreations.ca/blog/displaying-exif-data-in-craft-cms-templates/) [![Adding Glossary Tooltips to Redactor Fields Using Craft CMS and Tippy.js Thumbnail](https://caffeinecreations.ca/uploads/blog/_680x320_crop_center-center_65_none_ns/tooltips.jpg)### Adding Glossary Tooltips to Redactor Fields Using Craft CMS and Tippy.js](https://caffeinecreations.ca/blog/adding-glossary-tooltips-to-redactor-fields-using-craft-cms-and-tippy-js/)