<![CDATA[Beyond the Buzzwords, with Katherine Munro]]>https://katherine-munro.comhttps://substackcdn.com/image/fetch/$s_!q44v!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd050995d-590b-4ad8-9845-fabeb40968a1_1280x1280.pngBeyond the Buzzwords, with Katherine Munrohttps://katherine-munro.comSubstackSun, 05 Apr 2026 00:44:43 GMT<![CDATA[LLMs as Judges: Practical Problems and How to Avoid Them]]>https://katherine-munro.com/p/practical-problems-with-llms-as-judgeshttps://katherine-munro.com/p/practical-problems-with-llms-as-judgesWed, 03 Sep 2025 09:57:28 GMT
A collection of sticky notes and emojis related to evaluation (graphs, ticks and crosses, etc)
All images: Author provided.

My last post was all about conceptual problems with using Large Language Models to judge other LLMs. In it, I presented the “gotchas” that teams should watch out for when building LLM-powered products. Of course, the point is not to say that all LLM evaluations are bad, or that human judges are always better. There are defini…

Read more

]]><![CDATA[Large Language Models as Classification Engines: Overkill, or Awesome?]]>https://katherine-munro.com/p/llm-classifiers-overkill-or-awesomehttps://katherine-munro.com/p/llm-classifiers-overkill-or-awesomeFri, 11 Jul 2025 06:25:40 GMT
A simple cartoon featuring speech bubbles paired up with various labels.
With real-world examples from our customer-service voicebot. Source: Author-provided.

Have you ever wanted to build a trillion-parameter labelling machine? And by that I mean, would you use a Large Language Model to solve a classification problem?

It’s an unlikely fit for an LLM. And yet, there are good reasons to use LLMs for such tasks, and emerging arc…

Read more

]]><![CDATA[Using one LLM to Judge Another? Here’s Five Reasons You Shouldn’t]]>https://katherine-munro.com/p/the-problem-with-llms-as-judgeshttps://katherine-munro.com/p/the-problem-with-llms-as-judgesFri, 25 Apr 2025 07:47:58 GMT
A collection of sticky notes with ‘watch out!’, ‘gotcha’, and other such messages, which will be discussed in this article.

As teams across the world scramble to build things with Gen AI and Large Language Models, countless startups are scrambling to build LLM evaluation tools to serve them. Many such tools use LLMs to judge the system’s output, be it the final results or just the LLM-component’s responses. This makes sense, for some stages of the product development process…

Read more

]]><![CDATA[Steal My Idea: Evaluating LLM Systems with Production Data at Scale]]>https://katherine-munro.com/p/evaluating-llm-systems-with-realistic-data-at-scalehttps://katherine-munro.com/p/evaluating-llm-systems-with-realistic-data-at-scaleTue, 11 Feb 2025 10:02:05 GMT
Building things with LLMs that need to work in the real-world? This post is for you. Source: Author provided.

In my last post [1], I described how my team and I have been testing our WIP conversational assistant, despite having no baseline or benchmarks, and despite the LLM testing landscape being relatively immature. But there’s still a gap when it come…

Read more

]]><![CDATA[How Big Tech Is Exploiting Content Creators, And (Trying To) Get Away With It]]>https://katherine-munro.com/p/how-big-tech-exploits-content-creators-for-aihttps://katherine-munro.com/p/how-big-tech-exploits-content-creators-for-aiFri, 10 Jan 2025 21:28:33 GMTThis post originally appeared on my Medium blog, katherineamunro.medium.com

Union of Musicians and Allied Workers protesting at Spotify’s corporate headquarters in San Francisco.
That’s right you heard us, Open AI. Source: Patrick Perkins on Unsplash

If you’re reading this, you’re part of the content creator ecosystem: either as a fellow writer, casual consumer or a Medium subscriber. You help keep the system running, which means you have a stake in the su…

Read more

]]><![CDATA[No baseline? No benchmarks? No biggie! An experimental approach to agile chatbot development]]>https://katherine-munro.com/p/agile-chatbot-development-lessons-learnedhttps://katherine-munro.com/p/agile-chatbot-development-lessons-learnedFri, 23 Aug 2024 10:42:05 GMT
Today’s post recaps my recent talk on lessons learned trying to bring LLM-based products to production. You can check out the video here.

What happens when you take a working chatbot that's already serving thousands of customers a day in four different languages, and try to deliver an even better experience using Large Language Models? Good question.

It'…

Read more

]]><![CDATA[How to save the planet? Just add data, AI, and a little financial selfishness.]]>https://katherine-munro.com/p/to-save-the-planet-just-add-datahttps://katherine-munro.com/p/to-save-the-planet-just-add-dataMon, 22 Jul 2024 12:07:35 GMTWelcome back to my series of key takeaways from the DSC Adria Data Science conference. Part one talked AI strategy and trends, part two dived into building safer, better LLM systems, and today, we have they conference keynote by Luis Seco, Director of the Risk Lab at the University of Toronto. I was both fascinated and inspired by Seco’s vision for how …

Read more

]]><![CDATA[To hack a Large Language Model: Speak its style, but not its language.]]>https://katherine-munro.com/p/how-to-hack-an-llm-and-building-better-ragshttps://katherine-munro.com/p/how-to-hack-an-llm-and-building-better-ragsFri, 28 Jun 2024 06:45:45 GMT
A conference speaker presenting a slide titled "How to secure AI?". The slide is split into four sections and corresponding points: Model: Define specific business problem AI solves; Structure software supply chain; protect against unauthorized access, modification and disclosure. Infrastructure: Platform level controls to ensure security across organisations; Identity and access management; Collect and retain logs. Data: Include AI in Data Governance and management processes; Encrypt stored data and data in transit; Address business risk to data around the AI system. Application: Input and output filtering and validation; Run red team tests; Enforce content safety policies; Create an incident response plan
Four pillars to focus on when securing AI systems, presented by Kristian Kamber, splx AI.

Today’s post continues my key takeaways from DSC Adria in Zagreb, Croatia. While last time I focussed on high-level data and AI topics like strategy, trends and future predictions in the (Generative) AI space, this time I’m getting into the details: look out for pr…

Read more

]]><![CDATA[If you're optimizing for data, you're already losing: Lessons from a Data & AI Conference]]>https://katherine-munro.com/p/if-youre-optimizing-for-data-youre-too-latehttps://katherine-munro.com/p/if-youre-optimizing-for-data-youre-too-lateFri, 14 Jun 2024 06:12:22 GMT
A newspaper clipping with a photo of teachers holding pickets denouncing calculator use.
Maths teachers in 1986, protesting against calculator use in schools. One of my favourite tidbits from the DSC Adria data science conference. Source: The Daily Item.

Recently I had the pleasure to attend and speak at DSC Adria in Zagreb, Croatia. I took a tonne of notes, and over the next couple of posts I’m going to share the most interesting and useful…

Read more

]]><![CDATA[Editor's Digest: The Modern Data Stack]]>https://katherine-munro.com/p/editors-digest-the-modern-data-stackhttps://katherine-munro.com/p/editors-digest-the-modern-data-stackFri, 17 May 2024 10:29:13 GMT
"Afraid to ask Andy" meme, where Andy the officeworker looks distressed and the caption reads "I don't know what the modern data stack is, and at this point I'm too afraid to ask."
In today’s post: A collection of useful resources on the Modern Data Stack: why it's used, how it's composed, and whether to build versus buy.

Well hello again, everyone! I’ve been away finishing some personal projects, but I’m finally back, and today’s post is a direct result of one of those endeavours: I’ve been editing the third edition of The Handboo…

Read more

]]><![CDATA[Seven practical resources for building effective LLM applications]]>https://katherine-munro.com/p/practical-resources-for-building-llm-systemshttps://katherine-munro.com/p/practical-resources-for-building-llm-systemsFri, 23 Feb 2024 13:46:50 GMTHi and welcome to another resources roundup, and I’m starting this with a quick announcement: I’m thrilled to say that after many years working as a data scientist, I’m going back to my computational linguistics roots, and joining Switzerland’s largest telecommunications provider to help them build out their conversational AI systems. So if my content s…

Read more

]]><![CDATA[Yes, you still need old-school NLP skills in "the age of ChatGPT"]]>https://katherine-munro.com/p/despite-gpt-you-still-need-nlphttps://katherine-munro.com/p/despite-gpt-you-still-need-nlpFri, 02 Feb 2024 06:59:19 GMT
A meme with two badly drawn cartoon faces. The first one is crying and has the caption: "I just need to do some tokenization, case normalisation, stopword removal, lemmatising and POS tagging, and then I can extract features to train my entity recognizer." The second figure looks happier, and has a caption "hahaha ChatGPT go brrrr"
How it feels being an “old-school” computational linguist in 2024. Source: me and my pain.

In which I try not to sound like an old has-been (and end up doing exactly that)

Back when I started studying a masters of Computational Linguistics, no-one I knew of had even the faintest idea what Natural Language Processing (NLP) was. Not even me.1 But I quickly r…

Read more

]]><![CDATA[Resources Roundup: She didn't just say what I think she did, did she?]]>https://katherine-munro.com/p/spicy-takes-on-data-science-analytics-and-mlhttps://katherine-munro.com/p/spicy-takes-on-data-science-analytics-and-mlSun, 12 Nov 2023 17:49:22 GMTWelcome back to another resources roundup, and this time, I’m feeling a little sassy. It’s been too long since anyone told me I had the sexiest job of the century, and high time to bring that sexy back, courtesy of a little controversy. So this post is all about spicy takes on data science 🌶️🌶️🌶️.

Read more

]]><![CDATA[Resources Roundup: So... maths is cool now? A dive into data literacy.]]>https://katherine-munro.com/p/data-literacy-or-why-maths-is-cool-nowhttps://katherine-munro.com/p/data-literacy-or-why-maths-is-cool-nowSun, 20 Aug 2023 10:25:19 GMTMy last two resources roundups were packed with interesting, useful tidbits about data visualisation (see Making graphs that don’t suck) and storytelling with data (see Using data to make people care, listen, and act). Closely related to this, and the theme of today’s post, is data literacy. I’m including some of my recent favourite resources on this to…

Read more

]]><![CDATA[Resources Roundup: Using data to make people care, listen, and act]]>https://katherine-munro.com/p/resources-for-crafting-effective-data-storieshttps://katherine-munro.com/p/resources-for-crafting-effective-data-storiesSun, 23 Jul 2023 05:54:49 GMTIt is a truth universally acknowledged, that a single man data scientist in possession of a good fortune dataset, must be in want of a wife jupyter notebook full of bad matplotlib plots and a captive audience to dump them on.

If you see what I did there, congratulations on being as geeky as me. And even if you didn’t, I hope you’re at least curious. With…

Read more

]]><![CDATA[Never tell a developer they're doing great: Lessons from a Data & AI Leadership Conference]]>https://katherine-munro.com/p/never-tell-a-developer-theyre-doing-greathttps://katherine-munro.com/p/never-tell-a-developer-theyre-doing-greatTue, 27 Jun 2023 06:10:37 GMTI’m a nerd. In German you’d call me ‘Streber’. When I go to a conference, I’m that girl in the front row tapping away furiously on my laptop, afraid to miss a thing.

Afterwards I… I … well, I never look at my notes again. Until now. Today, in a break from my usual posting style, I want to share my key takeaways from the Data & AI Leadership Day, which m…

Read more

]]><![CDATA[Resources Roundup: Making graphs that don’t suck. ]]>https://katherine-munro.com/p/making-graphs-that-dont-suckhttps://katherine-munro.com/p/making-graphs-that-dont-suckFri, 05 May 2023 15:55:16 GMTSo, it’s been another week month (I promise I’m working on my publishing frequency!) and I’m back with another resources roundup: this time featuring all the best bits of advice and instruction I’ve seen lately on creating better data visualisations. Enjoy!

Read more

]]><![CDATA[Resources Roundup: Algorithmic bias, testing your preconceptions, and how to help build gender equality]]>https://katherine-munro.com/p/algorithmic-bias-testing-your-preconceptionshttps://katherine-munro.com/p/algorithmic-bias-testing-your-preconceptionsWed, 08 Mar 2023 07:15:33 GMTAbout once a month, I compile a handful of the most interesting and useful things I’ve shared via Twitter on a specific theme into a short article about that topic. March is Women’s History Month, so this twitter Resources Roundup focuses on the relationship between technology — particularly that powered by AI and automated algorithms—and women. Spoiler…

Read more

]]><![CDATA[I don't want to talk about ChatGPT either, but here we are.]]>https://katherine-munro.com/p/i-dont-want-to-talk-about-chatgpthttps://katherine-munro.com/p/i-dont-want-to-talk-about-chatgptThu, 23 Feb 2023 11:48:22 GMTI’m sick of hearing about ChatGPT. And you know what’s sad about that? I’m a data scientist specialised in Natural Language Processing (well, computational linguistics, but that’s the non-sexy name that nobody uses). ChatGPT is a feat of technology in my very own, beloved domain, that should have me filled with joy and wonder: not because it’s perfect, …

Read more

]]><![CDATA[Resources Roundup: How to get your Machine Learning project to production]]>https://katherine-munro.com/p/six-killer-resources-to-help-youhttps://katherine-munro.com/p/six-killer-resources-to-help-youThu, 16 Feb 2023 12:53:45 GMTThis resources roundup from my Twitter feed covers the stages of bringing a Machine Learning project from idea into production. There are links in here for data scientists, ML Ops experts, and anyone working in data driven companies. I've even done a little deep dive in the model development section. Let me know what you think!

Before you start, know wha…

Read more

]]>