Beyond the Dashboard | Principle 1: Avoid the Data Delusion
In the Data Delusion, we focus on metrics while losing sight of what truly matters. We celebrate tiny, irrelevant wins, creating an illusion of progress. Data without human judgment is just noise – it's time to use it to sharpen our questions, not just track our speed.
TL;DR (for bullet-point lovers)
- Avoid mistaking busywork for meaningful progress by questioning if data is creating an illusion of clarity.
- Data without human judgment is just noise; it should be an input for strategy, not a substitute for it.
- Use AI to sharpen your questions and find real problems, not to automate trivial tasks that accelerate waste.
- Focus on whether experiments fundamentally change your direction, not on small, strategically irrelevant wins.
- Shift your team from celebrating data to making decisions by asking what you will do differently with the information.
The Data Delusion
In the introduction to this series, I made a promise: to move beyond the beautiful dashboards that feed our eyes but starve our minds. We agreed that our modern control rooms, gleaming with data, often create an illusion of clarity while judgment quietly withers.
So, where do we begin to dismantle this illusion? We start with the most seductive and foundational lie in the entire system: the Data Delusion.
Let's get right to it with two questions:
- How many data-based experiments did your teams run last quarter?
- And how many of those experiments fundamentally changed your product direction or killed an initiative?
If you ran none, that's a lack of learning – a terminal diagnosis on its own.
If you ran dozens of experiments that change nothing, that isn't "test and learn"; it's successfully testing that you're learning nothing.
That's the Data Delusion in action.
We built a culture where “data-driven” became a mantra. The intent was noble: decisions based on facts, not hunches. But in practice, it often devolved into performance theater.
You’ve seen it: A/B tests that move metrics by 0.2%. Sometimes that matters. More often, it’s optimization without a destination.
Statistically significant.
Strategically irrelevant.
This is the very heart of the Illusion of Progress, a systemic trap where we confuse motion with meaning. The dashboards are green, velocity looks great, and teams are shipping faster than ever. But zoom out, and you realize the team is on a hamster wheel with great velocity but no direction. We’re celebrating hitting deadlines, but the outcomes are unclear. We’ve become so focused on performance questions ("Did we ship it?") that we've forgotten the impact questions ("Did it work?").
When everything is measurable, teams can debate endlessly instead of deciding. Data becomes a shield. Or worse, it fuels Prioritization Theater, where scorecards are reverse-engineered to justify what’s already been decided. In this drama, urgency and escalations win, not logic. Judgment, the uniquely human skill of turning information into strategy, erodes.
Data without judgment is noise.
How AI Turns Data Delusion into a Flood of Noise
AI doesn’t fix a judgment gap; it exposes it at scale.
If your team is already drowning in data, AI is a firehose. It can generate more charts, run more analyses, and find more statistically significant correlations between irrelevant variables, faster than ever before.
Mindlessly applied, AI becomes an engine for accelerating waste. It helps you:
- Generate more hollow insights: It can summarize user feedback, but if you ask the wrong questions, you just get a well-structured summary of noise.
- Automate trivial optimizations: It can suggest 100 variations of a button color, burying the fact that the feature itself is unwanted. It’s like hiring a Michelin chef to rearrange sprinkles on your cupcake.
- Create a false sense of analytical rigor: Teams can present complex, AI-generated models that look impressive but lack any connection to a coherent strategy.
The real danger is that AI is exceptionally good at making motion look like progress. It doesn’t just help you run in circles; it gives you a detailed printout of your lap times.
However, when used with intent, AI becomes a powerful tool for augmenting judgment, not replacing it. Instead of asking it for the answer, use it to sharpen the questions. A leader can use AI to:
- Surface the real user problem: Use it to cluster qualitative themes from hundreds of interview transcripts or support tickets to find patterns you’d otherwise miss.
- Accelerate time-to-signal: Instead of manual analysis, AI can extract objections from sales calls or summarize open-text survey feedback in minutes, freeing the team to focus on what the signals actually mean.
- Simulate responses and test assumptions: Use AI to generate prompts for assumption testing or simulate user reactions to new copy or flows, providing a quick, low-cost way to de-risk ideas before a single line of code is written.
Great product leaders act like captains. Data provides charts and forecasts; it doesn’t plot the course. The captain reads the charts, interprets the weather, and steers toward a destination. AI gives you better charts and faster weather alerts, but the act of steering and exercising judgment, remains fundamentally human.
• Are our metrics helping us decide, or avoid deciding?
• Are we running tests to learn and inform strategy, or to validate what we’re already building?
• How does this 0.2% lift connect to our strategic goals?
• What behavior are we hoping to change with this test, and how will we know if we succeeded?
If the answers aren't crisp, there’s a judgment gap.
The Strategic Price of Delusion
So what’s the real cost? The Data Delusion isn't just bad reporting; it’s a systemic failure that quietly grinds your strategy to a halt. When you confuse motion with meaning, you don’t just get inefficient you get lost.
Here are the strategic implications when delusion takes over:
- You enter strategic paralysis. Your organization gets stuck on a hamster wheel of activity. While dashboards are green and velocity is high, nothing of strategic value changes because data has become a shield to avoid making hard decisions. Judgment, the very engine of strategy, erodes.
- You burn resources optimizing trivialities. Your teams are trained to hunt for small, safe wins (the 0.2% lifts that are statistically significant but strategically irrelevant). They learn to avoid the big, ambiguous problems where real value is created.
- AI becomes an engine for accelerating waste. When applied mindlessly to a culture of delusion, AI doesn't find answers; it just creates a firehose of noise. It automates pointless optimizations and generates hollow insights, getting you to the wrong destination with a false sense of analytical rigor.
- Your ability to learn dies. In a healthy culture, discovery kills bad ideas. In the Data Delusion, every experiment is a "win" or "inconclusive," which means you're just successfully testing that you're learning nothing of consequence.
Ultimately, the delusion substitutes the hard work of strategic thinking with the easy work of tracking motion, leaving you perfectly measured but fundamentally lost.
Final Thoughts
The Data Delusion is tempting because it feels safe. It offers the comfort of numbers and the appearance of rigor. But it's a trap that substitutes the hard work of strategic thinking with the easy work of tracking motion. Perfect data will never fix a broken strategy, and AI will only get you to the wrong destination faster. The first step out of the delusion is admitting that judgment, not data, must lead.
What Comes After the Delusion?
Admitting there's a problem is the first step to recovery. Now that we've put a name to the Data Delusion, it's time for the cure: becoming truly data-informed. This is where we move from blindly following numbers to using them to sharpen our own strategic judgment.
In the next article, we'll break down Principle 2: Adopt a Data-Informed Approach. To get all eleven principles in this series delivered straight to your inbox, subscribe to the Thinking Lens newsletter. Don't let an algorithm decide if you see the next post.
PAQs – Potentially Asked Questions
My team celebrates every statistically significant A/B test win, even small ones. Are you saying this is wrong? How do we balance celebrating wins with strategic focus?
It's not wrong to celebrate, but the celebration should be proportional to the strategic impact. A 0.2% lift in conversion on your core checkout flow is a different universe from a 2% lift on a feature used by 0.1% of your user base. The delusion isn't the celebration; it's what you're celebrating. A culture that celebrates any statistical win without regard to its strategic relevance trains teams to hunt for small, safe optimizations. As a leader, your job is to be the killjoy who asks, "That's a great result. Now, how does this get us closer to our main objective this year?"
My team runs A/B tests all the time. Are you saying they're a waste of time?
No, but their value depends entirely on the question they are answering. If a test is designed to validate a core hypothesis or produce a meaningful learning that informs your next move, it's incredibly valuable. However, if you're spending weeks testing minor color changes on a button for a feature no one uses, you're likely optimizing a local maximum while ignoring a larger strategic problem. The question to ask is: "Will the result of this test change a major decision we need to make?". If the answer is no, you might be in the data delusion.
What are some clear signs that a team is suffering from "data delusion" versus doing healthy, rigorous experimentation?
The signs are often systemic and cultural. Look for these patterns:
- No failed initiatives: If every experiment is declared a "win" or "inconclusive," it’s a red flag. Healthy discovery kills bad ideas. Data delusion seeks to validate existing ones.
- Metrics are disconnected from outcomes: The team can tell you they increased a metric by 5%, but can't explain how that improved the customer experience or moved a strategic goal.
- Data is used as a weapon: In meetings, data is presented not to foster understanding, but to shut down debate or push a preconceived agenda. This is a classic symptom of "Prioritization Theater".
- The "Why" is missing: Team members can explain what they are building and how they are measuring it, but struggle to explain why it matters in the first place.
Can you give a specific example of how AI could amplify data delusion in a product team?
Imagine a team is working on a user onboarding flow with a high drop-off rate.
- A "data delusion" approach: They ask an AI to analyze user session data and "find correlations" with drop-off. The AI identifies dozens of minor correlations: users who hesitate for 3 seconds on a certain field are 1.2% more likely to drop off; users from a specific country have a 0.8% lower completion rate. The team spends the next quarter running micro-optimizations on these weak signals, proudly presenting AI-generated charts, but the overall drop-off rate barely budges because the core problem is that the product's value is unclear to new users.
- A "judgment-led" approach: The leader frames the problem differently: "Our onboarding fails to show value." They use AI to summarize open-ended feedback from users who dropped off. The AI clusters the feedback and reveals the top theme: "I didn't understand what this product does for me." Armed with this insight, the team redesigns the first step of the onboarding to clearly state the value proposition. This is using AI to augment judgment, not replace it.
As a leader, what is the first, smallest step I can take to start shifting my team away from data delusion?
Start by changing the questions you ask. The next time a team presents results, ask these two things:
- What was the riskiest assumption we were testing with this work?
- What decision will we make differently now that we have this information?
These questions gently shift the focus from celebrating data as an output to using data as a tool for learning and decision-making. It doesn't require a new process or framework; it begins to rebuild the habit of structured thinking and connects activity back to strategic intent.
This philosophy of "judgment" and "thinking systems" sounds great for a small, senior team. How can this possibly scale in a large organization where you need standardized processes to prevent chaos?
This is a common misconception, that process and judgment are opposites. In reality, a good process scales judgment. The issue in most large organizations isn't too much process, but the wrong kind of process. They have processes for reporting, not for reasoning.
Scaling a data-informed culture doesn't mean every one of a thousand employees gets to make gut-feel decisions. It means creating systems and frameworks that force structured thinking at every level. For example:
- A mandatory "hypothesis statement" on every new initiative proposal.
- A "feature kill rate" metric that celebrates shutting down things that don't deliver value, making it safe to admit failure.
- Layered dashboards that provide the board with outcomes and teams with diagnostic levers, preventing micromanagement.
These are scalable systems. They don't remove process; they install a process that prioritizes strategic thinking over performance theater. This creates alignment without creating chaos.
Each principle in this series builds upon the last to form a coherent system for better decision-making. Here is the full list of principles we are exploring:
Intro: Beyond the Dashboard Series
Principle 1: Avoid the Data Delusion
Principle 2: Adopt a Data-Informed Approach
Principle 3: Choose What to Measure
Principle 4: Use Frameworks as Filters, Not Blueprints
Principle 5: Focus on Adoption, Not Just Delivery
Principle 6: Know Your Tool Stack’s Boundaries
Principle 7: Build Layered Dashboards to Scale Thinking
Principle 8: Manage Multi-Product Portfolios Separately
Principle 9: Reconcile Metric Definitions Before Analysis
Principle 10: Build Thinking Systems, Not Reporting Systems
Principle 11: Turn AI into a Judgment Multiplier