At a recent dbt meetup in Amsterdam, I shared some statistics that made the room go quiet: 85% of data projects fail. Most dashboards go unused. The daily reality for data teams involves weeks of work on dashboards that get checked once a month, if at all. Data models that took months to build get bypassed by marketing teams who export to Excel instead.
This isn’t just anecdotal frustration—it’s an industry-wide crisis. VentureBeat reports that 87% of data science projects never make it to production. The S&P Global found that 42% of businesses scrapped most AI initiatives in 2024, up from 17% in 2023. For those of us who’ve dedicated our careers to data, these numbers should be a wake-up call.
At Tasman Analytics, we’ve worked with over 70 clients in the past eight years—from scrappy startups to established enterprises like PensionBee and Ecosia. We’ve seen what works and what doesn’t. And increasingly, we’re convinced that the solution isn’t technical—it’s philosophical. Analytics engineers need to start thinking like product managers.
The Root Cause: You’re Running a Product Team Without Product Management
Here’s the insight that transformed how we approach client engagements: “You’re not just running a technical team, you’re not just building a technical product, you’re actually running a product team… without Product Management.”
Think about what analytics engineers actually build:
- You think: Models, Dashboards, SQL Queries
- Reality: Data Products, User Interfaces, Product Decisions
This perception gap is killing data initiatives. 61% of data teams are still viewed primarily as “report generators” rather than strategic advisors. When stakeholders see you as a service desk rather than a product team, they treat your outputs accordingly—as commodities to be consumed and discarded, not products to be adopted and evolved.
Three Principles for Product-Oriented Analytics
1. Manufacturing Mindset Beats Collection Mindset
The traditional approach to data has been collection-focused: “Let’s capture this data and see what we can do with it.” This worked when storage was the constraint and data volumes were manageable. Today, it’s a recipe for failure.
As Zhamak Dehghani articulated in her groundbreaking 2019 article on Data Mesh, we need to move from treating data as an oil to be extracted to treating it as a product to be manufactured. This isn’t just semantic—it’s a fundamental shift in how we approach data work.
Manufacturing principles applied to analytics:
- Quality Control: Instead of fixing issues downstream, build quality into every stage. Use tools like Avo for event tracking governance (yes, throw away those Google Sheets tracking plans). Implement schema validation, automated anomaly detection, and continuous monitoring.
- Standard Operating Procedures: Create consistent, repeatable processes. Standardised naming conventions, reusable transformation logic, and explicit business rules aren’t nice-to-haves—they’re essential for scaling.
- Efficient Production: Design for performance from the start. Domain-modelled tables, appropriate freshness SLAs, and proper indexing aren’t premature optimisation—they’re product design.
2. Trust Is Your Most Fragile Product
Trust in data has two components, and most teams only focus on one:
Technical Trust (what we typically focus on):
- Testing frameworks
- Data quality monitoring
- Pipeline reliability
Business Trust (where we actually lose):
- Understanding of business logic
- Relevance to team needs
- Consistent accuracy
As I learnt the hard way when incorrect lifetime value calculations delayed a due diligence process by four weeks: “If your FP&A team doesn’t think that you really understand the business very well, you’ve lost it at that point.”
The solution? Separation of concerns in your data stack. We advocate for a clear three-layer architecture:
- Source & Staging layer (handles technical complexity)
- Domain layer (the “Static Target” containing business logic)
- Presentation layer (optimised for consumption)
This isn’t just good engineering—it’s good product design. When business logic lives in a dedicated domain layer, changes become traceable, testable, and communicable. As Airbnb discovered when building their Minerva platform, having a clear separation between technical and business concerns was critical to achieving trust across 12,000 metrics used company-wide.
3. Data Product Thinking Scales Best
Here’s the uncomfortable truth about our profession’s future: “SQL writing and writing data models quite quickly, I think a large part of that is gonna be commoditised.”
The best analytics engineers don’t just write better SQL—they ask better questions first. In the GenAI-accelerated era, technical skills alone won’t differentiate you. What will? Product thinking.
ThoughtWorks’ Technology Radar moved Data Mesh from “Trial” to “Adopt” status by 2024, signalling that treating data as a product has become a mainstream best practice. This isn’t just about following trends—it’s about survival in a rapidly evolving field.
The Product Brief: Your New Secret Weapon
At Tasman, we’ve operationalised product thinking through a simple but powerful tool: the Product Brief. Before writing a single line of SQL, we document:
Step 1: Define Intent & Users
Not just “build a dashboard for marketing performance,” but:
- Intent: Define specific business outcomes
- Primary Users: Name them, understand their workflows
- Secondary Users: Consider the full ecosystem
Step 2: Map Key Decisions
What specific decisions will this data inform?
- Marketing: “Which accounts to target for upsell campaigns?”
- Sales: “What’s the likelihood of expansion success?”
- Customer Success: “When to schedule proactive check-ins?”
Step 3: Define Success Metrics
Not technical metrics, but business outcomes:
- “90% faster implementation of data-driven campaigns”
- “Sales team uses score in 80%+ of account reviews”
- Quality requirements with specific SLAs
Step 4: Establish Ownership & Dependencies
Clear ownership model:
- Data Definition: Analytics Engineering Team
- Business Logic Validation: Domain Expert
- End-user Training: Sales Operations
This might seem like overhead, but consider the alternative: 85% failure rate. A few hours of upfront thinking beats weeks of rework every time.
Implementation Reality: This Won’t Be Easy
Let’s be honest about the challenges. 97% of data engineers report feeling burnt out. Teams are drowning in tickets. The last thing anyone wants is “one more process.”
But here’s what we’ve learnt from successful implementations:
Start Small: Pick one high-visibility project. Apply product thinking. Demonstrate value. The success will create demand for more.
Automate Documentation: Use AI to help create product briefs. The format is structured enough that LLMs can assist effectively.
Make It Visible: When stakeholders see the thinking behind your work, perception shifts from “report generators” to strategic partners.
Measure Everything: Track not just technical metrics but business outcomes. CDOs with business-facing KPIs are 1.7x more likely to demonstrate clear ROI.
The Path Forward: Choose Your Destiny
The analytics engineering field stands at a crossroads. As I told the Amsterdam meetup attendees: you need to choose between becoming either:
- Exceptional engineers who go deep on technical excellence
- Product thinkers who bridge business and technology
The middle ground—where most of us live today, building dashboards and writing SQL—that’s what gets commoditised first.
Spotify’s data platform transformation shows the way forward: they didn’t just migrate technology, they transformed how 300+ teams think about data as a product, achieving 85% organic adoption before mandating the change.
Call to Action: Start Tomorrow
You don’t need to transform overnight. Start with these three steps:
- Pick your next request: Instead of diving into SQL, write a one-page product brief
- Talk to users: Not about requirements, but about decisions they need to make
- Measure adoption: Not just accuracy, but actual usage and business impact
The data industry is evolving rapidly. With Gartner positioning data mesh and product thinking as key trends for 2024-2025, the organisations that adapt will thrive. Those that don’t will join the 85% failure statistic.
At Tasman, we’ve bet our business on this approach. Our clients are 8x more likely to exit successfully because we don’t just build data pipelines—we build data products. We don’t just deliver dashboards—we deliver decision-making capabilities.
The question isn’t whether analytics engineering needs to evolve—it’s whether you’ll lead that evolution or be left behind by it.
References and Further Reading
- Dehghani, Z. (2019). “How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh.” Martin Fowler’s Blog.
- Dehghani, Z. (2022). “Data Mesh: Delivering Data-Driven Value at Scale.” O’Reilly Media.
- ThoughtWorks Technology Radar (2024). “Data Product Thinking” – Adopt status.
- VentureBeat (2019). “Why do 87% of data science projects never make it into production?”
- S&P Global (2024). “AI Project Failure Rates Report.”
- Gartner (2023). “Future of Data Architecture: Data Mesh and Data Fabric.”
- Robert Chang (2023). “Scaling Metrics @ Airbnb” – dbt Roundup Interview.
- Spotify Engineering Blog (2023). “How Spotify Upgraded Its Data Platform.”
- Data Science PM (2024). “Why Big Data Science & Data Analytics Projects Fail.”
- Monte Carlo Data (2024). “Measuring The Impact Of Your Data Platform With These Metrics.”