I've spent the last few years working with data platforms, and honestly, something's been bothering me.
Every time I looked at our cloud bills, or watched brilliant engineers spend hours doing repetitive work that software should handle, or saw teams locked into pricing models that punished them for actually using the tools they paid for... I kept thinking: "There's got to be a better way."
The Problem That Wouldn't Go Away
Here's what I noticed: data engineers are brilliant at solving complex problems. They understand data flows, they build intricate pipelines, they handle edge cases most people never think about. But they're spending 40-50% of their time on stuff that's just... tedious.
Schema inference. PII detection. Data quality checks. Writing the same transformations over and over. It's important work, don't get me wrong. But it's also the kind of work that software could help with, if we built it right.
The insight that stuck with me: If we could automate even half of that repetitive work, we wouldn't just save time. We'd fundamentally change how teams budget for data engineering.
The Pricing Problem
And then there's the pricing model most data platforms use.
Pure consumption-based pricing creates this weird dynamic where teams are afraid to actually use their tools. Every quality check, every analysis run, every scan adds to the bill. There's no predictability. You can't budget for next quarter because you don't know what your usage will be.
But here's the thing: some level of usage-based pricing makes sense. Running 10 agent executions isn't the same as running 10,000. The compute cost is real.
The problem isn't usage-based pricing itself. It's the lack of transparency, the surprise bills, the feeling that you're being gouged every time you use the platform.
What If We Could Just... Fix It?
That's where Solidafy came from. Not from some grand vision or business school playbook, but from a simple question: what if we built data tools with honest, transparent pricing?
Here's what we wanted:
- Predictable base costs — Know exactly what you'll pay each month. Generous included limits so most teams stay within their base tier. Fair, transparent overages if you need them.
- Actual automation — Not just "click here instead of there" automation, but real intelligence that handles the repetitive stuff so engineers can focus on the interesting problems.
- Zero-access architecture — Your data stays in your cloud. All processing happens in YOUR infrastructure. We orchestrate the tools, you own the data. No one should have to send their sensitive data through someone else's platform.
- Tools that enhance the experience — Data engineering shouldn't feel like a chore. The tools should make the work more enjoyable, not add to the cognitive load.
The Agentic Approach
One thing I got a bit excited about (probably too excited, if I'm honest) was this idea of "agentic" platforms. Not in the buzzwordy sense, but in the practical sense of: what if the platform could actually understand what you're trying to do and help you do it?
Like, you point it at a CSV file, and instead of manually defining every column type, handling nulls, figuring out PII, writing quality checks... the platform just does it. And then presents it to you for confirmation. "Hey, I think this is what you want. Am I right?"
That kind of assistance, where software is genuinely helping instead of just providing a fancy UI for manual work, that's what I wanted to build.
"The best tools don't make you think about the tool. They let you think about the work."
Starting Small, Thinking Big
I started with PII scanning because it's something every team needs, and it's one of those things that's simultaneously critical and tedious. Get it wrong, and you've got compliance problems. But doing it right means reviewing thousands of fields, many of which are obvious (yeah, that email column probably contains emails).
Perfect candidate for automation.
From there, the roadmap kind of wrote itself: data quality checks, compliance reports, data warehouse automation, CDC replication, lineage tracking. All the pieces that make up a modern data platform, but built with this philosophy of "automate the tedious stuff, empower the engineers."
What Keeps Us Going
Honestly? It's the idea that we can change how teams think about data platform costs.
Instead of that anxiety-inducing conversation every quarter ("How much will this cost us?" or "Can we afford to scan this dataset?"), teams know exactly what they're paying. Predictable base cost. Generous limits. Transparent overages. No surprises.
And engineers can spend more time on the interesting problems. The ones that actually need human intelligence and creativity. Not schema inference. Not PII detection. The real challenges.
It's Early Days
I won't pretend Solidafy is perfect or finished. It's not. We've got the PII scanner working, and I'm really proud of how it performs. But there's so much more to build.
What I can say is this: every feature we add, every module we ship, I'm asking the same questions:
- Does this actually help engineers?
- Does this reduce tedious work?
- Does this make the experience better?
- Are we staying true to predictable, fair pricing?
If the answer to any of those is "no," we rethink it.
Why Share This?
Partly because I think it's important to be transparent about why we're building what we're building. Too many products are just "solutions looking for problems," and I want to be clear: this isn't that.
And partly because I'm hoping to connect with other people who feel the same way—that data engineering could be better, that pricing models could be fairer, that we could build tools that genuinely help instead of just extracting value.
If that resonates with you, I'd love to hear from you. Seriously. Book a call, tell me I'm wrong about something, share what frustrates you about your current tools. That kind of feedback makes everything better.
Thanks for reading this far. If you've made it here, you probably care about this stuff as much as I do.
Here's to building better tools, fairer pricing, and making data engineering a bit more enjoyable.
— Kuben
