Share
It was October 21, 2022, and we’d just come out of the Coalesce Conference in New Orleans. For five days, data and IT professionals from all over the world met to discuss the future of dbt (or Data Build Tool, which simplifies the running of SQL against databases and data warehouses). While we found the technical side of dbt beautiful, many other attendants focused on business problems and the adoption of the tool. That inspired us to think about new projects that could bring dbt to our customers. The only problem was some of these customers still used Excel, not databases.
It was still warm enough to go into the swimming pool of our Airbnb, and while we stared at the sky, one of us said,
“Hey, can you imagine making a dbt adapter for Excel?”
We all laughed at the joke. If you didn’t, don’t worry, it just means you should hang out with us more often.
But as the laughter faded, someone else said,
“Should we actually make it?”
April Fool's Day: the dbt-Excel adapter release
At Xebia, we take our jokes seriously, so we decided to create a dbt adapter for Excel as a big prank and release it on April Fools’. But we’re still passionate about data tools, so we wanted to make sure the adapter worked. And after all, many companies still worked with Excel, so maybe it was worth the shot.
Now, this meant we needed a database to connect to as an intermediate step. We had plenty of suggestions for this at Xebia. Eight of our consultants are GitHub contributors for dbt and know their way around different adapters. In the end, we settled for DuckDB.
A dbt adapter for DuckDB already exists, but the issue is that DuckDB does not read Excel files. This is where the engineering work came in: we needed to transform Excel files into .parquet ones that DuckDB could read. We put our heads together and managed to create the core of our program.
But we didn’t want to stop there. We wanted to ensure users could run functions they knew from Excel: VLOOKUP, AVERAGEIF, and the rest of their favourites. That meant we had to create a few macros to implement these functions. And then, of course, we wanted to save the final output to an Excel file.
It took us a few days and many litres of coffee to get there, but we finished the adapter before April Fools’. The engineering part of the project was done; the marketing part remained. So, we turned ourselves into actors and made this promotional video that we published on our social media.
Many people laughed with it; others, again, they just need to spend some more time with us. And while we enjoyed the fleeting stardom, we felt our inner nerd hadn’t been completely seen. Many people in the audience laughed, but few realised the tool was real.
The adapter, which you can download here, allows you to connect with dbt from your computer by using Excel without the need for a database. In other words, you get familiar with dbt without resorting to a virtual machine that you might leave open for too long and without a database that you might not have at hand. The dbt-Excel adapter could be part of what’s called the Minimum Viable Data Stack, a set of data tools that allows your company to start working with data without a large initial investment.
If your company is still using Excel and starting its data transformation, maybe it’s good to use this dbt-Excel adapter during the prototyping phase. It will help you become familiar with some of dbt’s core benefits like version control, testing or mixing intermediate to advanced SQL computation while you view the final result in Excel.
Before revamping your whole data landscape, we always recommend trying out different tools and pilot projects. After that, if you feel confident about the results, you can incrementally invest in other professional services that strengthen your data pipelines. You can always give us a call if you want to discuss this transition.
For us, building the adapter was a great ride: we got our hands dirty, we played with the tool we love—dbt— and we learned a lot along the way. We did it as a joke and as a serious project. We did it because we love our job and love to laugh with it from time to time. But mostly, we did it because when we were at that pool in New Orleans, it just seemed like a pretty cool idea.