I stumbled upon a SaaS app called Equals, which is a browser-based spreadsheet. My first thought as a data analyst was that this was pretty slick. My second thought as a head of a business intelligence team was that it's too bad it's not Excel. In an earlier post, I was talking about how users don't need special software to create another line chart. They want to see the data first. Excel is the go-to natural tool for users to do that. The problem Excel has is that it doesn't scale to large dat...

It's been a long week at work. My team and I have been struggling to figure out a solution for using Kerberos authentication with our legacy reporting software. It's been frustrating. Nothing I try seems to work. This kind of thing happens from time to time. I run into a problem that needs solving, but I just can't seem to crack it. A part of me has a hard time putting it down to focus on other tasks. It's not a matter of pride. It's not that I need to be the one to solve it, it just needs to ...

MS Build was yesterday, and as has become the custom, Microsoft unloaded a huge wave of announcements including the latest release to Power BI and new capabilities in Fabric. I'm going to need some extra coffee today because I was up late trying to digest all this info (which is not recommended). Here's my summary of interesting highlights: First in Power BI Matrix visuals now have new format options to make them look more like Excel PivotTables. This seems to be the most popular highlight on...

Recently, I posited that people don't need more tools to create charts. If I come off as cynical of BI tools, that's not my intention. I enjoy working in BI. I take a lot of satisfaction building solutions that help drive business improvement, and these BI tools help me do that. When I look at a line chart I made, I have the advantage of knowing all about the data behind it. My worldview of the data is not limited to a few spreadsheets handed to me. I often conduct my exploratory work from a da...

BI tools like Tableau, Qlik, and Power BI, started off targeting business users, not technical folks. They wooed them with fancy charts and visuals and promised to show insights from their data. But this selling point is only the last mile of BI. This skips past the hardest problem about working with data: accessing and modeling. Business users never really needed another way to make a line chart. These BI tools limit users to making line and pie charts only to export the data later. This mode...

Most data teams operate as a service-oriented function. It's just the natural consequence of the role. Business users need data, data analysts are hired to retrieve that data, eventually reports and dashboards abound. As the tools improve, it gets easier and faster to create them. But every user wants something different. Now that they are cheap to make, the data team is expected to meet the demand, and reports and dashboards proliferate. This leads us to Jevon's Paradox. Easy to make, but too...

These are my three guiding rules for writing good SQL Readable Maintainable Performant If nothing else, first make it readable. Well-structured SQL is faster to review and easier to debug. Consistent style, use indentation, give identifiers meaningful names, and add comments. Next focus on the maintainability of your query. Good news is that if your query is very readable, then it is already easier to maintain. Find ways to avoid repetitive logic or repeated placement of filters or paramete...

For the past couple of weeks, I've been experimenting with SQLMesh, a new open source data modeling/transformation tool. SQLMesh is intentionally positioned to compete with dbt, which has become the go-to option for companies adopting the ELT paradigm of managing data transformations in their cloud data warehouse. My team and I started adopting dbt a couple years ago. It's been a slow transition so far. For a variety of reasons, our primary data warehouse still uses traditional ETL. For now, w...

There are a lot of different Python Integrated Development Environments (IDE). This article will discuss using VS Code, a free open-source offering from Microsoft vs Spyder, another open-source offering. Spyder is provided out of the box with the popular Anaconda Python distribution. Anaconda generally targets those working in data science because it provides many common data science-based packages pre-installed. Spyder gives an interactive experience, showing the user both a .py file for codin...

Python scripts can quickly become complex beasts, especially when nested if statements and long routines come into play. But what if there was a way to tame this complexity, making your code more readable and maintainable? I'm going to describe a way to structure your Python script so that it is easy and clear to follow regardless of complexity. This pattern may not be suitable for every scenario, but for most processes that are contained to one or two scripts I find it very useful. The guiding...