Transitioning away from Excel #
Businesses often begin their data journey using Excel sheets until the complexity outgrows the tool’s capabilities. Eventually, someone within the organization, often an analyst or a business executive familiar with BI tools like PowerBI, realizes the need to transition to Business Intelligence (BI) for more sophisticated reporting.
In this pivotal moment, that individual is assigned the task of creating initial reports, marking the first steps of the company’s data-driven approach. However, this starting point comes with its own set of challenges, which we will explore further.
The challenges with a solo data practitioner #
If All You Have Is a Hammer, Everything Looks Like a Nail
This age-old quote still holds true, especially in the context of modern data analytics today.
Solo data practitioners, whether analysts or engineers, often face distinct challenges that can impact the effectiveness of their work.
1. Analysts’ Limitations: Analysts commonly rely solely on BI tools and SQL queries, which might lead to overlooking alternative solutions. The lack of diverse tools and perspectives can hinder problem-solving and limit the potential for innovation. Analysts often lack knowledge about the importance of proper data infrastructure, and the know-how on how to set it up is even scarcer, partly because data analytics is often perceived as a non-technical field, leading companies to prioritize hiring data scientists or analysts (or promoting somebody to be one..) without considering whether they possess the expertise to establish a strong foundation.
2. Isolation of Engineers: When only one engineer is involved, they might isolate themselves from the business and its analysts. Bridging this gap is essential for better collaboration and successful outcomes.
3. Single Data Skill Set Risk: Embarking on a data practice with only one specific skill set can neglect other crucial aspects, which is often the case initially. Successful analytics solutions require input from both the business and engineers, mirroring the importance of full-stack developers in software engineering.
If you embark on a data practice with only one specific skill set, you risk leaving other critical aspects behind. Analytics solutions that deliver value need input from the business and engineers alike. Like Full-stack developers and DevOps engineers in software development, in data analytics, we require people who can effectively blend business knowledge with technical expertise to create successful outcomes.
Navigating the People Puzzle in Data #
The solution to data-related challenges often appears to be assembling a fully-fledged team, including a product owner, data scientist, data engineer, analytics engineer, and BI analyst. However, hiring and training such a team remains an unrealistic and initially unnecessary option for many companies, especially in the early stages of business growth when data mastery is critical.
While ad hoc data handling is a common fallback, it brings its own set of problems, such as data quality issues, performance bottlenecks, and unusable dashboards due to multiple entities working with data independently.
Businesses often require a combination of these roles, making the hiring process challenging. Data scientists might lack expertise building ETL pipelines, while data engineers may struggle to create actionable BI dashboards.
A potential solution lies in adopting the concept of a “full-stack data engineer” (or full-stack data scientist), someone equipped with a diverse skill set to handle various data aspects effectively. While adding more terminology may seem confusing, this approach could be a pragmatic way to address data challenges, particularly for companies seeking a balanced data team without overwhelming complexity.