Towards Avoiding the Data Mess: Industry Insights from Data Mesh Implementations (2024)

4.1. Theoretical Understanding

Interviewees generally support the main principles of data mesh introduced by Dehghani ($\Theta-M,O$) and emphasize the importance of the combination of all four main principles to adopt a successful data mesh. However, interviewees assign the highest value to the principles of domain-oriented decentralized data ownership and data as a product when asked to favor one over the other.Furthermore, interviewees state the need for further guidelines that assist in the adoption and operation of a data mesh ($A,\Delta,E,\Theta,K$). $A$ states: “the concept is described as highly theoretical, and I believe it is a challenge to transfer this theory into practice for many organizations”, whereas $E$ describes the main principles as “too broad” to adopt a data mesh out of the box—highlighting the importance to provide practice-oriented guidelines for professionals going forward.

4.2. Motivational Factors

Prior to the adoption of the data mesh concept, interviewees face a variety of problems and thus consider multiple motivational factors.

MF1: Reduce bottlenecks ($A-E,H,\Theta,K-O$). The first factor combines the related concepts of bottlenecks, scalability, and speed and is consistent with the initial proposal of Dehghani (Dehghani, 2022).Bottlenecks refer to a lack of capacity in the central data management system or the team responsible for the data provision and result in a backlog of data requests from multiple domains. As a consequence, domains are unable to scale data-driven use cases at the preferred rate, e.g., $M$ states “in the food delivery space, they were not able to scale anymore because this big data warehouse in the middle was the bottleneck”.The concept of speed has two dimensions.It describes the time the central team needs to provide requested data and the time to market for new products and services; $B$ highlights the importance: “speed and delivery of shipping customer features, improving customer journeys, and scaling to more markets. We are hyper growth company. Speed matters when you’re on a hyper-growth”.

MF2: Leverage domain knowledge ($A-Z,I,K,\Xi,O$). In centrally governed organizations IT departments provide data, however, they lack domain knowledge to ensure the task-specific quality of the data. Interviewees emphasize the importance of domain knowledge to provide data of a higher quality. $\Gamma$ emphasizes, “the person didn’t have the domain knowledge […] therefore the results were bad”, and $E$ goes even further, stating “we need to somehow give that capability of dealing with data back to the people who actually understand what the hell the use cases are.”

MF3: Break down silos ($A,\Gamma,E-I,\Lambda,M,\Xi,O$). Accessibility is of great importance for cross-domain data-driven use cases. However, silos within organizations prevent other domains to access the data. For example, the marketing department (domain A) may need to request sales (domain B) data to track the success of a regional marketing campaign. However, if sales fail to deliver the data, the marketing department loses out on valuable insights. Providing a real-world example $O$ states: “We found out that nobody in the company has a full idea which kind of data exists and where the data is accessible.”

MF4: Establish data ownership ($I,\Lambda,N$). The fourth motivational factor highlights the significance of data ownership. It refers to the end-to-end responsibility of a data product in terms of quality and accessibility. This concept is comparable to traditional product ownership roles. $I$ describes the underlying problem in the following way: “we had the problem that when we pushed the data to the central data. Data ownership was lost”. Further, $I$ relates the concept of stronger ownership with better data quality: “I think it’s very simple. The main reason we did this was to improve data quality. And the bet was that stronger data ownership would lead to better data quality.”

MF5: Adopt modern architecture ($Z,K,\Lambda$). Managers and employees within an organization are aware of an upcoming trend in data architectures—they see how organizations throughout the industry adopt a data mesh and “don’t want to miss the boat” ($\Lambda$); “some companies just heard from this hype and this is why they look into it” ($K$). However, data mesh is not a universal approach. Hence this finding is very worrisome because it may push organizations or business units toward a data mesh adoption for the wrong reasons.

MF6: Reduce redundancies ($B,I,O$). Silos, poor communication, and lack of transparency can lead to redundant work in large organizations, including data-related areas like data preparation ($B,I$) or entire use cases ($O$). Various data consumers individually prepare data for their specific use cases but fail to share the results with the organization—“what we discovered in our research was that teams improve the data significantly, but don’t share it back with the company” ($B$). Similarly, $I$ points out: “we’re pushing all the way back to the data producer, who hopefully can solve the [data quality] problem instead of having to repeatedly deal with this issue further down the line”. As a consequence, redundant efforts across domains, divert resources away from value creation ($B$). Furthermore, interviewees from large, decentralized organizations complain that they spend significant resources on the provision of data for other departments—a task that data mesh aims to simplify ($I,O$).

After discussing motivational factors, it’s crucial to recognize the challenges individuals may encounter when working with the data mesh concept. We now analyze obstacles that can hinder the progress of data mesh adoption.

4.3. Challenges

Organizations face multiple challenges throughout the adoption of data mesh. We mainly focus on challenges that are unique and of high relevance to data mesh-related topics.

C1: Federated data governance ($A,\Delta-O$). We identify the shift from centralized toward federated data governance as the main challenge for professionals. Interviewees state that the federated approach introduces difficulties for activities and responsibilities previously managed centrally. Whereas they stress the importance of federated data governance to establish rules according to domain needs, interviewees highlight limitations regarding the automated execution; especially concerning security, regulatory, and privacy-related topics ($A,E-\Theta,N,O$).$O$ notes that employees within the domain are unaware of which data are protected and regulated.$N$ warns that managers may “end up with one foot in jail” for non-compliance with data protection regulations. Furthermore, interviewees state the lack of a central unit results in insufficient prioritization of use cases and projects across domains ($\Delta,Z,I$). $I$ argues that decentralized ownership and federated governance result in a lack of observability, which in turn complicates proper prioritization.

C2: Responsibility shift ($B,\Gamma,Z,\Theta,I-\Lambda,N$). The first main principle in reference (Dehghani, 2022) states that ownership shifts from a central authority towards the domains. However, decentralized ownership within the domains includes end-to-end responsibility for the accessibility and quality of the data products. This new responsibility creates a number of sub-challenges. First, the data product owners perceive the task of providing data for other domains as extra work ($\Gamma,\Theta,K$)—“they had to fight with all the other items in the backlog and they didn’t see any value in it” ($\Gamma$). In addition, domains frequently receive no direct compensation for the data provision efforts that benefit other domains ($\Theta,K,N$). Furthermore, domains’ business activity is usually not centered around the provision of data products—consequently, they deprioritize the task ($B,\Gamma,Z,I,\Lambda$).

C3: Metadata quality ($E,\Theta,I$). The third challenge, as reported by interviewees, relates to the data product model. Data products store various data types and also offer metadata, aiding users without domain knowledge in data interpretation (Gröger, 2021; Dehghani, 2022). However, interviewees point out a gap between the descriptions provider domains deliver and the information consumer domains need to correctly interpret the data. $E$ states that “data modeling is the most crucial part in most companies and there is no standard yet on how to do this” and subsequently elaborates that “if you solve data modeling […] you probably have the gold mine for data these days”. While the challenge of metadata quality is not unique to data mesh—“in former times, the problem already existed, but nobody really took care of it” ($\Theta$)—it is complicated by unclear expectations regarding the data products and their use cases ($I$).

C4: Comprehension ($\Delta,Z,\Theta,\Lambda,X$). The fourth challenge relates to a lack of comprehension of the data mesh concept. Interviewees report that employees within the organizations simply use new terms without an organizational or technical change, e.g., to call all data sets data products ($\Delta$) or categorize a highly centralized data storage as a data mesh ($\Lambda$). $\Lambda$ further elaborates he/she is under the impression “that the warehouse from yesterday is now called a mesh, just like a meeting today is called a sync”. Indicating a severe misinterpretation of the data mesh terminology among practitioners complicating efforts for data mesh initiatives to move forward.

Interviewees have pointed out additional challenges that, whereas not exclusive to data mesh, remain noteworthy. These challenges deserve attention because they can inform the development of specific implementation strategies for data mesh—neglecting them could potentially jeopardize the successful adoption of data mesh.

C5: Resource limitations ($A-\Delta,Z-\Theta,K-M,\Xi$). C5 describes the lack of financial, technical, and human resources—“It’s very resource intensive to build up a data mesh, and thus the data products. You need a lot of data engineers with ETL knowledge.” ($A$). $\Theta$ further elaborates: “everyone wants to get data-driven, but when it comes to the effort, companies are usually not eager to put more people or money into that”.

C6: Acceptance issues and resistance($\Gamma,Z,\Theta,\Lambda,M,\Xi,O$). C6 relates to acceptance and resistance within organizations, for instances, $O$ states: “Most of the time I was a somebody, let me say, a kinder-gardener. Yeah, just to tell the people, please change your mind. Please change your perspective. Please start looking at data from the perspective of the data”.

Both, C5 and C6 are well established in change management research, consequently, we do not discuss them any further but relate to the existing literature on this topic (By, 2007; Oakland and Tanner, 2007; Anastassiu and Santos, 2020).

4.4. Implementation Strategies

In the interviews, we identify implementation strategies that support organizations to overcome the challenges we formulated in the previous section. We highlight relationships between challenges and implementation strategies to allow organizations to address challenges according to their needs. The implementation strategies are the main contribution of this work as they provide preliminary guidelines for professionals and researchers. In the following, we present implementation strategies and provide detailed descriptions. Figure 5 on page 9 summarizes our central findings.

IS1: Cross-domain unit ($A,B,\Delta,Z-I,N$). Interviewees identify the transition towards federated data governance (C1) as the main challenge. To address this challenge, we argue that organizations should introduce a cross-domain steering unit responsible for strategic planning, use case prioritization ($A,\Delta,Z,H$), and the enforcement of specific governance rules—especially concerning security, regulatory, and privacy-related topics. This becomes necessary when full automation is not available yet. $\Delta$ further argues that human resource allocation from a central or a support team is essential in early phases, as it can help domains with strong use cases but little technical expertise. However, a cross-domain steering unit could also help to enforce high metadata standards (C3) ($\Delta,Z,H,\Theta,I$).Furthermore, interviewees state that it can be beneficial to provide key data products centrally ($B,\Delta,N$). $B$ argues that central ownership in early phases provides a role model domains can follow for the provision of domain-owned data products. Furthermore, it ensures smooth operations of data that are of strategic value ($B,N$). $B$ elaborates: “So one of the things that we do is we are taking 20 data sets that are of high, criticality. Datasets that we own centrally and turn into products. This way we act as a role model for other domains.”Nevertheless, we stress that a cross-domain steering unit can only complement and support the federated governance structure and may grow obsolete with the increasing maturity of a data mesh, e.g., the fully automated policy execution that is centrally provided by the platform.

IS2: Empower & observe ($B,\Delta,Z-K,M-O$). The second implementation strategy addresses the shift of responsibility (C2) and issues related to metadata quality (C3) and the transition towards federated data governance (C1). One of the key ideas of data mesh is to leverage domain knowledge. However, a central department needs to provide the technical infrastructure, support, templates, and guidelines to facilitate the provision of high-quality data products ($B,\Delta,Z-I,N$). This finding is in line with the third main principle of data mesh, the self-serve data platform highlighting the importance of empowering the domains. On multiple occasions during the interviews, interviewees stress the importance of observing the quality and usage of data products ($Z-K,M,\Xi,O$) pointing out two major benefits. First, usage data of data products can motivate data product owners within the domains because it visualizes the impact the data product has. This data can help domains highlight their importance within the organization and provide them with leverage to negotiate additional resources (C5). Furthermore, we suggest that organizations observe and automatically score the quality of data products, preferably by an independent unit. This score should include a metadata score, a data quality score, and availability metrics. Depending on the criticality of the data and data product quality standards, organizations should also consider a human-in-the-loop responsible for initial approval ($\Delta$).$O$ argues that a data product quality score creates an incentive for data product owners to ensure a high quality of the metadata (C3). $\Delta$ further emphasizes, “it doesn’t need a lot of escalation […] just by knowing it will be reported […] they handle it very differently now”. Therefore, we suggest that a central steering unit tracks and ranks key data products to nudge data product owners to provide high-quality data products.

IS3: Quick wins ($A-O$). Whereas the challenge of acceptance issues and push-back within an organization (C6) is well established, data mesh-specific implementation strategies can help to navigate organizational challenges.We synthesize findings from interviewees to formulate a fine-grained multi-step process that guides professionals through the data mesh exploration & bootstrap phase with respect to organizational challenges.First, data mesh initiatives should start “where the budget is” as financial resources are crucial for the success of the data mesh adoption ($Z$, C5). If central IT has the resources to kick-start the transformation towards a data mesh they should be responsible. If the organization is already highly decentralized, influential domains should lead the way. Either way, communication across multiple domains and stakeholder from central IT is crucial across the iterative transformation phase. In the next step, domains and central IT develop a concept that considers all stakeholders’ interests to ensure future adoption, e.g., using surveys or interviews. Afterward, the main driver of the data mesh initiative should select multiple pilot data products. The pilot data products should exhibit the following characteristics: It should span across multiple use cases ($A,Z,\Theta,\Lambda$), be inexpensive ($A,\Delta,K-M$), be small ($A,\Lambda,M$) but impactful ($A,Z,I-\Lambda,N,O$), and allow for easy and quick wins ($A,H,\Lambda,\Xi,O$). Consequently, drivers of the data mesh initiative are able to present successful use cases to get approval from the top management and promote data mesh throughout the organization ($H,\Xi,O$). Simultaneously, the inexpensive and small approach guarantees acceptance for the initial pilot program. Moreover, educational initiatives and community-building efforts should support initial developments of data products. $\Delta$ emphasizes the significance of celebrating early achievements through events such as pitch nights, community gatherings, or hackathons to build momentum within the organization.

IS4: Conscious adoption ($B,\Delta,H,I-\Lambda,\Xi,O$).Before organizations decide to adopt a data mesh, they should go through a thorough assessment of their current data management systems, architecture, its shortcomings, and the potential benefits of a data mesh. Moreover, they should consider the organizational structure and size. If the organization decides to move forward it is crucial to consciously and carefully introduce data mesh-related terminology. $\Lambda$ & $O$ go as far as to suggest not to use data mesh-related terminology at all. $\Lambda$ & $O$ argue that this creates a profound knowledge of the underlying concept and avoids hype-related misunderstandings (C4). Aiming to improve understanding, $I$ states: “What I tend to do to convince people of this, I relate it to something that they know, which is typically an API. Basically, think of a data product as an API. You wouldn’t release an API to the world without a team behind it and a well-defined definition of what it is”.

IS5: Dedicated ownership ($A,\Lambda,O$).Responsibility shift (C2) highlights the challenge that domain employees perceive the provision of data products as extra work and do not prioritize the task. Since organizations have to work with limited resources (C5), managers need to carefully consider the number of employees and the capacities those employees can spend on the provision of a data product. Our interviews show that managers should favor a smaller team with people dedicated to data product provision tasks over a bigger team where commitment is lower for three main reasons. First, there is no prioritization conflict, because the main prioritization is the provision of data products. Moreover, a main prioritization shields the position from possible reorganization and restructuring efforts within the organization—“what happens when the reorganization comes along? […] everything that is not your main job is eliminated. You do your main job or you get a new job” ($\Lambda$). Last, dedicated data product owners can build stronger expertise, especially in the context of sensitive data and regulatory requirements ($O$).

IS6: Focus on stewardship ($O$). A data steward is a role within an organization or data product team that is responsible for the overall data management, such as rules for interaction with data (Otto, 2011). According to $O$, a data product steward owns the data products and ensures quality, fitness, and availability—including metadata quality (C3). Furthermore, a data product steward has complete knowledge of the existing use cases of a data product; this is a key extension of the data product owner role that focuses on the provision of data products for self-service. $O$ outlines a protocol wherein data consumers submit access requests for data products, providing a short explanation of their intended use. The steward is presented with multiple decision pathways: first, grant approval, second, grant approval and facilitate a connection with a related existing use case to explore potential synergies, third, reject the request due to regulatory or security constraints, or fourth reject due to the presence of an overlapping existing use case. For example, $O$ details an instance where a business unit intended to sell data already marketed by another part of the organization. Proceeding with this plan would have diminished revenues from the current sales and led to the duplication of organizational structures.Consequently, stewards can leverage their knowledge to effectively reduce redundant work (MF6). However, professionals need to balance the advantages against the possible creation of bottlenecks (MF1) and silos (MF3) as previous approval may come at the expense of rapid development and self-service. In a data mesh environment, either the data product owner can take on these additional responsibilities or share them with another member of the data product team.

4.5. Impact

Out of 15 interviewees, ten share experiences from a transformation towards a data mesh in the late exploration & bootstrap phase ($\Gamma,\Delta,Z,\Theta,K-O$). Furthermore, eight interviewees share insights from the second, expand and scale phase ($A,B,\Delta,E,H,I,\Xi,O$). Consequently, all interviewees are able to provide (preliminary) details on the impact of their data mesh adoptions.

I1: Accessibility ($A-\Gamma,\Delta,Z-\Theta,\Lambda,N,O$).The most prominent impact of data mesh adoptions that we have observed is the improvement in accessibility, which directly relates to the reduction of bottlenecks and silos (MF1, MF3); “You can do your data analysis on your own without asking anyone. And if you find something to improve, you just do it on your own. And again, you have to ask nobody.” ($\Gamma$).

I2: Speed ($\Gamma-E,H,I-\Lambda,\Xi$).Furthermore, interviewees observe an increase in speed. As we describe in MF1, the reduction of bottlenecks relates to speed improvements along two dimensions. First, time to get access and second, time to market for new products and services. Interviewees observe improvements for both; “a case that would have taken five months in the past, can be done now in three sprints of six weeks” ($\Xi$).

I3: Data quality ($A,\Gamma,\Delta,M-\Xi$).Moreover, interviewees observe an increase in data quality due to clearly defined responsibilities; “data quality is better as one data owner focuses on a specific data domain” ($A$). $\Gamma$ reports the data to be “much more up to date and less error-prone”, whereas $\Delta$ reasons that data owners have a stronger incentive due to increased accountability to deliver high-quality data products: “if it gets tracked, it gets reported and of course, no one wants to be at the bottom of the list”.

I4: Reduction of redundancies ($B,\Delta,H-I,M,O$).“Develop once, use many times” ($\Delta$); directly related to MF6, interviewees observe a successful reduction of redundancies. In particular, $B$ states that providing the most popular data sets in a prepared, clean format provides “massive value because you have lots of engineering teams reusing those datasets”.Additionally, $\Theta$ observes a more efficient extract, transform, load (ETL) process due to standardization: “they implemented hundreds of data pipelines to hundreds of systems. Hundreds of times of effort. With data mesh, you simply provide a standardized API to the data marketplace”.$O$ further mentions that domain knowledge can be transferred more easily.

I5: Data-driven ($M-\Xi$).Along with further impacts, we observe that organizations become more data-driven. This is in line with the overall strategy of organizations and according to interviewees correlates with a change in employees’ mindset and a broad set of measures.

I6: Trust ($I,K,M$).Additionally, interviewees note an increase in data trust due to a higher domain knowledge (MF2) and data quality; “suddenly, you have better trust in data because, behind the data, there are business people responsible for these data; the quality, the data, and the product managers.” ($M$).

I7: Reliability ($A$). Moreover, $A$ reports higher reliability in comparison to the data warehouse concept as a consequence of improved accessibility and data provisioning; previously, the central database experienced crashes during peak demand periods when parallel queries led to system overloads and failures “because of the distributed access we observed lower down-times of data access” ($A$).

I8: Usage ($B,H,\Xi,O$).Finally, interviewees report an “explosive growth” ($H$) in the number of use cases enabled by the improved data accessibility. More precisely, the lack of previous access barriers allows employees to develop new and iterate over use cases in a much shorter amount of time.

Our observations indicate a generally positive impact from adopting data mesh, demonstrating that organizations can benefit even in the early stages of transition.

4.6. Organizational Archetypes

We acknowledge that the approach to an organization’s data management systems and architecture is highly individual. Consequently, there is no one-size-fits-all approach and organizations need to weigh trade-offs and apply implementation strategies according to their needs. Nonetheless, in our findings, we identify two archetypes of organizations that share similar motivations, challenges, and implementation strategies. In this subsection, we describe both archetypes and highlight the most important findings.

A1: Startup & scaleup ($B,\Gamma,\Delta,E,I,M,\Xi$). Startups & scaleups are young organizations that operate in a dynamic, hyper-growth environment. They are early data mesh adopters and on average boast a higher readiness and maturity. Furthermore, startups & scaleups usually consist of a younger workforce with a progressive organizational culture underpinned by a mature data culture where data and AI are integral to their product or service offerings. Due to the newness of the organization, startups & scaleups usually lack legacy systems and are able to start architectural changes on a greenfield approach. Additionally, due to their recent establishment and a more limited range of products or services, these organizations often feature a more centralized structure. As a result, data mesh initiatives are typically driven top-down, supported strongly by top management.

Because of the hyper-growth environment, their focus is on speed and scalability (MF1). However, leveraging domain knowledge to improve data quality (MF2) and the reduction of redundancies (MF6) are also of great importance to improve ML capabilities and avoid redundant work in data preparation. MF1-2 and MF6 result in the following main challenges.

First, the dynamic environment and data-driven product and service offering typically require a high meta(data) quality (C3). Further, transitioning towards a federated organizational structure can be challenging (C1).In contrast, acceptance and resistance to new initiatives (C6), as well as comprehension (C4), are typically less problematic due to flatter organizational hierarchies and the top-down approach. Neither are monetary resources (C5) typically the main challenge, as organizations favor speed and growth over profitability.

Given these settings, several implementation strategies are of great importance. The establishment of a cross-domain steering unit (IS1) is suggested to address standardization challenges and to provide clear guidance across domains. Although quick wins (IS3) are less critical in these top-down driven initiatives, empowering teams and observing outcomes (IS2) is crucial to creating the right incentives and ensuring the production of high-quality data products. Furthermore, dedicated ownership (IS5) is vital in maintaining clear responsibilities and accountability in a fast-paced environment, essential for the ongoing success and scalability of data mesh initiatives. IS6, focusing on stewardship, should be applied very carefully, as it has the potential to hinder the rapid development of new products or services. This is especially true since the duplication of whole use cases is less likely to occur in smaller organizations with lower hierarchies.

This approach allows startups and scaleups to effectively navigate their unique challenges and capitalize on their inherent agility and readiness for innovative data management practices.

A2: Established organizations ($A,\Gamma,\Delta,Z,H,\Theta,K,\Lambda,N,\Xi,O$). Established organizations have a long history and are large in terms of headcount and revenue. They consist of a senior workforce and possess a hierarchical structure with semi-autonomous units that cover multiple fields of business. Because of the company’s history, established organizations frequently possess (multiple) legacy systems and more conservative organizational cultures. Regularly, they rely on centralized data storage concepts like data warehouses or data lakes.

The main motivational factors for established organizations to embrace data mesh are a reduction of bottlenecks and data silos for better accessibility (MF1, MF3), an improvement in (meta) data quality (MF2), clear data ownership (MF4), and a reduction of redundancies (MF6).

The primary obstacle they encounter pertains to organizational challenges. Organizational frameworks often tend to be inflexible, leading to issues related to the federated data governance (C1) and resistance (C6) within the organization. The rigid approach towards allocating financial resources (C5) hinders the required transformation for embracing data mesh. This is particularly noteworthy because quantifying the financial advantages of enhanced data quality and accessibility can be a complex endeavor.Consequently, they do not provide good leverage for internal budget negotiations. Furthermore, expensive legacy systems make the business case for data mesh harder to justify.

Given the outlined challenges, it is of great importance to create quick, effective, but inexpensive wins in the early stages of the adoption (IS3). With this approach data mesh initiatives can fly under radar in the pilot stage and if successful leverage pilot data products or domains to promote widespread adoption and negotiate budget.

Further, we suggest establishing a cross-domain steering unit (IS1) to consolidate multiple data mesh initiatives, enhancing overall data accessibility across the organization. Implementing dedicated ownership (IS5) and the empower and observe strategy (IS2) should also be considered, as these align well with the hierarchical structure of established organizations, ensuring clear responsibilities and fostering accountability within the data governance framework.

5.1. Summary & Literature Embedding

Data mesh is a socio-technical concept including architectural aspects that enables organizations to become more data-driven. However, the architectural concept of data mesh is novel and needs further scientific exploration—especially with regard to motivational factors, challenges, implementation strategies, impact, and archetypes.To derive industry insights we conducted 15 semi-structured expert interviews.

We find that interviewees’ perception of data mesh and the associated motivational factors are generally in line with Dehghani’s (Dehghani, 2022) proposed concept. The interviewees formulate the need for faster, scalable solutions to reduce bottlenecks, leverage domain knowledge, and break down silos (MF1-3). In addition, the establishment of data ownership and the reduction of data redundancy are well documented (MF4, MF6). Furthermore, we find that the current hype around data mesh motivates stakeholders within organizations to adopt data mesh for the sake of adoption (MF5). However, data mesh is not a universal concept for data architectures and data mesh adoption should not become an end in itself. This insight highlights the need for clear fact-based discussions and communication. Reference (Priebe etal., 2021) address this concern with a preliminary review of selected big data architectures.

In addition to motivational factors, we find several challenges that are partly reflected in current literature (Otto, 2022, c.f.). First, we identify the transition towards federated data governance as a main challenge (C1). Reference (Vestues etal., 2022) provide further evidence for this challenge as they argue that federated governance can be unfeasible in specific situations and highlight the importance of central steering units to provide cross-domain insights. Furthermore, they raise concerns regarding the rightful access to data products (Vestues etal., 2022). Reference (Podlesny etal., 2022) extend this argument and provide extensive analysis for privacy challenges with regard to the data mesh concept. Additionally, researcher describe agreement among domains and compliance issues as a “moving target” (Joshi etal., 2021).In alignment with reference (Vestues etal., 2022), we identify the shift of responsibility as a main challenge (C2). The authors argue that domains lack competencies, motivation, and resources to provide data products (Vestues etal., 2022). Moreover, limited resources (C5), as well as acceptance and resistance issues (C6) are well-established challenges in the context of organizational and cultural change management; professionals and researchers should consider the rich body of literature to complement the findings of this work (Oakland and Tanner, 2007; By, 2007; Joshi etal., 2021; Vestues etal., 2022, c.f.).As the interviews reveal, the adoption of modern architectures (MF5) can be associated with a low comprehension of the data mesh concept (C4), as adopters do not want to ”miss the boat”—instead of having deeply factual rooted motivation. This relationship finds little attention in recent publications, which is surprising as interviewees mention the challenge across multiple use cases. Neither does metadata quality (C3) in the specific context of data mesh.

Addressing the aforementioned challenges, we formulate implementation strategies (IS1-6). First, to address C1 and C3, we argue that professionals should introduce a cross-domain steering unit (IS1). A recent study supports this claim. Reference (Whyte etal., 2022) reasons that organizations should embrace decentralized and centralized capabilities for the next-generation data platform. Furthermore, we argue that organizations need to empower domains and closely observe and track their progress (IS2) to address C2, C3, and C5. In line with reference (Dehghani, 2022), we describe the need to empower domains. Nevertheless, the great importance of observing data products as motivation for providers and for prioritization purposes is novel and needs further research. Furthermore, reference (Dehghani, 2022) suggests an iterative approach for the adoption of the data mesh concept. We can confirm that approach after the analysis of our interview transcripts and provide further insights in IS3 on how to navigate organizational challenges (C5, C6).

Conscious adoption (IS4) is an important aspect when implementing a data mesh. Organizations should first go through a thorough assessment of their current data architecture before adopting data mesh in an organization—instead of a purely hype-related motivation (Burgess, 2018, c.f.). We further suggest professionals favor small dedicated teams over a larger number of employees for the provision of data products (IS5). This implementation strategy addresses the need to prioritize data products using limited resources (C2, C5) (Vestues etal., 2022). Lastly, we argue that organizations should consider extending the data product owners’ responsibilities or creating a data product steward role (IS6) to ensure data quality, compliance, fitness, and accessibility. In cases where organizations have a strong need to control data product access and overview use cases, this may be an interesting consideration. However, we acknowledge that IS6 somewhat conflicts with Dehghani’s (Dehghani, 2022) emphasis on the easy accessibility of data products throughout the organization and may create new bottlenecks or silos (MF1, MF3)—therefore, IS6 needs to be applied carefully.

Across all interviews, interviewees state the positive impacts of their data mesh adoption effort. Our findings include improved accessibility (I1), speed (I2), data quality (I3), a reduction of redundancies (I4), a more data-driven organization (I5), an increase in trust (I6), reliability (I7), and data and analytics use cases (I8). The overall positive impacts highlight that organizations can reap the expected benefits even in the early stages of the adoption, confirming Dehghani’s (Dehghani, 2022) perspective on data mesh impacts.

We introduce two organizational archetypes embracing the data mesh concept: Startup & scaleup (A1) and established organizations (A2). Professionals should assess implementation strategies based on their unique situations, for that purpose, A1 and A2 provide first steps to more specific guidelines.

5.2. Contributions & Implications

This work enhances the empirical understanding of data mesh, addressing a vital gap in the existing scientific literature, which is currently limited and largely theoretical due to the novelty of the research area. With only a small number of independent, scholarly works exploring the practical aspects of data mesh, this paper provides vital insights that are both comprehensive and scientifically rigorous.

We identify motivational factors (MF1-6) for organizations adopting data mesh, such as the need for agility, scalability, and improved data accessibility. Understanding these factors aids in justifying investment and effort in transitioning to data mesh architectures.

Further, this research identifies several challenges (C1-C6) that enable organizations contemplating the implementation of a data mesh to prepare and develop strategies to address these challenges proactively, thus mitigating risks that could potentially jeopardize the data mesh implementation.

Next, our outlined implementation strategies (IS1-6), tailored to address specific challenges, offer preliminary guidelines that can facilitate successful implementation and reduce associated risks.

Moreover, researching the impacts (I1-8) of adopting a data mesh is crucial as it provides empirical evidence on the benefits, helping organizations to assess the real-world effectiveness and scalability of this architecture enabling better strategic planning.

Last, archetypes based on organizational characteristics can guide companies in customizing their approach to data mesh adoption—highlighting more likely challenges to occur and corresponding implementation strategies.

The implications of this study are profound, particularly given the scarcity of independent research on the topic. For practitioners, this paper serves as a preliminary guide to understanding and implementing data mesh in a way that is both strategic and aligned with their specific organizational challenges. The insights it brings to addressing common pitfalls and leveraging effective strategies can significantly reduce the risk and enhance the success of data mesh initiatives.

This research also fills a gap for the academic community, providing a well-documented analysis of how data mesh is being implemented across different industries. It sets a foundation for future research to build upon, particularly in exploring how these strategies and challenges evolve as data mesh matures and becomes more widely adopted.

Finally, this work serves as a valuable reference for both academics and practitioners in the landscape of (big) data management. It offers a clearer understanding of motivations, strategies, and impacts and transparently communicates these aspects while also highlighting potential challenges. In this way, it helps to position data mesh in the broader landscape of (big) data management.

5.3. Limitations & Future Research

To answer our research question, we conduct 15 semi-structured expert interviews.The qualitative nature of our work results in limited quantitative validity. However, we justify the qualitative approach with the novelty of the research topic.Nevertheless, future research should investigate the findings on a quantitative level, e.g., using surveys.

With respect to our sampling method, we acknowledge a potential bias, recognizing that interviewees might frame their data mesh implementations as more successful than they actually are, especially as they aim to publicly position themselves as leaders in the domain of data mesh. To address this challenge, we have been transparent in communicating our focus solely on research activities. Moreover, we anonymize interviewee data to encourage them to speak freely about both the positive and negative aspects of their experiences within their organizations. This approach is designed to mitigate bias and ensure a more accurate and nuanced understanding of data mesh implementations.

Our main contribution are industry insights for professionals adopting a data mesh. However, we acknowledge that these practices are only applicable to a certain degree depending on the organizations’ individual situation. Thus, professionals need to adjust their strategy accordingly—incorporating only relevant aspects.

The presented organizational archetypes are a first step towards creating more fine-grained guidelines for organizations based on specific characteristics. However, we acknowledge that the industry insights in general and the archetypes specifically lack quantitative evidence. This creates an enormous opportunity for future work, as this exploratory qualitative approach can be complemented with a quantitative study.In this context, researchers could investigate data mesh in small and medium-sized organizations to extend the framework of archetypes.

Moreover, future work should more deeply cover the technological realization of the data mesh concept. This could include designing technical data products in detail and integrating data warehouses, data lakes, or blob storage to realize the data mesh concept. In addition, possible data mesh topologies and considerations for ideal data mesh node sizes can be explored in more detail.

In summary, a promising field of research lies ahead.