Data User Guide | Evaluation

In the evaluation phase, you step back and critically assess whether the concept you developed is actually worth pursuing. The key question at this stage is straightforward. Does this idea create real value for your organization and under which conditions could that value realistically materialize? Before investing further resources, it is useful to reflect on three aspects.

  • First, what kind of value could this idea potentially create?
  • Second, whether your organization can realistically realize this value in practice.
  • Third, how your organization would ultimately capture the value that is created.

Thinking through these questions helps you decide whether the concept should be developed further or whether adjustments are necessary before moving to the next stage of the framework.

To structure this reflection, it is helpful to distinguish between three closely related dimensions of value.

Value potential refers to the improvements your concept could enable. For example, data use may increase efficiency, reduce uncertainty, improve prediction accuracy or enable new services.

Value realization focuses on whether these improvements can actually be achieved in practice. Organizations realize value only when they successfully integrate data use into operational processes, decision-making routines and governance structures.

Value capture concerns how and by whom the created value is ultimately appropriated. In commercial settings, value capture may occur through increased revenue, cost reductions, improved margins or stronger competitive positioning. In collaborative or ecosystem settings, value capture may also occur indirectly through learning effects, improved coordination or network advantages.

Research in information systems and strategic management shows that data alone rarely create value. Instead, value emerges when organizations embed data into decision-making processes, operational routines and service systems (Alaimo & Kallinikos, 2017; Constantinides & Barrett, 2015).

From a resource-based perspective, organizations generate competitive advantage when they combine data with organizational capabilities that are valuable, rare and difficult to imitate (Barney, 1991). In this sense, data become valuable only when organizations mobilize them through analytics, operational processes and service configurations.

Research on data-driven decision making further shows that organizations benefit from data use when it improves decision quality, enables process optimization and supports service innovation (Lim & Maglio, 2018; Beverungen et al., 2019). Empirical studies also demonstrate that data-driven decision making can lead to measurable productivity and performance improvements (Brynjolfsson et al., 2011).

In regulated domains such as healthcare, value creation may also include broader societal benefits. Data use can contribute to improved public health outcomes, better evidence for policy decisions and new scientific insights (Moore, 1995). Within the European Health Data Space, organizations may therefore need to articulate not only their internal economic rationale but also the societal contribution of their use case. This reflects the growing importance of accountability and public value in data ecosystems (Klievink et al., 2017).

Not all benefits of data use can immediately be expressed in financial terms. Some benefits are tangible and directly measurable. Examples include cost reductions, productivity gains, improved model performance or shorter processing times (Brynjolfsson et al., 2011).

Other benefits are more indirect and may become visible only over time. These include improved organizational learning, stronger analytical capabilities, better coordination across teams or a stronger strategic position within data ecosystems (Teece, 2007; Kilgus et al., 2024).

While these benefits may not immediately translate into financial returns, they often contribute to tangible value in the longer term. Considering both dimensions helps avoid overly narrow economic assessments and supports a more realistic evaluation of data-driven initiatives.

By reflecting on these aspects, you develop a clearer understanding of the effort and the potential value associated with your concept. This prepares the ground for the next step of the framework, the Data Value Assessment, where you examine the contribution of specific data sources in more detail.

Understanding different forms of data value

When evaluating a concept, it is useful to recognize that data value can take different forms. Some benefits are directly measurable, while others emerge more indirectly through organizational learning or improved decision-making.

Tangible value refers to measurable effects. Examples include improved prediction accuracy, reduced error rates, cost savings, efficiency gains or increased revenue. In many technical contexts, researchers estimate such value by analyzing how additional data improve model performance or reduce uncertainty in a specific task (Lundberg & Lee, 2017; Ghorbani & Zou, 2019). At the organizational level, data-driven decision making has also been linked to measurable productivity improvements (Brynjolfsson et al., 2011).

However, not all value generated through data use can immediately be expressed in financial or performance metrics. Intangible value includes improved coordination, stronger analytical capabilities, better strategic decision-making or enhanced organizational learning (Kilgus et al.). While these effects may not immediately translate into financial returns, they often contribute to tangible value over time.

For the evaluation phase, this means that you should consider both dimensions. Some ideas create value through measurable improvements such as cost reductions or performance gains. Others contribute to capabilities, knowledge or collaboration that may generate tangible value only in the longer term.
The following action module provides practical methods that help you assess these different value dimensions in a structured way.

Clarifying the purpose of evaluation:

Before assessing implementation effort and value, it is important to distinguish between different purposes of evaluation. Data and data-driven concepts may be evaluated to estimate expected implementation costs, to assess economic return, to understand strategic impact, or to examine long-term generative potential. The selected evaluation perspective influences which analytical dimensions become relevant. For additional structured approaches to data evaluation, organizations may consult the “Methods & Analytical Tools” section.

Understanding Different Value Creation Logics

Data-driven initiatives generate and realize value in different ways. Before you start working with the evaluation canvas, you should therefore clarify what type of initiative you are assessing.
In practice, you can distinguish between three typical value logics.

  • Cost-oriented initiatives focus on improving internal processes. In these cases, you use data to reduce time expenditure, lower coordination effort or decrease error rates. The main value arises from efficiency gains and cost savings.
  • Revenue-oriented initiatives focus on creating new sources of income. Here, you use data to enable new services, enhance existing offerings or strengthen your market position. Value emerges through monetization opportunities, improved competitiveness or scalable service models.
  • Study-oriented initiatives focus on generating knowledge, evidence or strategic insights. In this case, the value may not immediately appear as revenue or cost savings. Instead, you may improve decision quality, reduce uncertainty, strengthen organizational reputation or contribute to societal knowledge.

You can use the following canvas for all three initiative types. However, the emphasis should differ depending on the type of initiative. If you analyze an internal service, focus primarily on operational improvements and cost effects. If you evaluate a revenue-generating concept, emphasize market impact and scaling potential. If you assess a study, highlight strategic insights, informational value and potential societal contributions.

Economic and Strategic Evaluation

Economic and Strategic Evaluation - Calculation

The following calculation modules translate qualitative value reflections into rough quantitative estimates. The purpose is not to construct a detailed business case, but to develop a transparent and comparable understanding of potential economic impact.

Not all data-driven initiatives generate value in the same way. Internal process improvements primarily create value through efficiency gains and cost reduction. Market-facing services or products create value through customer uptake and revenue generation. Research-oriented initiatives may generate primarily strategic or societal impact, where monetary estimation remains indicative.

Before proceeding, clarify which value logic applies to your concept and select the corresponding calculation approach. The emphasis should always remain on plausibility and consistency rather than precision.

Assessing Implementation Effort

After reflecting on the potential value of a concept, the next step is to assess the effort required for implementation. Even ideas with strong value potential may prove difficult to realize if organizational, technical or data-related requirements are too demanding.

At this stage, you do not need to estimate precise costs or timelines. Instead, focus on forming an initial judgement about the overall implementation effort.

Typical factors influencing effort include the availability and quality of the required data, the need for data harmonization, technical development requirements and the degree of organizational coordination involved.

Considering these aspects together helps you form an overall impression of the expected effort. Based on this reflection, you can classify the concept as low effort, medium effort or high effort.

The following canvas helps you structure this assessment in a transparent and comparable way.

Use this framework to reflect on the expected implementation effort of every specified concept individually. The assessment is based on several dimensions that typically influence how demanding a data-driven initiative is in practice. Rather than focusing on exact costs, the aim is to form a shared qualitative understanding of complexity.

Work through each assessment dimension and consider how strongly the idea challenges existing structures. The categorisation into low effort, medium effort, or high effort should reflect the overall impression across all dimensions rather than a single factor.

Implementation Effort Categories:

  • Data Complexity: Describes how demanding the data environment is. This includes the number of data sources, data quality, level of structure, and the effort needed to access or combine data.
  • Organizational Change: Refers to how strongly the idea affects existing roles, workflows, or collaboration structures. Consider whether new coordination, responsibilities, or adjustments to current practices are required.
  • Technical Development: Captures the expected level of technical work needed to realize the idea, such as configuration of existing tools, integration tasks, or the development of new analytical components.
  • Scope of Implementation: Reflects how broad the initiative is in terms of actors involved, duration, and organizational reach, ranging from small pilots to multi-organizational initiatives.

Reality Check

Before interpreting the calculated results, perform a brief reality check to validate your assumptions. Review whether the estimated time savings and cost inputs reflect a realistic scenario rather than an ideal outcome. Discuss the estimates with relevant stakeholders and consider whether organizational constraints, data availability, or adoption challenges might influence the actual impact.

The goal of this step is not to question the idea itself, but to ensure that the evaluation remains grounded in practical experience and realistic expectations.

References

Aaltonen, A., Alaimo, C., & Kallinikos, J. (2021). The making of data commodities: Data analytics as data work. Information Systems Journal, 31(1), 17–39.

Alaimo, C., & Kallinikos, J. (2017). Computing the everyday: Social media as data platforms. The Information Society, 33(4), 175–191.

Barney, J. (1991). Firm resources and sustained competitive advantage. Journal of Management, 17(1), 99–120.

Batini, C., Cappiello, C., Francalanci, C., & Maurino, A. (2009). Methodologies for data quality assessment and improvement. ACM Computing Surveys, 41(3), 1–52.

Beverungen, D., Matzner, M., & Janiesch, C. (2019). Information systems for smart service systems. Information Systems Journal, 29(6), 1091–1106.

Brynjolfsson, E., Hitt, L. M., & Kim, H. H. (2011). Strength in numbers: How does data-driven decision-making affect firm performance? SSRN Electronic Journal.

Constantinides, P., & Barrett, M. (2015). Information infrastructure development and governance as collective action. Information Systems Research, 26(1), 40–56.

Ghorbani, A., & Zou, J. (2019). Data Shapley: Equitable valuation of data for machine learning. Proceedings of the 36th International Conference on Machine Learning (ICML).

Hagiu, A., & Wright, J. (2020). When data creates competitive advantage. Harvard Business Review, 98(1), 94–101.

Han, X., Wang, L., Wu, J., & Fang, X. (2026). Data valuation for vertical federated learning: A model-free and privacy-preserving method. MIS Quarterly, 50(1), 177-210.

Kilgus, T., Patecka, A., Schurig, T., Kari, A., Gubser, R., Gersch, M., Wessel, L., & Fürstenau, D. (2024). Creating value from the secondary use of health data: International examples, best practices, and opportunities to scale. Communications of the Association for Information Systems, 55, 507–534. https://doi.org/10.17705/1CAIS.05520

Klievink, B., Bharosa, N., & Tan, Y.-H. (2017). The collaborative realization of public values and business goals. Government Information Quarterly, 34(1), 67–79.

Lim, C., & Maglio, P. P. (2018). Data-driven understanding of smart service systems through text mining. Service Science, 10(2), 154–180.

Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems (NeurIPS).

Melville, N., Kraemer, K., & Gurbaxani, V. (2004). Information technology and organizational performance: An integrative model of IT business value. MIS Quarterly, 28(2), 283–322.

Moore, M. H. (1995). Creating public value: Strategic management in government. Harvard University Press.

Peppard, J., & Ward, J. (2016). The strategic management of information systems (4th ed.). Wiley.

Teece, D. J. (2007). Explicating dynamic capabilities: The nature and microfoundations of sustainable enterprise performance. Strategic Management Journal, 28(13), 1319–1350.

Teece, D. J. (2010). Business models, business strategy and innovation. Long Range Planning, 43(2–3), 172–194.

Yoo, Y., Henfridsson, O., & Lyytinen, K. (2010). The new organizing logic of digital innovation: An agenda for information systems research. Information Systems Research, 21(4), 724–735.

Proceed to next phase