Validation of 5G Use Case Solutions – Simultaneous Assessment of Business Value and Social Acceptance in Early Stages of the Research and Innovation Projects

Per J. Nesse^1,*, Hanne Stine Hallingby², Luigi Briguglio³ and Ioannis Markopoulos⁴

¹Telenor Research and Innovation, Norwegian University of Science and Technology, Trondheim, Norway
²Telenor Research and Innovation, Oslo, Norway
³CyberEthicsLab, Rome, Italy
⁴NOVA, Athens, Greece
E-mail: per-jonny.nesse@telenor.com; hanne-k.hallingby@telenor.com; l.briguglio@cyberethicslab.com; Ioannis.Markopoulos@nova.gr
*Correspondng Author

Received 13 April 2022; Accepted 08 November 2022; Publication 17 March 2023

Abstract

This article recommends a lean approach when validating next generation mobile technology (5G) use case solutions in research and innovations projects (R&I). This validation should include a simultaneous and iterative assessment of business value and social acceptance beyond the traditional assessment of technological performance. The context studied in the article is R&I projects funded by the European Commission. Here a consortium of academic and enterprise partners jointly develops 5G product and service solutions for different industry verticals. From the literature review we learn that partnerships and ecosystems accelerate innovations as well as the significance of performing the validation already in the initial steps of the innovation projects. Based on a case study of seven completed 5G R&I projects we revealed that lean methodology for development of use case solutions were not applied. Furthermore, the assessment of potential business values from the 5G use case solutions, was postponed to the later stages of the R&I projects, and that assessing the social acceptance of these solutions were absent. After piloting the lean start-up methodology in an ongoing 5G R&I project (5G SOLUTIONS) we found that the partners advocated technological performance, business value and social acceptance to be ranked equally important to successfully validate the 5G use case solutions in the early steps in the innovation process. The two pilots we executed was performed as online workshops with partners from the 5G SOLUTIONS consortium and the media and entertainment vertical. The prototype was presented and feedback from the stakeholders were collected using surveys and interactive software tools. We recommend further studies and pilots of our lean start-up validation methodology in forthcoming 5G R&I projects funded by Horizon Europe for confirmation of our findings and further refinement.

Keywords: Validation, Research and Innovation projects, Business value, Social acceptance, Technological performance, 5G SOLUTIONS.

1 Introduction

Introducing product and service innovations is a risky business, and often innovations launched by start-ups, SME’s and even large corporations fail. Reasons for this failure is often due to lack of understanding market and user needs as well as missing business models [1]. This is a major risk in fifth generation of telecommunication systems (5G) which is currently in its early innovation phase. 5G is expected to influence product and service development as well as the business opportunities for many industry verticals [2]. 5G will introduce lower latency, higher capacity and increased bandwidth compared to current 4G technology. Significant economic and social value is expected be generated from use case innovations activated by 5G. According to a study by HIS Markit [3], $13.2 trillion in global economic value will be made possible by 2035, generating 22.3 million jobs in the 5G global value chain alone.

The European Commission (EC) established the Private Public Partnership (5G PPP) agreement with European ICT industry regarding 5G in 2013. Horizon 2020 is a program that supports the 5G PPP agreement with an earmarked funding of EUR 770 million; the EU industry is set to match this investment by up to 5 timers to more than EUR 3 billion [4]. So far three waves of R&I projects are funded with a mix of industry, SME and academia/research institutional partners using lean innovation approach and frameworks. However, the societal benefits (regulatory, health and public acceptance wise) from implementation of the 5G technology that could contribute to trigger wider interest, facilitate awareness, and mitigate distrust, is less frequently described. In the last years Europe has seen an increased number of nationwide protests the 5G technology and its deployment and impact it will have on people’s health, safety, and the environment [5, 6]. Disruptive technologies such as 5G may also affect current operations and practices in a market/industry and hence raise barriers and slow adoption and implementations [7].

In general, uncertainty and fuzziness related to customer, technology and competition is high early in the project and with a negative impact on success. Moreover, lean start-up methods are most frequently discussed in relation to projects with technology/market newness performed by start-up’s [8]. Inside large established firms, the focus on lean start-up challenges are mostly organizational hurdles for implementation [9, 10]. Although lean and agile methods ensure trials through their iterative approach [11] they are used less frequently by research institutes where the lack of entrepreneurial activities also is a major obstacle [12].

5G R&I projects are large, complex and includes many consortium partners, established corporations, SME’s/start-ups, as well as research agencies [13]. How radical innovations are handled within actors in 5G PPP projects are less addressed compared to the context of large corporations’ internal organizations [9]. Methods for how to perform business validation of use cases in projects mostly concern later innovation stages [14]. How to perform early validation in R&I projects is especially interesting research wise, because of the high technological and commercial uncertainly and fuzziness and their large impact on success [15]. 5G-SOLUTIONS is a 5G R&I project which has systematically used agile methods for developments of vertical use cases prototypes also in the initial phases [16]. In the midst of the project, the absence of significant societal validation was identified, and the ambition was extended to explore and test such methods. Societal validation is an immature field and part of the work involved the development of such methods. We suggest that some sources of risks and failures for innovations may be found in the lack of emphasize on early development stages as well as the societal aspects in 5G R&I partnership projects. With this backdrop, we raise the following research question: How can the lean start-up methodology support validation of 5G use case solutions in early stages of research and innovation projects? Our objective is to suggest a method for early validation of use cases solutions in 5G R&I projects that overcomes the limits of current methodologies and practices. The business and social aspects of the validation method is focused due to the significant values 5G is expected to generate here.

A use case describes all the ways an end-user wants to use a system [17], and is most often made up of three basic elements [17, 18]: Actors, System and Goal. The actors are the users that interact with the system (a technology, a process, a device etc.) to perform an activity/task and hence reach an overall goal. Designing a use case involves different steps [18] such as identifying all the users, describing their profiles and roles, describing (scenario based) different sets of activities and requirements to the system (in templates and diagrams), and finally reviewing and validating whether the response of the system met the users’ requirements and goals. In the context of the 5G R&I projects, we see a use case as a concrete situation or event, where one or alternative solutions may be applied, and where identified persons hold stakes in the situation/event. In this article we refer to this using the term “use case solution”. In the initial stages of the R&I projects, an early version (minimum viable product (MVP)/prototype) of the solution are presented for feedback and revisions.

2 Literature Review

A (business) ecosystem refers to a system of different actors who collaboratively provide a comprehensive innovative solution for a customer or end user [19]. The Triple Helix model [20] is an ecosystem that describes how innovations are created in collaboration between three major actors: industrial companies, academia/universities, and the public/government. The collaboration creates a win-win solution for all actors – academia achieves financial support for new research programs, industry secures a subsidized approach to valuable research results, and the government ensures economic growth, advanced industries, and a competent workforce. In the quadruple helix, citizens and media/creative industries/culture actors are included, reflecting the increasing importance placed on public values and societal led innovation [21]. Chesbrough [22] argues that SMEs are attractive innovation partners and suppliers for large industrial companies because of their size, focus, entrepreneurial persons, and speed [7]. Moreover, they often embody important innovation experiments in technologies or business models, experiments that could teach a great deal to observant large firm. Granstrand and Holgersson [23] argues that ecosystems for innovations also should include artifacts such as product and technologies as well as competitive relations. Mixed with cooperative relation transcends into a form of coopetition and together with shared technologies this enables value creation as well as value capture within and between the system actors. However, the triple helix model is lacking practical guidance on the use case level, and more concrete bottom-up innovation process approaches needs to be embraced also by research and innovation projects.

2.1 Business Validation

Design thinking and lean start-up/agile way of work are state of the art bottom-up innovation process principles applied in industries and businesses today for specific product or services. Design thinking is an iterative process useful when you want to explore and tackle research-based problems that often are ill-defined or unknown. Lean start-up methodology is a build-measure-learn (develop, prototype and test) feedback loop developing an MVP to begin the process of learning as quickly as possible to get a desired product to customers’ hands faster [24]. The importance for short and repetitive development cycles, centering on customer feedback is also incorporated in the agile methods [25]. Edmondson [26] claims that companies fail to have a strategy to learn from failures when testing products concepts. She recommends the developers to check whether: (1) the circumstances for the pilot are real vs. optimal, (2) that the actual customers are involved rather than employees from the developer, and (3) the extent to which actions are taken based on results from pilot. Design thinking can also support innovations with breakthrough technologies where the market is distant and product applications/user needs are not yet identified according to Magistretti et al. [27]. Their study of an electronics company shows how different design thinking logics and tools can be applied in the earlier phases of research and development projects (the R of R&D).

Table 1 compares two best practice methods for more agile and leaner innovation. Cooper [28] has suggested an agile/iterative stage-gate method that allows a project team to quickly finish designing a product prototype through a series of “build-measure-learn” feedback loops. Blending iterative cycles with the stage-gate method during the idea-to- launch stages enables early and continuous customer feedback on important product features, more efficient planning and to fail fast, cheap, and often on the path towards a successful product [25, 29]. The analysis stage involves initial screening and assessment of needs and wants, market segments, competitions as well as assessment of technological solutions, vendors, patents, risk and business aspects, laws, and regulations. The business case stage involves specification and alpha testing of product/service concept/MPV’s and designing an initial business model, market studies and financial analysis. The following stage involves product development, beta product tests along with market trial sales, while the final launch stage includes production/operation and marketing start-up together with go-to market launch [30].

Table 1 Two theoretical model approaches to lean innovation process stages

	Research and Innovation Process Stages
Lean Models	Stage 1	Stage 2	Stage 3	Stage 4
Agile “stage gate” model [28, 29]	Analysis	Business Case	Development/Test	Launch
	• Initial screen • Use and need study • Preliminary technical and market assessment	• Concept test (PoC/MVPs) • Detailed market research/value to customer study • Business and finance analysis	• Prototype development • In house alpha testing and beta test/field trial • Test market/sell, business model	• Production/ start-up • Marketing plan execution • Go to market launch
	Customer	Solution	Value Proposition	Growth
	Discovery	Discovery	Discovery	Discovery
Lean start-up R&D model [12]	• Initial concept • Value of new business • Committed people	• High value concept • User acceptance • Resources to move forward	• Validated and desired solution • Initial business model • Resources to move forward	• Money to create new business ideas • Channels, brand

Still [12] has suggested a similar four stage model for innovations in universities and public research institutes. Her lean start-up-based model is accompanied with a Finnish use case where it has been applied successfully in the early phase of the innovation process.

For the first two stages in Table 1, both approaches recommend studies of users, and their needs/demands as a basis for developing a concept (MVP) where functions/features/requirements are tested and validated/accepted by the target customers. For the two latter stages of the lean innovation process in Table 1, business models and business plan canvas methods are applied [12]. An often-used business model method contains nine building blocks – value proposition, customer relationship, channels and distribution, customer relationship, customer segments, revenue streams, key activities, key resources, key partners and finally the cost structure [31]. The business plan documents the implementation of a particular business idea/use case. This includes the funding of costs related to activities necessary to perform in relation to development, production, distribution (channels) and further growth of the product brand.

Validation of successful new product innovation use cases most often emphasize commercial and technological performance. Initially, Cooper [32] suggest profit, pay-back time, and market share when measuring success. Later, Cooper [33] introduce more qualitative go-kill criteria such as strategic fit, competitive advantage, and market potential. Zhang and Doll [15] introduced three project outcomes: product (performance, cost etc.), process (time to market, project teamwork etc.) and financial outcome (profit, market share). In general, societal aspects found in more holistic approaches are not focused on design thinking and lean start-up methods.

2.2 Social Validation

As previously mentioned, it is essential to analyze (in addition to technical and economic aspects) the social aspects that influence the acceptance of 5G technologies. We find early signs of societal acceptance methods in Pinto and Slevin [34] who separated between project success (time, cost, performance) and client success (satisfaction, use and effectiveness). Upham et al. [35] define acceptance (p. 103) as “a favorable or positive response (including intention, behavior and – where appropriate – use) relating to a proposed or in situ technology or socio-technical system, by members of a given social unit (country or region, community or town and household, organization)”. The definition is based on the classification levels of social acceptance by Wüstenhagen et al. [36] and related to energy technology and innovations. In addition, they point out three different actor groups related to these three levels: consumer or citizen acceptance (i.e., public), stakeholder (with interest in the outcome) acceptance, or political acceptance (e.g., policy support) and individual/end user acceptance composed of attitudinal elements such as behavioral intentions and actual behaviors to use or adopt the technology. A large review [37] of the social acceptance literature for energy technology and fuels reveals a trend in recent publications from political issues related to social acceptance to be concentrated around individual-level contextual and psychological factors. Shah et al. [38] studied consumers intention to purchase 5G services in China. The survey study revealed a significant relationship between the consumers environmental awareness, knowledge, and health-consciousness attitude, to their intentions to purchase such 5G services. In relation to technology, trust refers to specific beliefs about the way that technology operates through a work environment. More specific, it is defined as individuals’ judgment or expectation of a given technology’s helpfulness, reliability, and functionality [39]. Different studies suggest that trust has an imperative impact on the adoption of technology, e.g cloud technology or e-commerce [40, 41] or peoples’ intention to use a technology, e.g mobile banking or self-service hotel technology [42, 43]. Akbari et al. [44] shows that trust has an important influence on the intention to accept 5G technology in their survey study of consumers in Iran and USA. Here, trust mediate the relationships among perceived ease of use, perceived usefulness, and the intention to accept 5G technology.

Nielsen [45] divided (product) acceptability in practical and social acceptability. The former consisted of factors such as usefulness, costs, combability and reliability, while the latter was viewed as the users/peoples’ attitudes toward a technology or innovation in general, as opposed to the usability of the product/service innovations. Venkatesh et al. [46] developed a unified model for acceptance and use of technology (UTAUT) with four determinants for usage and behavior: (1) Performance expectancy (believing that the technology will assist them in performing the job duties), (2) Effort expectancy (perceives a technology to be easy to use), (3) Social influence (the extent individuals change their behavior to meet the demands of a social environment)), and (4) Facilitating conditions (believes that his/her organization supports the change).

Kim [47] claim that acceptability is a higher-level concept than that of usability involving more complex social, organizational, and financial aspects. User acceptance is especially important to study in relation to user experience when validating innovative (computing) technologies. Typical for these innovations is that they are adopted by early users rather than the general end-users. Factors beyond usability that affects user acceptability were safety, credibility, culture, politics, technical experiences, demographic characteristics, and brand/price [47]. Koelle, Ananthanarayan and Boill [48] reviewed social acceptance from a Human Machine Interface (HMI) point introducing both internal (performer/users’ social acceptance) and external stakeholders (spectators/social acceptance). If the HMI causes a negative change to self and external image, it shows a lack of social acceptability. Their review concluded that there is no agreement upon the way to ask for social acceptance. Georgieva, Beaunoyer and Guitton [49] suggests a conceptual model for ethical and socially acceptability in relation to use of tracking technology during the pandemic. Here social acceptance was understood as the conditions (transparency, socio-cultural determinants, security, and reversibility) favoring comprises to reach the largest consensus among stakeholders/population possible. Vik et al. [14] introduces a balanced readiness level assessment tool for exploring new and emerging agricultural technologies. This tool contains five readiness dimensions (technology, market, regulatory, acceptance, organizational) and nine levels of maturity for each dimension. The acceptance readiness dimension captures the social acceptance (society and individual) of the new technology. Level 1 describes a situation where the technology is or will be seen as illegitimate or socially unacceptable; while level 9 is the situation when use and production of the technology is generally accepted or not questioned at all.

Yuen et al. [50] studied determinants (and their interrelationship) on public acceptance of autonomous vehicles (AV), which is one of several 5G enabled applications. The results reveal that the influence of the innovation diffusion factors (relative advantage, compatibility, complexity, trialability and observability) on public acceptance (consider using the AV myself when available, recommend AV to family and peers, encourage others to use AV etc.) is fully mediated by the public’s perceived value (meet my mobility requirements, cost savings, social and environmental friendly etc.) of AV’s, which again is partially mediated by the public’s trust (safe and reliable etc.) in AV’s. Sailesh and Mekker [51] extend the technology acceptance model (TAM) when examining public acceptance of connected vehicles (CVs). Based on a survey of US adults they found that perceived data privacy and security associated with the technology was found to shape trust, attitude, and behavioral intention to use CVs, in addition to the predictors of original TAM: perceived usefulness and ease of use. Saari et al. [52] apply a reduced version of the technology acceptance model (TAM3) is in the context of social robots. A survey of end users in a workplace setting was performed during the fuzzy front-end of the innovation phase of a robot development project. The results reveal that perceived usefulness had a significant influence on participants’ behavioral intentions (using the robot when it is available). However, the factors influencing the perceived usefulness vary between the innovation adopter types. For early adopters, result demonstrability influenced perceived usefulness, while the mass market users perceived enjoyment influences perceived usefulness. The difference innovation features/functionalities are important to be aware of when launching the first versions of the new social robot in the market. All three studies rely on the behavioral intention can be applied as a measure of acceptance of technology. Venkatesh et al. [46] verify this correlation between actual use of the technology and the intention to use it.

Although there is a gap in social acceptance research studies related to 5G technology enabled innovations, we find contributions that are applicable for our content. Generally, Upham et al. [35] recommend social acceptance to be assessed by different actor groups (e.g citizens, stakeholder/political and individual/end users). The literature review also reveals that dimensions such as usefulness, trust, health, safety and environmentally friendliness etc. have impact on the intention and actual use of the new technology and innovation. These dimensions are important to validate early in the research and innovation projects since they can delay the implementation and slow the adoption of the project outcome [7]. Unified acceptance models such as UTAUT goes beyond previous single TAM models [53], but lack a systematic approach to validation of the disruptive impact on use case level for businesses and the society as a whole [54]. Moreover, we point out the significance of trials and validation of technical, economic and social aspects of novel solutions at early stages to increase the positive influence on the innovativeness and financial success of the product [29, 55, 56]. Optimal testing requires involvement and collection of requirements from actual customers [26, 57] and agile/design thinking methods and tools promote success when applied in the these initial research and innovation stages [27]. The extended Triple Helix model [20] offers a framework for collaborative ecosystem-based innovation partnerships but lack practical guidance on the innovation use case level. This involves actor groups such as SMEs, corporations, and public agencies as well as academia/university spin offs [12].

3 Research Methodology

Here we describe the methods applied for data collecting and analysis in order to answer the research question presented in the introduction chapter.

3.1 Data Collection

The data sources and collection methods are covered in Table 2.

Table 2 Description of data sources

Method	Information Sources	Year
Case study of seven completed 5G R&I projects	Seven selected 5G R&I projects from three Horizon 2020 calls/phases: • Phase I (2015-2017): 5G EXCHANCE, 5G CHARISMA, 5G SESAME • Phase II (2017-2020): 5G MONARCH, 5G CITY • Phase III (2019-2021): 5G VINNI, 5G HEART	2021
	Various documents from 5G project websites and EU database (Cordis): • Project deliverables, scientific articles, newsletters etc.
Case study of single ongoing 5G R&I project (incl. pilots).	User/customer experiment in single ongoing Horizon 2020 R&I project: • Phase III (2019-2022): 5G-SOLUTIONS, Media/ Entertainment use case	2021
	Feedback from internal and external stakeholders (user/customer) on business and social aspects related to 5G prototype/MVP presented at early innovation stage • Twelve internal and twenty-three external 5G stakeholders joined two interactive user/customer trial workshops and online feedback survey

A case study was performed to provide insight of the extent that business and social validation activities has been executed during the 5G PPP R&I projects. The study involved a selection of seven completed R&I projects among the portfolio of roughly fifty 5G PPP projects funded by EC, also referred to as strategic selection [58]. Data sources were project web sites connected to the three Horizon 2020 calls/phases (5G PPP Phase 1, Phase 2 and Phase 3.3 Vertical trial project) containing public deliverables, publications/scientific articles as well as dissemination reports. Other newsletters and web links associated to the single project web sites were also studied. The CORDIS database for EC funded projects (https://cordis.europa.eu/) were also studied regarding project information such as timeline, portfolio of consortium partners and deliverables. White papers developed on subjects such as trials, test and experimentation, stakeholders and business validation were also applied.

In the case study of the ongoing 5G-SOLUTIONS, the adjustment and use of agile methodology has been a part of the project deliverables and the way of work in recurring test cycles. Although we regard this as business validation pioneer work in the context of 5G R&I projects, we here foremost have elaborated on its application in combination with a social validation method. In this respect, two online workshops and pilots were performed with the purpose of testing 5G-SOLUTIONS’ combined business and social acceptance method during the pre-defined test cycles of the 5G-SOLUTIONS project. This also included a pretest of the business and social acceptance survey questionnaire. Both workshops were performed during 2021 (June and November), the first with participants from the 5G-SOLUTIONS consortium (internal) and the second with participants recruited outside (M&E vertical) the consortium (external). Information about project purpose, timeline, partnerships etc. were gathered from website (https://www.5gsolutionsproject.eu/). Twelve persons, equally representing different value chain users/customers, i.e telecom operator/MNO, service/application provider and end users, participated in the internal workshop. Twenty-three persons affiliated with the M&E value chain, 5G fora and 5G-SOLUTIONS joined the external workshop. The external participants were recruited through invitation based on their connection to the M&E industry or the 5G PPP community, hence a strategic selection [58]. All the workshop participants were confronted with three major issues/questions – how current 4G enabled solutions was a disadvantage to meet their demand, how the 5G prototype/MVP could mitigate these challenges, and the open issues still to be solved by the MVP. At this stage, the technological solution was immature, hence the need to use descriptions, sketches and videos simulating the functions and benefits from the prototype/MVP for this industry vertical. Feedback from the participants were collected using Mentimeter software. For all the statements in the survey questionnaire related to the technological performance, business and social impact of the MVP, the stakeholder was asked to respond to a 1-5 Likert-scale, i.e “Strongly disagree”, “Disagree”, “Neither agree nor Disagree”, “Agree” and “Strongly agree”. The workshop participants filled in the online questionnaire during the last section of the workshop.

3.2 Data Analysis

The analysis of the case study of the seven completed 5G R&I projects was based on secondary information, i.e the EU database Cordis.europa.eu and related documents from these 5G projects websites such as project deliverables, scientific articles, newsletters etc. After reading through the documents, we mapped which business and social acceptance related activities that were executed, and at which stage of the innovation process, using Coopers and Stills agile/lean stage-gate model, ref. Table 1. This innovation process framework involves four stages from idea to launch. To guide us with the mapping of the activities to these stages, we looked for keywords in the project documents, such as build-test-revise-learn philosophy, scenarios, persona’s method, SWOT analysis, business model or lean canvas, techno-economic analysis, business plan, user experience, trust, PESTEL analysis, accessibility, trust, health/safety, environment acceptance etc. In case on uncertainty, we mapped the project activities to the stage where the majority of the related keywords occurred. Applying multiple data sources, the way we have done in this case studies corresponds to the triangulation principle described both Yin [58] and Bryman & Bell [59]. Moreover, the pattern matching approach applied in the data analysis helped strengthen the internal validity of the case study Yin [58].

The analysis of the two digital workshop findings was based on data from the ongoing 5G R&I project 5G SOLUTIONS. The objective was to pilot a 5G prototype/MVP/protocept [33] from the M&E industry vertical. The data generated from these workshops was twofold – feedback on three major questions presented through a web based interactive tool (Mentimeter) and feedback on questions presented through an online survey tool (Questback). All questions were related to the virtual prototype/MVP presented for the workshop participants. The Mentimeter data was analysed and mapped according to our three main validation aspects or dimensions: technological performance, business value and social acceptance. The most frequently mentioned keyword within each if the dimensions were bolded in the table presenting the findings. For the analysis of the survey data, we generated an average score for all the 18 different dimensions that was put forward as statements. The participants responded to the statements by scoring according to a 1–5 Likert scale on the degree these dimensions were applicable for the 5G M&E use case solution. The 18 dimensions were equally distributed into our three groups: technological performance, business implications and social acceptance. All the 18 dimensions were presented in a histogram, see Figure 2. Based on the lessons learned from the online survey in the first workshop, we became aware of additional dimensions which were added in the second workshop survey. We also obtained feedback from the respondents in the open questions in the survey. These findings are presented at quotes throughout the article.

The total workflow with the separate user/customer pilot sequences in the two workshops are visualized in appendix A. We recognize the user pilot workshop as a single case study and being analogous with a single experiment method (how, why) of theory testing and development as well as having a replication [60]. In relation to the feedback form the survey research method was tested on the workshop participants reporting their average scores [61]. The stakeholders participating in the workshops are also here viewed as a strategic selection [58].

4 Empirical Findings

Here we present our research findings. We start with the review of how validation of business value and social acceptance has been executed in seven completed 5G R&I projects. Next, we introduce the findings from the workshops where we piloted a lean method-based trial and validation of a use case solution on an early stage in the ongoing 5G SOLUTIONS R&I project.

Table 3 Overview over 5G R&I projects business validation focus

	Research and Innovation Process Stages
5G R&I Projects	Stage 1	Stage 2	Stage 3	Stage 4
5G EXCHANGE			X	X
5G CHARISMA		X	X	X
5G SESAME			X
5G MONARCH			X
5G HEART	X		X
5G CITY			X	X
5G VINNI	X		X	X

4.1 5G R&I Projects Assessed

According to 5G PPP [16], business validation is the process to assess whether a certain system design is appropriate for the purpose and meets the business requirements within given constraints. Business requirements are the characteristics of the system from the end-users’ and customer’s point of view and expressed by the customers. Constraints are external factors that impact and limit system design and can be technical, financial, or regulatory of nature. Business validation can embrace everything from ideation, through development of robust value propositions, to estimating market sizes, designing business models, and carrying out business case analyses [16]. 5G PPP also proposes the lean methodology as a basis for the business validation process within research domain, since this may help the projects in commercialization of better products, while being faster in the market and with sustainable business models.

Overall, our study of seven selected 5G PPP projects, see Table 3, revealed that most of the business validation activities have been carried out in the two latter stages.

5G HEART and 5G VINNI reported activities using design thinking tools in the former stages, such as customer journey descriptions and storyboards analyzing personas’ user tasks and pain points [14]. With exception from the 5G CHARISMA project, the level of validation of user and social aspects of technology was difficult to observe early in the innovation process, ref. stage 1 and 2 in Table 3. The business model canvas was most often used in the latter validation stages to display ecosystem roles and stakeholders, the business opportunities and targeted market segment. In stage 3 we observed that all 5G PPP projects address, techno – economic analysis of cost, revenues, and market evolution estimations to some degree. The use of field trials of the 5G technological performance was also frequently executed in dedicated facilities located at the premises of the consortium members. However, methods for assessment of product concepts/MVPs by users/society early on in the validations process (stage 2) did not seem to be present. In stage four we see that 5G VINNI [62], 5G CITY [63], as well as 5G CHARISMA [64] and 5G EXCHANGE [65] all includes descriptions in business plans for launch of product solutions in dedicated market segments.

4.2 5G-SOLUTIONS Pioneering Case

5G-SOLUTIONS is a 5G R&I project which covered use cases within four industrial verticals living labs (LLs): factory of the future; smart energy; smart cities/port [66]; and media and entertainment. In addition, a group of use cases across (multi living lab) the four verticals were studied. The consortium consisted of 26 partners, of which 10 were SME’s. One work package was dedicated to requirements analysis, use cases and methodologies (business and technology validation). Four work packages were related to technological field trials in the different LL’s which occurs during three testing cycles. It was explicitly stated that a lean start-up methodology was to be applied for the business validation. The overall methodology chosen by 5G-SOLUTIONS for business validation is a four-stage process where the technological validation goes hand in hand with the business validation process [67]. The process is very much in line with the best-practice methods introduced in Table 1.

• The customer validation (stage 1) in this approach includes a description of the use case stakeholders (personas) involved in addition to a definition of the problems/pain points they are experiencing with current products/solutions.

• The solution alignment (stage 2) involves development and testing of 5G enabled MVPs developed from requirements by personas on needs and pain points.

• The business model (stage 3) involves creating a win-win business model for stakeholders and other partners for the most exploitable use cases from each industry vertical/LL.

• The growth trajectory (stage 4) includes business plans with detailed business models (product/service solution, market, and organization) and financial projections for the 5G use case solution offered in the market with indeterminate parameters, prices, costs, sales volumes etc.

Figure 1 Lean user pilot validation process method for 5G-SOLUTIONS use cases MVPs.

The technological validation of the prototypes/MVPs in 5G-SOLUTIONS was performed in relation to predefined trial cycles during the project. Trials for user/customer and business validation were proposed to be performed in parallel with the technological field trial adding feedback on the MVP from other stakeholders not directly involved in the technology field trials. This started already in stage 2 (Solution alignment) of the 5G-SOLUTIONS project. Figure 1 describes how 5G-SOLUTIONS adjusted a lean method to its purpose.

• Problem: Here the use case problem and personas/stakeholders are described: Description of as-is situation and current technology solutions in use by stakeholders was also included. Empathy mapping helps visualize types of customer/users on their attitudes and behaviors, needs and desires (who they are, what they need to do, what they see, hear, say and do, think and feel [68]. Customer journeys adds to the understanding of the interactions (touchpoints) the customer has with consortium partners [69].

• Prototype: Virtual prototypes/protocepts/MVPs [33] in the form of information material and videos are designed with the purpose to describe and visualize the 5G use case MVP, see appendix B and C. These protocepts are based on the mapping of the gains offered by the value proposition with the insight in the customer pains and needs from the empathy and journeys. When available, screen shots from technical field trials is added.

• Pilot/trial: Here business and consumer test pilots of the 5G MVP solution with relevant use case stakeholders are performed. For small sample pilots, physical or virtual focus groups/workshops with 20 $+$ stakeholder personas facilitated by a moderator are suggested. The suggested sequence of steps in the workshop/pilot are the following, see appendix A: After introducing the use case subject (1), the participants are asked to provide feedback on current problems and limitations and how they are solved today (2). Next, the 5G MVP concept developed are presented (3) using text, illustrations, and videos, asking for feedbacks on its benefits with respect to technological performance, business value social acceptance (4). Based on the feedback a discussion among the participants is moderated (5 and 6) and open issues are put forward and discussed (7). Finally, an online survey ranks the dimensions individually and among each other (8).

• Pivot: The findings from small and large sample pilot tests were aligned with results from technology validation field tests. The use case owners then could decide on whether they should move ahead with the MVP solution (“persevere”) or modify the solution design and specifications (“pivot”). If the decision was to pivot, it would not be necessary to return to start if the problem and pain points are understood. Then the project team proceeds to redesigning the prototype, hence the dotted arrow.

In Table 4 the results from the two workshops (step 5, 6 and 7) are presented. The bold key words are the most frequent words mentioned by the internal and external participants with respect to current pain points, potential 5G benefits and open issues going forward.

Table 4 Response from workshop stakeholdes on Mentimeter questions

Mentimeter Questions	Technological Performance	Business Value	Social Acceptance
Why is current 4G a limitation for commercialization and consumption of M&E services?	High latency Low speed Low bandwidth elasticity Low predictability High energy consumption (Internal)	High production cost Lack of content/service availability (Internal)	Low user experience Security (Internal)
	Low bandwidth Buffering Lack of 4K media distribution Low stream synchronization (External)	High cost Lack of high-quality value-added content (External)	No QoS guarantee/User experience No stable performance Security (External)
How can 5G assist in the offering and consumption of M&E services?	Speed Network slicing Reliability Support service providers, App – Network interaction Expectations alignment (Internal)	Expand opportunities Reduced production cost Variety of services (Internal)	Privacy Security Increased user experience On demand service High quality video (Internal)
	Lower latency Increased coverage More devices supported Edge computing Smarter deployments (External)	Live events Cost reduction More applications Immersive and interactive gaming (External)	Better QoS/User experience Stable connections Several channels (External)
What are the open issues to be solved?	Coverage for roaming Slicing implementation Energy efficiency Lack of skills Edge computing NetApp’s availability (Internal)	Cost vs. benefit Business and cost model User reliability Value chain Fear of change (Internal)	Cost of service for user User experience Privacy (Internal)
	Ease of implementation Modular capabilities Access to API’s European infrastructure (External)	Business models Ecosystem Supply chain Clear slicing offering (External)	Security Guaranteed service level agreement Cost service (user) (External)

Figure 2 Response from online workshop participants on the validation of the technological performance, business value and social acceptance of the M&E use case prototype presented.

Grouping the findings, we identified the business value and social acceptance dimensions in addition to technological performance dimensions reported by the 5G M&E internal and external vertical stakeholders. Hence, also the tension between the different aspects became clearer. Regarding the current 4G limitations, cost, lack of content/availability of services, low user experience and QoS guarantees, and security were the most relevant shortcomings, hence expected 5G benefits related to business value and user/social acceptance factors. Regarding open issues necessary to be solved, cost of service, the user experience and whether the technology solved privacy issues was related to the user and society side.

The online feedback (step 8) supported and detailed the findings from the Mentimeter questions. Here six statements related to the three dimensions had been presented in the survey to the same stakeholders: performance (technology), business as well as social acceptance of use case prototype/MVP. On average, the significance of technology scored highest on technological performance, then business value and then social acceptance t, see Figure 2. Still, the differences were low. It is fair to say that the emphasis on societal aspects signifies its importance in the lean innovation processes.

User experience scored highest among the social acceptance sub dimensions, followed by increased safety and equal opportunity and inclusiveness, see Figure 2. New business models/ecosystem and new revenue opportunities followed by reduced time to market scored highest with respect to the business value items/sub dimension. Regarding the technological performance dimension, increased network capacity, slicing and data rate were reviewed as most important benefits.

All in all, the workshops were successful as a tool for validation of 5G use case prototypes. According to the online feedback form, 100% of the externals were either satisfied or highly satisfied with the workshop. Positive comments to the scores were also added, such as “Well run and informative” and “the workshop offered a lively, interactive discussion with multiple stakeholders from various domains. It kept the interest high and discussion with participants triggered our thoughts”.

5 Discussion

The research question in this article is how lean start-up methodology can support validation of use 5G case solutions in early stages of R&I projects. The rationale behind studying the next generation mobile technology 5G is the significant influence it will have on business opportunities in many industry verticals and as well as for the society and end users. It is also expected to disrupt previous generations and through novel features and capabilities. The literature review points out the importance of the front-end stages of innovation projects, whether the methodology applied is based on sequential or iterative principles [15, 32]. Furthermore, previous research advocate the importance of providing customer insight throughout the early stages of the innovation process [26, 56, 57], but also during the following test and development stages of the innovation process [11, 26].

Our case study of a selection of completed 5G R&I projects revealed that most of the business validation activities were not performed in the front end of the projects, but in the latter stages of the innovation projects [16]. Moreover, we learned that the validation of the social acceptance of the technology were not included in these 5G R&I projects. Based on this, we wanted to pilot the use of the lean methodology on a 5G R&I project involving trials at early stage of the project. Here we also wanted to include the social acceptance dimension in the overall validation of use case prototypes/MVPs in 5G R&I projects. The argument for introducing the social acceptance in addition to the economy/business and technology dimensions, were based on the skepticism and resistance from the population and opinion on the potential negative effects from the next generation mobile network technology 5G [5]. Hence, it becomes vital for the stakeholders to measure the degree of social acceptance and manage the influence this will have on the adoption and success of forthcoming 5G product and service innovations by citizens/end users/public agencies.

5G SOLUTIONS was the 5G R&I project we decided to pilot our methodology on. This project consisted of a consortium of 26 ecosystem partners, including SMEs, and covering 20 different use cases within four living labs or industrial verticals. The modified lean start-up methodology presented in Figure 1, put efforts on the test/trial stage and introduces a toolbox for the presentation and assessment of the 5G use case MVPs especially on the business and social aspects [54, 70]. The emphasis on combining technological and non-technological trial aspects follows up EC’s emphasis on piloting and experimentations in relation to 5G PPP projects (p.8) “A number of sectors recognize the need to run pilot trials to increase predictability, reduce investment risks, and validate both technologies and the business models. Experiments are also needed to provide input for the standardization organizations” [71]. However, the EU statement does not explicitly include the social acceptance dimension.

The workshop/trial toolbox included virtual prototypes/MVPs, videos or PowerPoint presentations [72], describing the proposed use case solution in terms of customer value, how it worked, how it might affect the customer work-life etc. [73]. Here both project internal and external stakeholders were involved [32, 74]. Moreover, Mentimeter and Questback software tools where use to collect and analyze the participants feedback and discussion. When external stakeholders were involved in this way, this type of beta testing/pilots supports a more realistic level of simulation and actual circumstances compared to involving employees (internal stakeholders) only [26, 30]. Hence, we avoided that the use case owner ends up with pilots where the technology is paramount to validate [26, 75]. Moreover, by introducing a systematic approach to validating social acceptance of use case solutions we align with the recent research pointing out the subdimensions beyond the single focus on users’ behavior and usability of use case solutions [46]. Here our findings add insight into the previous findings of the 5G CHARISMA project [76] through the advocacy of privacy, security, equal inclusiveness, and democratization being vital sub dimensions for the success of 5G adoption of 5G in the M&E domain.

The 5G-SOLUTIONS model also adds domain specific context to the social acceptance dimension beyond the generic 1–9 levels presented by Vik et al. [14]. Moreover, the emphasize on involving external stakeholders also proved to mitigate barriers for practicing lean start up methods in joint R&I projects where corporations are involved [9], by increasing the focus on the joint R&I project rather than disclosing company internal product information. Our method also provides additional value to Coopers agile stage-gate hybrid method [29] including new qualitative go-kill indicators related to social acceptance (health, data security, environmental footprint, gender-equality opportunities, accessibility etc.). Moreover, our 5G-SOLUTIONS enabled 5G use cases also adds a disruptive innovation context to these indicators which is less discussed in Cooper’s models [28, 29].

Looking at the feedback from the internal and external participants in the two user pilot tests, it revealed the existence of contradictory and open issues that needs to be coped with from a technological (network slicing, edge computing, orchestration) and business/user point of view. For instance, cost of service and user experience for customer is necessary to handle going forward. Not the least, the tensions coming from the important societal concerns became clear. This customer and technology fuzziness [15] are typical for the early innovation stages and is a major reason for failure [1], and can also slow adoption [7]. In line with our findings that the early phases in lean innovation processes are important to address and iterate, it is also better to identify such tensions early.

Table 5 suggests dimensions that can be used for validating innovations in the 5G context which mitigate the research gap observed in the literature. It introduces the social acceptance dimension which lacks in the bottom-up validation of MVPs in current lean methodology [24], and the triple helix top-down approach model [21]. The dimensions contain six items/sub dimensions with mean scores. Weights per dimension were applied in the online Questback survey questionnaire.

Table 5 Overall judgement of technology, business and social acceptance dimensions

Dimension	Technological Performance	Business Impact	User and Social Impact
Weight	36%	32%	32%
Item/Sub dimension	Lower latency	Revenue increase	Increased user experience
(1–5 score)	Higher data rate/speed	Cost reduction	Improved health and safety
	Increased reliability	From CAPEX to OPEX	Improved trust and privacy
	Increased coverage	Time saved/time to market	Improved physical environment
	Increased network capacity	Increased complexity and uncertainty	Equal opportunities and inclusiveness
	Improved network slicing	New business models and ecosystems	Improved social justice and ethics
Use case score	Tot. mean score/6*Weight $=$ Technology score	Tot. mean score/6*Weight $=$ Business score	Tot. mean score/6*Weight $=$ User/Social score
	(Technology score $+$ Business score $+$ User/Social score)/3 $=$ Use case score

The determinants (complexity, new business models etc.) applied in Table 5 are based on the input from literature review, the first of the two 5G M&E workshop and findings from study of completed 5G R&I projects. From a technological perspective the literature showed that 5G will introduce lower latency, higher capacity, increased bandwidth and networks slicing (separata logic networks on same physical network infrastructure) compared to current 4G technology [2]. This will support the product/service performance and business opportunities for many industry verticals. For the development and production of media product/service from event venues (concerts, festivals, sport etc.) CAPEX savings are expected from less need for production equipment (e.g TV production vans) [54]. Moreover, there will be a transition to increased OPEX spending, although some savings are also expected such as less workforce on-site as well as time savings from less travels to events location. Streaming HiFi video from live events to end user smartphones also increase user experience having close to live experience in real time on their devices as if they were at event venue. Business value determinants often referred to are product (performance, cost etc.), process (time to market, project teamwork etc.) and financial outcome (profit, market share) [15]. Profit, pay-back time, and market share [32] and more qualitative go-kill criteria such as strategic fit, competitive advantage, and market potential [33]. According to Pinto and Slevin [34], client success (satisfaction, use and effectiveness) was necessary to be separated from project success (time, cost, performance). Upham et al. [35] and Wüstenhagen et al. [36] focus on three actor groups when validating the social acceptance of the innovation’s: public, stakeholders and individuals/end users, where the latter is shown to be focused in recent studies [37]. Consumers environmental awareness, knowledge, and health-consciousness attitude is shown to influence on the intentions to purchase such 5G services [38]. Moreover, perceived data privacy and security associated with the technology was found to shape trust, attitude, and behavioral intention to use connected vehicles innovations [51]. Other studies suggest that trust has an imperative impact on the adoption of novel technology based innovations [40–43] as wells as for 5G technology enables innovations [44].

The calculated mean score results per dimension in Table 5 enables the projects to identify the gap (technology vs. business vs. social acceptance) as compared to expected KPIs. This can in turn be used to propose actions and revisions (redesign) of the MVP solution – improving the gap (pivoting) before iterating a new testing cycle. Proposed actions should preferably be based on scoring tables as suggested, and collective decision making involving across functional team members of the business organization.

Based on the online survey feedback from the 5G SOLUTIONS stakeholders technological, business and user/social acceptance dimensions of the 5G use case solutions should have close to equal weight. The results from the workshop trials show the following ranking: Technological performance: 36%, Business value: 32% and Social Acceptance: 32%. These results indicate the need for a more holistic approach to validation of 5G use case innovations than we find today. Such an approach should also cover innovations in general when they are expected to embrace the majority of public and industry sector as well as individuals. The importance of the social acceptance dimensions is in line with the previous research findings on similar novel technologies with major impact for the individuals and the society, such as energy/windmills [35, 36], autonomous cars and vehicles [50, 51], mobile banking [42], social robots [52], and cloud storage [40].

This consequence of a holistic approach can be additional tensions when the case owners shall decide on whether to preserve, i.e move ahead with the existing MVP or pivot (modify the design and specifications of the MVP) during the next development cycles in the different stages of the innovation project. In the latter case it would be necessary to return to start and align with the problem and pain points previously collected. If there is a mismatch, the project team proceeds to redesigning the prototype, hence the dotted arrow in Figure 1. According to Cooper [25] and Ries [24], innovation project often ends up doing unnecessary redesign due to late feedback on prototypes/MPVs from stakeholders/actor groups external to the project. We believe, that applying our lean methodology in early stages of R&I projects, as demonstrated in the 5G SOLUTIONS project, successfully can support ecosystem decision makers in on whether to move head or redesign the prototype/MVP in an easier, transparent and more holistic way than current practice in the EC funded 5G R&I projects. However, we expect that the portfolio of dimensions within the economy/business and social acceptance dimensions, will be expanded going forward. Through similar lean start-up-based trials in other vertical use case solutions, new domain specific dimensions will appear and give additional insight for the validation of prototypes and product/service concepts. See more in conclusions chapter.

6 Conclusions

Validation of innovative product and services during the process of development is challenging, but necessary for a successful launch and commercialization result.

Traditionally, 5G R&I projects funded by EC have focused on technological performance when validating early versions of use case product and service solutions. Although this validation has occurred from the initial innovative stages, the validation of business value has been postponed to the later stages. Including validation of social acceptance of the use case solutions has been absent. This is revealed through literature review and our analysis of a selected group of completed 5G R&I projects from the Horizon 2020 program.

The main findings and message from this article are that we successfully performed a trial early in an 5G R&I project applying the lean start-up methodology. This methodology is novel for these types of projects with multiple stakeholders involved, and to our knowledge this has not been presented in previous research publications. The continuous development and testing of use case solution throughout the project stages, reduces the risk of failure and improves success for disruptive innovations such as the 5G. Our findings show that visualizing an early version of a 5G use case solution, in the form of information material and videos, helps the stakeholders and end users performing their validation at the initial stages of the innovation process. Moreover, our findings reveal that validation during these stages should include equal focus on business and social dimensions as well as the technological ones. Our findings are applicable for innovative 5G use case solutions within the M&E vertical, but we expect them also to be valid for other disruptive innovations within other industry verticals such as e-Health, Smart cites, Industry 4.0, etc. where the technology is expected to be embraced throughout the society. Based on this, we recommend the use of a lean start-up-based method along with simultaneous validation of business value, social acceptance, and technology performance, in 5G R&I projects funded through the forthcoming Horizon Europe program This includes infrastructure projects as well as advanced solutions in vertical industries. Beyond the focus on validation at the initial innovation stage, we recommend lean-start-up tools to be applied by the start-ups, corporations, and other ecosystems partners also in the later commercialization (business model and growth trajectory) stages of the innovation process.

Further research should include online pilots of the lean start-up method involving a larger sample of stakeholders (200 $+$ ) for the M&E use case solution. Moreover, the lean method and pilots should include other 5G R&I projects to confirm the M&E results and mitigate the possibilities of the selected sample being skewed. This should also include studies of how the lean method is being applied and modified and refined to better fit the project context by the R&I project consortium partners.

Acknowledgement

This project has received funding from the European Union’s Horizon 2020 research and innovation program under Grant Agreement No. 856691.

Appendix

A: Workshops Sequences Including Trial of M&E Use Case Prototype

B: Virtual Presentation of M&E Use Case Prototype (Video M&E) https://5gsolutionsproject.eu/living-labs/media-entertainment/

C: Virtual Presentation of M&E Use Case Prototype (Info. text M&E) https://5gsolutionsproject.eu/living-labs/media-entertainment/

References

[1] CB Insights, 2020. [Online]. Available: https://www.cbinsights.com/research/startup-failure-reasons-top/.

[2] 5G PPP, “5G Empowering vertical industries,” 2020b. [Online]. Available: https://5g-ppp.eu/wp-content/uploads/2016/02/BROCHURE\_5PPP\_BAT2\_PL.pdf.

[3] IHS Markit, “The 5G Economy – How 5G will contribute to the global economy,” 2019. [Online]. Available: https://www.qualcomm.com/media/documents/files/ihs-5g-economic-impact-study-2019.pdf.

[4] European Commision, “5G,” 2020. [Online]. Available: https://ec.europa.eu/digital-single-market/en/towards-5g.

[5] “Stop 5G Global Protest Day,” [Online]. Available: https://stop5ginternational.org/.

[6] “Stop 5G UK – 5G Resources and Response UK & Ireland,” [Online]. Available: https://stop5g.co.uk/about-us/\#. [Accessed 2021].

[7] G. Moore, Chrossing the chasm: Marketing and Selling Disruptive Products to Mainstream Customers, 3rd Edition, HarperCollins, 2014.

[8] R. Bortolini, M. Cortimiglia, A. Danilevicz and A. Ghezzi, “Lean Startup: a comprehensive historical review,” Management Decision, vol. 59, no. 8, pp. 1765–1783, 2019.

[9] J. Karlson and J. Nordström, “Implementing Lean Startup Methodology in Large Firms-Benefits, Challenges and Solutions,” Chalmers Univeristy of Technology, Sweden, 2012.

[10] M. Docherty, “Adapting Lean Startup Principles for Corporate Innovation,” 2016. [Online]. Available: https://www.linkedin.com/pulse/adapting-lean-startup-principles-corporate-innovation-docherty.

[11] E. Ries, The Lean Startup- How constant innovation creates radically sussessful businesses, Penguin Books Ltd, 2011.

[12] K. Still, “Accelerating Research Innovation by Adopting the Lean Startup Paradigm,” Technology Innovation Management Review, vol. 7, no. 5, 2017.

[13] 5G PPP, 2020. [Online]. Available: https://5g-ppp.eu/wp-content/uploads/2017/03/5GPPP-brochure-final-web-MWC.pdf.

[14] J. M. A. S. E. Vik and R. Søraa, “Balanced readiness level assessment (BRLa): A tool for exploring new and emerging technologies,” Technological Forecasting and Social Change, pp. 1–9, 2021.

[15] Q. Zhang and W. Doll, “The fuzzy front end and success of new product development : A causal model,” European Journal of Innovation Management, vol. 4, no. 2, pp. 95–112, 2001.

[16] 5G PPP, “Business Validation in 5G PPP vertical use cases,” 2020d. [Online]. Available: https://5g-ppp.eu/wp-content/uploads/2020/06/5G\_White\_paper\_Business-validation-v1.0a.pdf.

[17] E. Larson and R. Larson, “Use cases: What every project manager should know,” in Larson, E. & Larson, R. (2004). Use cases: what every PMI Global Congress, Anaheim, CA, USA, 2004.

[18] Technopedia, “What Does Use Case Mean?,” 23 9 2014. [Online]. Available: https://www.techopedia.com/definition/25813/use-case. [Accessed 2022].

[19] R. Adner, “Match your innovation strategy to your innovation ecosystem,” Harvard Business Review, vol. 84, pp. 98–107 $+$ 148, 2006..

[20] H. Etzkowitz and L. Leydesdorff, “The dynamics of innovation: from National Systems and “Mode 2” to a Triple Helix of university – industry-government relations,” Research Policy, vol. 29, pp. 109–123, 2000.

[21] M. McAdam and K. and Debackere, “Beyond ‘triple helix’ toward ‘quadruple helix’ models in regional innovation systems: Implications for theory and practice,” R&D Management 48, 1, 2018, pp. 3–6, 2018.

[22] H. Chesbrough, “How Smaller Companies can benefit from open innovation,” Japan Spotlight, Vols. January-February, pp. 13–15, 2010.

[23] O. Granstrand and M. Holgersson, “Innovation ecosystems: A conceptual review and a new definition,” Technovation, pp. 1–12, 2020.

[24] E. Ries, The Startup Way: How Modern Companies Use Entrepreneurial Management to Transform Culture and Drive Long-Term Growth, Currency, 2017.

[25] R. Cooper, “Idea-to-Lauch Gating Systems: Better, Faster, and More Agile,” Research-Technology Management, vol. 60, no. 1, pp. 48–52, 2017.

[26] A. Edmondson, “Strategies from learning from failure,” Harvard Business Review, vol. 89, no. 4, pp. 48–55, 2011.

[27] S. Magistretti, C. Dell´Era, R. Verganti and M. Bianchi, “The contribution of Design Thinking to the R of the R&D in technological innovation,” R&D Management, vol. 52, no. 1, pp. 108–125, 2022.

[28] R. Cooper, “Perspective: The Stage-Gates Idea-to-Launch Process – Update, Whats new and Next Gen systems,” Journal of Product Innovation Management, vol. 25, pp. 213–232, 2008.

[29] R. Cooper, “Agile-Stage-Gate Hybrids: The Next Stage for Product Development,” Research-Technology Management, vol. 59, no. 1, pp. 21–29, 2016.

[30] A. Nguyen-Duc, K. Khalid, S. S. Bajwa and T. Lønnestad, “Minimum Viable Products for Internet of things Applications: Pitfalls and Practices,” Future Internet, vol. 11, no. 50, 2019.

[31] A. Osterwalder, Y. Pigneur, G. Bernarda and A. Smith, Value Proposition Design: How to Create Products and Services Customers Want, John Wiley & Sons, 2014.

[32] R. Cooper, Winning at New Products – accelerating the process from idea to launch, Perseus Publishing, 2001.

[33] R. Cooper, “What’s Next?: After Stage Gate,” Research-Technology Management, vol. 57, no. 1, pp. 20–31, 2014.

[34] J. K. Pinto and D. Slevin, “Critical success factors across the project life cycle,” Project Management Journal, vol. XIX, no. 3, pp. 67–74, 1988.

[35] P. Upham, C. Oltra and À. Boso, “Towards a cross-paradigmatic framework of the social acceptance of energy systems,” Energy Research & Socsial Science, vol. 8, pp. 100–112, 2015.

[36] R. Wüstenhagen, M. Wolsink and M. Bürera, “Social acceptance of renewable energy innovation: An introduction to the concept,” Energy Policy, vol. 35, no. 5, pp. 2683–2691, 2007.

[37] J. GaedeIan and H. Rowlands, “Visualizing social acceptance research,” Energy Researvh and Social Science: A bibliometric review of the social acceptance literature for energy technology and fuels, vol. 40, pp. 142–158, 2018.

[38] S. Shah, T. Zhongjun, S. A and X. Zhou, “Consumer’s intention to purchase 5G: Do environmental awareness, environmental knowledge and health consciousness attitude matter?,” Technology in Society, vol. 65, pp. 1–9, 2021.

[39] D. McKnight, M. Carter, J. Thatcher and P. Clay, “Trust in a specific technology: An investigation of its components and measures,” ACM Transactions on Management Information Systems, vol. 21, no. 12, pp. 1–25, 2011.

[40] M. H. Shuyuan, M. Ocasio-Velázquez and C. Booth, “Trust or consequences? Causal effects of perceived risk and subjective norms on cloud technology adoption,” Computers and Security, vol. 70, pp. 581–595, 2017.

[41] S. Tams, J. Thatcher and K. Craig, “How and why trust matters in post-adoptive usage: the mediating roles of internal and external self-efficacy,” Journal of Strategic Information Systems, vol. 27, pp. 170–190, 2018.

[42] A. Alalwan, Y. Dwivedi and N. Rana, “Factors influencing adoption of mobile banking by Jordanian bank customers: extending UTAUT2 with trust,” International Journal of Information Management, vol. 37, no. 3, pp. 99–110, 2017.

[43] T. Oliveira, M. Alhinho, R. Rita and G. Dhillon, “Modelling and testing consumer trust dimensions in e-commerce,” Computers in Human Behaviour, vol. 71, pp. 153–164, 2017.

[44] M. Akbari, A. Rezvani, E. Shahriari, M. A. Zúñiga and H. Pouladian, “Acceptance of 5G technology: Mediation role of Trust and Concentration,” Journal of Engineering and Technology Management, vol. 57, pp. 1–14, 2020.

[45] J. Nielsen, Usability engineering, Boston: Morgan Kaufmann, 1993.

[46] V. Venkatesh, M. Morris, G. Davis and F. Davis, “User Acceptance of Information Technology: Toward a Unified View,” MIS Quarterly, vol. 27, no. 3, pp. 425–478, 2003.

[47] H.-C. Kim, “Acceptability engineering: the study of user acceptance of innovative technologies,” Journal of Applied Research and Technology, vol. 13, pp. 230–237, 2015.

[48] A. Koelle, S. Ananthanarayan and S. Boill, “Social acceptance in HCI: A survey of Methods, Meadures and Design Strategies,” in CHI 20, Honolulu, USA, 2020.

[49] I. Georgieva, E. Beaunoyer and M. Guitton, “Ensuring social acceptance of technological tracking in the Covid-19 context,” Computers in Human Behavious, pp. 1–8, 2021.

[50] K. Yuen, Y. Wong, F. Ma and X. Wang, “The determinants of public acceptance of autonomous vehicles: An innovation diffusion perspective,” Journal of Cleaner Production, vol. 270, pp. 1–13, 2020.

[51] A. Sailesh and M. Mekker, “Public acceptance of connected vehicles: An extension of the technology acceptance model,” Transportation Research Part F: Traffic Psychology and Behaviour, vol. 88, pp. 54–68, 2022.

[52] U. Saari, A. Tossavainen, K. Kaipainen and S. Mäkinen, “Exploring factors influencing the acceptance of social robots among early adopters and mass market representatives,” Robotics and Autonomous Systems, vol. 152, pp. 1–14, 2022.

[53] V. Venkatesh, M. Morris, G. Davis and F. Davis, “User acceptance of information technology: Toward a unified view,” MIS Quarterly, pp. 425–478, 2003.

[54] L. Briguglio, P. Nesse, A. Di Giglio, C. Occhipinti, P. Durkin and I. Markopoulos, “Business Value and Social Acceptance for the Validation of 5G Technology,” in International Mediterranean Conference on Communications and Networking, Athens, 2021.

[55] M. Stanko and J. Bonner, “Projective customer experience: Projecting future customer needs that drive innovation performance,” Industrial Marketing Management, vol. 42, no. 8, pp. 1255–1265, 2013.

[56] B. Yoon and A. Jetter, “Connecting customers with engineers for the successful fuzzy front end: Requirement of tools,” in Portland International Conference of Management of Engineering and Technpology, PICMET, Portland, 2015.

[57] F. Schweitzer, M. Palmie and O. Gassmann, “Beyond listening: the distinct effects of proactive versus responsive customer orientation on the reduction of uncertainties at the fuzzy front end of innovation,” R&D Management, vol. 48, no. 5, pp. 534–550, 2016.

[58] R. Yin, Case study research : Design and methods, Beverly Hills: Sage publications, 2014.

[59] A. Bryman and B. Bell, Business Research Methods, 3rd Ed., Oxford: Oxford Univeristy Press, 2011.

[60] G. Lloyd-Jones, “Design and Control Issues in Qualitative Case Study Research,” International Journal of Qualitative Methods 2003, 2(2), vol. 2, no. 2, pp. 33–42, 2003.

[61] R. Yin, Case study research: Design and methods, 2nd ed., Thousand Oaks, California: Sage, 1994.

[62] 5G VINNI, “Deliverable D5.2 Business requirements and fundamentals of the 5G-VINNI Business Layer,” 2020. [Online]. Available: https://zenodo.org/record/3460445\#.XsJTnGgzY2x.

[63] 5G CITY, “Enabling 5G Neutral Hosts: 5G city architecture and business model,” 2020. [Online]. Available: https://www.5gcity.eu/wp-content/uploads/2020/05/WhitePaper\_Enabling-5G-Neutral-Hosts-5GCity-Architecture-and-Business-Model-v1.0.pdf.

[64] 5G CHARISMA, “Deliverable 5.2 Business model analysis report,” 2015. [Online]. Available: https://www.charisma5g.eu/wp-content/uploads/2015/08/CHARISMA-D5.5-Business-model-analysis-report.pdf.

[65] Cordis EU, “5G Exchange Deliverable 2.3 Business and Economic Layer,” 2018. [Online]. Available: file:///C:/Users/ttpero/AppData/Local/Temp/Attachment\_0-3.pdf.

[66] Eurescom, “Co-creation of smart sustainable cities – the 5G SOLUTIONS project,” 2020. [Online]. Available: https://www.eurescom.eu/fileadmin/documents/message/Eurescom-message-Winter-2020.pdf.

[67] 5G-SOLUTIONS, “D1.4A Methodologies for the validation of 5G,” 2019. [Online]. Available: file:///C:/Users/ttpero/Downloads/Attachment\_0\%20(2).pdf.

[68] D. Gray, “Updated Empathy Map Canvas,” 2017. [Online]. Available: https://medium.com/the-xplane-collection/updated-empathy-map-canvas-46df22df3c8a.

[69] A. Følstad and K. Kvale, “Customer journeys: a systematic literature review,” Journal of Service Theory and Practice, vol. 28, no. 2, pp. 196–227, 2018.

[70] Tangari, E., Occhipinti, C. & Briguglio, L., “Close-The-Loop Model: Social Acceptance of Technology for Sustainability,” 2020.

[71] European Commision, “ Communication from the Commision to the European Parlament, The Council, The European Council, The European Economic ans Social Commitee and the Committe of the Regions 5G for Europe: An Action Plan,” Brussels, 2016.

[72] N. Furr and P. Ahlstrom, Nail it Then Scale it: The Entrepreneur’s Guide to Creating and Managing Breaththrough Innovations, Nathan Furr and Paul Ahlstrom, 2011.

[73] E. Rasmussen and S. Tanev, “Lean start-up: Making the start-up more successful,” in Start-Up Creation – The Smart Eco-Efficient Built Environment, 2016, pp. 39–56.

[74] H. Mason and T. Rohner, The venturing imperative, Boston: Harvard Business Press, 2002.

[75] Mc Kinsey & Company, “Industry’s fast-mover advantage: Enterprise value from digital factories,” 2020. [Online]. Available: https://www.mckinsey.com/business-functions/operations/our-insights/industrys-fast-mover-advantage-enterprise-value-from-digital-factories.

[76] I. Neokosmidis, R. Theodoros, M. Parker, G. Koczian, S. Walker, S. Muhammad and E. Escalona, “Assessment of socio-techno-economic factors affecting the market adoption and evolution of 5G networks: Evidence from the 5G-PPP CHARISMA project,” Telematics and Informatics, pp. 572–589, 2017.

[77] 5GPPP, “The 5G Infrastructure Public Private Partnership,” 2020h. [Online]. Available: https://5g-ppp.eu/.

Biographies

Per J. Nesse is senior research scientist in Telenor Research and Innovation. He is a specialist in service innovation management, entrepreneurship and financial analysis and validation. He has extensive experience from managing research and innovation projects in EU research programs with international enterprises and universities. Nesse is also associate professor at Norwegian University of Science and Technology (NTNU), Industrial Economics and Technology Management department. Nesse holds a Ph.D from NTNU and a Master of Science in Business from Nord University in Norway. Dr. Nesse has published scientific articles in journals such as International Journal of Entrepreneurial Venturing, Journal of Information Technology and Management and European Journal of Operational Research and Nordic and Baltic Journal of Information and Communication Technologies.

Hanne Kristine Hallingby is a senior research scientist at Telenor Research and Innovation, Next Generation Network Technology and Cloud. She holds a PhD in Innovation studies from University of Oslo, Centre for Technology, Innovation and Culture, a Master of Business and Economics and Master of Science in International Marketing and Strategy from BI Norwegian business school. She started in Telenor in 1990 and has extensive experience as a researcher and project manager in Telenor operations. Her interests are technology change, innovation, ecosystems, and market dynamics in the telecom sector, as well as user aspects of telecommunication. She has published and is a reviewer in journals such as Technology Analysis & Strategic Management, Telematics and Informatics, and IEEE Communications Magazine.

Luigi Briguglio holds a master’s degree in Electronic Engineering (1996) from the University of Padua. He is Senior Researcher and ICT Consultant at CyberEthics Lab, where he applies his expertise as a Project Manager in ICT and Embedded Systems research projects. In July 2015 he got a Project Management Professional (PMP) certification. He is a member of the PMI-Central Italy Membership Committee, as well as member of ISC2. His topics of interests are software architecture, data management, cybersecurity, social acceptability of new technologies, project management, embedded systems, monitor and control systems, and automation. He worked in the industry of Energy Conversion and RES Generation by developing monitor and control system based on embedded devices and custom electronics. He had research experience at INFM (National Institute of Matter and Physics) for developing a DSP-based controller for a near-field scanning optical microscope.

Ioannis Markopoulos is leading NOVA’s Innovation & Project Management Department. He holds a PhD in Telecommunications from the National Technical University of Athens, Electrical Engineering Department. He is a National Technical University of Athens, Electrical Department Graduate and a member of the Technical Chamber of Greece. Since 2007, he is also PMP certified. He has been working and leading numerous European and Domestic R&D projects in the domain of broadband telecommunications, value added services, management, etc. He has also produced and participated in several publications. He participates in the Federation of Hellenic ICT Enterprises working groups. His research interests include next generation networks, big data analytics, customer experience measurement methods, value added & OTT services, and customer loyalty & trust methods. He participates in 5G-PPP TMV WGs and led the task force for mapping the vertical KPIs to the 5G network KPIs.