Summary of the impact

While indexes exist that measure the maturity of the provision of eGovernment services from the government perspective (e.g. UN eGovernment Development Index, http://unpan3.un.org/egovkb/global_reports/12report.htm), there are no reliable standards that incorporate the citizen perspective into benchmarking of government effectiveness. Brunel research has included both government and citizen assessments and, through a more holistic approach to eGovernment evaluation, has helped Turkey and other governments to improve their e-government services.

EU funded CEES (Citizen-Oriented Evaluation of e-government Services) project delivered a new evaluation model, called COBRA (Cost, Opportunity, Benefit, Risk Analysis), for benchmarking e-government services from the citizens' perspective. CEES led to COBRA's adoption by Turksat, the Turkish central e-government service provider which has 12 million citizen users — leading to e-government service improvement and more favourable citizen attitudes. E-government service providers such as ictQATAR and OMSAR (Office of the Minister of State for Administrative Reform) have adopted the COBRA framework to evaluate and improve their e-government services in Qatar and Lebanon respectively. COBRA has also been used in UK and Estonia leading to similar outcomes.

This research outcome enabled the launch of a new project, called I-MEET, that is extending the COBRA framework to include governments' perspectives and is being applied to Qatar, Lebanon and the UK.

Underpinning research

The research was conducted as a part of the EU FP7 project CEES between April 2009 and March 2013. The CEES consortium consists of three partners namely Brunel University, American University of Beirut, and Turksat. Brunel Business School has the coordinator role of the consortium. Prof Zahir Irani, Dr Habin Lee (Reader), Dr Vishanth Weerakkody (Reader) and Dr Aggeliki Tsohou (Senior Research Fellow) contributed to this project.

The outcome of the project is a citizen-oriented e-government services evaluation model, called COBRA, and a reference process model that defines the steps with which COBRA is applied to e-government service evaluation in different national contexts. The COBRA and reference process model were verified through four field trials conducted in Turkey (June 2010, Jan 2011 for COBRA), the UK (October 2012 for the reference process model), and Lebanon (November 2012 for the reference process model).

Evaluation of e-government services (e-services) is a sociotechnical challenge involving and a large number of stakeholders with conflicting interests, which may limit citizens' satisfaction and e-services take-up. Each e-service achieves certain system performance levels in the various aspects of interest and each citizen has different online behavioural experiences. The project aims to understand those factors that affect citizen's satisfaction and proposes a novel analytic approach to measure satisfaction from an e-service. The approach integrates measurement factors derived from the literature on e-services and new measurement factors based on citizen's behaviour and experience during the online interaction.

For each factor, tangible and intangible variables were identified through the normative literature review and focus group interview by Brunel University and AUB in the first 6 months of the project in 2009. These variables were assembled within taxonomies of cost; benefit; risk and opportunity. These factors and variables were validated using focus groups of users, e-government experts and academics. Statistical analysis was conducted to establish proof of relationships among the identified variables, factors and users' satisfaction. The Data Envelopment Analysis (DEA) Operational Research approach was then employed to balance the trade-offs between the input (cost and risk variables) and the output (benefit and opportunity variables) to derive an aggregate measure of satisfaction. For the purpose of improving inefficiencies, DEA focuses on the best practice of efficient e-services. Unlike traditional evaluation approaches, DEA identifies inefficient e-services and the magnitude of these inefficiencies together with an analysis of sensitivities. Then improvement targets are set, based on best-practice rather than average performance.

Usage experiences were reported on a sample of 2785 Turkish users of e-services provided through the e-government portal at TurkSat during June 2010 - November 2010. The analytical results demonstrated how to implement the new DEA-based approach for the measurement of user's satisfaction with e-services. They also established a proof of validity for DEA as an alternative method to traditional methods that seldom identify and quantify the individual factors for dissatisfying e-services. Based on DEA results, managers were able to establish a national best-practice benchmark and e-service performance indices, track progress and measure impact of policies on e-service design and improvement over time.

References to the research

Grant Information.

- Grant Title: CEES: Citizen oriented Evaluation of E-government Services — Reference Process Model.

- Sponsor: EU FP7 People, Industry Academy Pathway Program.

- Grant ID: IAPP 230658

- The project home page is at http://www.iapp-cees.eu/ The period of grant is from 01 April 2009 to 31 Mar 2013 (4 year projects).

- The total grant is €374,155.

- Grant Title: I-MEET: Integrated Model for Evaluating E-government services Transformation.

- Sponsor: Qatar National Research Foundation.

- Grant ID: NPRP-09-1023-5-158

- The project home page is at http://i-meet-egov.net/index.html

- The period of grant is from 01 Nov 2010 to 31 Oct 2013 (3 year projects).

- The total grant is $143,303.

Publications

Details of the impact

CEES evaluation of 13 e-services available by the Turkish government's e-Government gateway had a measurable direct impact on service quality and on user satisfaction.

By assessing the relative satisfaction of citizens with e-government services through the COBRA framework and by setting targets for improvement in core strategic areas (e.g. citizen health related e-service), Turksat was able to rank services against those that performed well — identifying critical factors for consumer satisfaction. Turksat was then able to allocate resources to these areas and improve their e-government provision. Improvements included:

• increased capacity at the e-government gateway call centre
• more stable infrastructures leading to greater continuity of service
• introduction of mobile access
• personalisation of service through `My Page' portals
• improved accessibility

(CEES report, 2013).

This led to positive evaluations that included increased citizen satisfaction index measurements and certifications of excellence associated with service improvements. For example in 2010 the citizen satisfaction index was measured as 65.5, a figure that rose to 66.18 in 2011 (CEES report, 2013) after the start of the improvements. Further, TURKSAT has been identified as a centre of excellence due to the CEES project (ICT RTD Technological Audit Project Potential in Turkey report, 2012 page 48 table 9.8). This has enhanced the TURKSAT brand and national credibility.

TURKSAT has shared CEES knowledge and results with 27 municipal partners to more fully coordinate e-government services. These municipal partners now offer services on the e- government gateway.

The CEES project formed the basis for I-MEET funded by Qatar National Research Foundation which extended the COBRA framework to include government evaluations and which is being applied to Qatar, Lebanon and the UK. For the application of the extended COBRA framework, workshops were organised in each country in 2012:

• an e-government service providers' engagement workshop was held in Qatar on 19^th Jan 2012 where the e-government service providers (ictQATAR) followed a presentation of the Turkey case outlined above and decided to adopt the COBRA framework for evaluating their e-government services
• a similar workshop was held in Lebanon on 14^th April 2012 with representatives from the Office of the Minister of State for Administrative Reform (OMSAR) who also decided to adopt the COBRA framework for the improvement of their e-government services.
• the COBRA framework was presented to local authorities in the UK during Transformational Government Workshop held in London on 13^th May 2012.

An Estonian research team used the COBRA framework to evaluate their e-government services. This resulted in a postgraduate research thesis called "Quality Assessment of Estonian e-government Services" by Hannaes Lehemets. (Thesis, Institute of Computer Science, University of Tartu, 2012). The results of the research were communicated to the Estonian State Information System Office (RISO) — http://www.riso.ee/en/. Specifically, a copy of the thesis was given to the head of the IT Architecture Office of RISO (Uuno Vallner) who acknowledged the conclusions of the study.

Brunel research leading to the COBRA framework is highly significant in that no reliable benchmarking index, that included citizens' and government perspectives, was previously available to evaluate e-government services. The research is therefore a step towards the development of a holistic and hence more reliable benchmark. Its reach is global in that the development of COBRA in the Middle Eastern context means that it has universal applicability and can potentially be used all over the world.

Sources to corroborate the impact

1. I-MEET research grant contract and research plan (to support the claim that new project was funded based on the output of CEES project)
2. CEES Report: Citizen Oriented Evaluation of e-Government Services: A Reference process Model, February 2013 to corroborate improvement in e-government provision and citizen satisfaction
3. Professor of Software Engineering, University of Tartu, Estonia. The Professor can provide factual statements on the use of COBRA framework of CEES project in a Master Thesis of one of his students in Estonia.
4. CEES project deliverable D12 Exploitation Strategy written by Turksat (available on http://www.iapp-cees.eu Dissemination page) — to support the claim that COBRA helped Turksat improve their e-government services and citizen satisfaction
5. Assistant Secretary General-Information Technology Sector, ICT Government Programs, ictQatar, Qatar. This person can be contacted to support the claim that ictQatar adopted COBTRA framework for the evaluation of their eGov services.
6. Head of the E-Government Unit, OMSAR, Lebanon. This person can be contacted to support the claim that OMSAR adopted COBRA framework for the evaluation of e- government services in Lebanon.
7. http://stps.metu.edu.tr/ict-rtd-technological-audit-turkey ICT RTD Technological Audit Project Potential in Turkey report, 2012 page 48 table 9.8 to corroborate CEES as CoE
8. The thesis of Hannes Lehemets available at Brunel Business School.