An evaluation tool co-developed by Brunel University has helped Turkey, Qatar and Lebanon governments to improve their e-government services
Submitting InstitutionBrunel University
Unit of AssessmentBusiness and Management Studies
Summary Impact TypePolitical
Research Subject Area(s)
Information and Computing Sciences: Information Systems
Economics: Applied Economics
Summary of the impact
While indexes exist that measure the maturity of the provision of
eGovernment services from the government perspective (e.g. UN eGovernment
Development Index, http://unpan3.un.org/egovkb/global_reports/12report.htm),
there are no reliable standards that incorporate the citizen perspective
into benchmarking of government effectiveness. Brunel research has
included both government and citizen assessments and, through a more
holistic approach to eGovernment evaluation, has helped Turkey and other
governments to improve their e-government services.
EU funded CEES (Citizen-Oriented Evaluation of e-government Services)
project delivered a new evaluation model, called COBRA (Cost, Opportunity,
Benefit, Risk Analysis), for benchmarking e-government services from the
citizens' perspective. CEES led to COBRA's adoption by Turksat, the
Turkish central e-government service provider which has 12 million citizen
users — leading to e-government service improvement and more favourable
citizen attitudes. E-government service providers such as ictQATAR and
OMSAR (Office of the Minister of State for Administrative Reform) have
adopted the COBRA framework to evaluate and improve their e-government
services in Qatar and Lebanon respectively. COBRA has also been used in UK
and Estonia leading to similar outcomes.
This research outcome enabled the launch of a new project, called I-MEET,
that is extending the COBRA framework to include governments' perspectives
and is being applied to Qatar, Lebanon and the UK.
The research was conducted as a part of the EU FP7 project CEES between
April 2009 and March 2013. The CEES consortium consists of three partners
namely Brunel University, American University of Beirut, and Turksat.
Brunel Business School has the coordinator role of the consortium. Prof
Zahir Irani, Dr Habin Lee (Reader), Dr Vishanth Weerakkody (Reader) and Dr
Aggeliki Tsohou (Senior Research Fellow) contributed to this project.
The outcome of the project is a citizen-oriented e-government services
evaluation model, called COBRA, and a reference process model that defines
the steps with which COBRA is applied to e-government service evaluation
in different national contexts. The COBRA and reference process model were
verified through four field trials conducted in Turkey (June 2010, Jan
2011 for COBRA), the UK (October 2012 for the reference process model),
and Lebanon (November 2012 for the reference process model).
Evaluation of e-government services (e-services) is a sociotechnical
challenge involving and a large number of stakeholders with conflicting
interests, which may limit citizens' satisfaction and e-services take-up.
Each e-service achieves certain system performance levels in the various
aspects of interest and each citizen has different online behavioural
experiences. The project aims to understand those factors that affect
citizen's satisfaction and proposes a novel analytic approach to measure
satisfaction from an e-service. The approach integrates measurement
factors derived from the literature on e-services and new measurement
factors based on citizen's behaviour and experience during the online
For each factor, tangible and intangible variables were identified
through the normative literature review and focus group interview by
Brunel University and AUB in the first 6 months of the project in 2009.
These variables were assembled within taxonomies of cost; benefit; risk
and opportunity. These factors and variables were validated using focus
groups of users, e-government experts and academics. Statistical analysis
was conducted to establish proof of relationships among the identified
variables, factors and users' satisfaction. The Data Envelopment Analysis
(DEA) Operational Research approach was then employed to balance the
trade-offs between the input (cost and risk variables) and the output
(benefit and opportunity variables) to derive an aggregate measure of
satisfaction. For the purpose of improving inefficiencies, DEA focuses on
the best practice of efficient e-services. Unlike traditional evaluation
approaches, DEA identifies inefficient e-services and the magnitude of
these inefficiencies together with an analysis of sensitivities. Then
improvement targets are set, based on best-practice rather than average
Usage experiences were reported on a sample of 2785 Turkish users of
e-services provided through the e-government portal at TurkSat during June
2010 - November 2010. The analytical results demonstrated how to implement
the new DEA-based approach for the measurement of user's satisfaction with
e-services. They also established a proof of validity for DEA as an
alternative method to traditional methods that seldom identify and
quantify the individual factors for dissatisfying e-services. Based on DEA
results, managers were able to establish a national best-practice
benchmark and e-service performance indices, track progress and measure
impact of policies on e-service design and improvement over time.
References to the research
- Grant Title: CEES: Citizen oriented Evaluation of E-government Services
— Reference Process Model.
- Sponsor: EU FP7 People, Industry Academy Pathway Program.
- Grant ID: IAPP 230658
- The project home page is at http://www.iapp-cees.eu/
The period of grant is from 01 April 2009 to 31 Mar 2013 (4 year
- The total grant is €374,155.
- Grant Title: I-MEET: Integrated Model for Evaluating E-government
- Sponsor: Qatar National Research Foundation.
- Grant ID: NPRP-09-1023-5-158
- The project home page is at http://i-meet-egov.net/index.html
- The period of grant is from 01 Nov 2010 to 31 Oct 2013 (3 year
- The total grant is $143,303.
- Lee, H., Irani, Z., Osman, I., Balci, A., Ozkan, S., and Medeni, T.
(2008), Research Note: Toward a reference process model for citizen
oriented evaluation of e-government services. Transforming Government:
People, Process, and Policy, 2 (4), 297 - 310. 10.1108/17506160810917972
- Osman, I. H., Anouze, A. L., Irani, Z., Al-Ayoubi, A., Lee, H., Balci,
A., Medeni, T., and Weerakkody, V. (2012) A New COBRA Framework to
Evaluate E-government Services: A Citizen Centric Perspective, Government
Information Quarterly, in second review process.
- Irani Z, Weerakkody V, Kamal M, Hindi M, Osman I, Anouze AL,
El-Haddadeh R, Lee H, Osmani M (2012) An Analysis of Methodologies
Utilised in e-government Research: A User Satisfaction Perspective. Journal
of Enterprise Information Management, 25 (3), pp. 298 - 313. 10.1108/17410391211224417
- Tsohou A., Lee H., Irani Z., Weerakkody V., Osman I., Latif A., Medeni
T., "Evaluating E- government Services From A Citizens' Perspective: A
Reference Process Model", CD- ROM/Online Proceedings of the European,
Mediterranean & Middle Eastern Conference on Information Systems
(EMCIS), (Eds. Ghoneim A., Klischewski R., Schrödl H., Muhammed K.), 7-8
June, 2012, Munich, Germany
- Tsohou A., Lee H., Irani Z., Weerakkody V., Osman I., Latif A., Medeni
T., "Proposing a Reference Process Model for the Citizen-Centric
Evaluation of E-government Services", Transforming Government: People,
Process and Policy, forthcoming
Details of the impact
CEES evaluation of 13 e-services available by the Turkish government's
e-Government gateway had a measurable direct impact on service quality and
on user satisfaction.
By assessing the relative satisfaction of citizens with e-government
services through the COBRA framework and by setting targets for
improvement in core strategic areas (e.g. citizen health related
e-service), Turksat was able to rank services against those that performed
well — identifying critical factors for consumer satisfaction.
Turksat was then able to allocate resources to these areas and improve
their e-government provision. Improvements included:
- increased capacity at the e-government gateway call centre
- more stable infrastructures leading to greater continuity of service
- introduction of mobile access
- personalisation of service through `My Page' portals
- improved accessibility
(CEES report, 2013).
This led to positive evaluations that included increased citizen
satisfaction index measurements and certifications of excellence
associated with service improvements. For example in 2010 the citizen
satisfaction index was measured as 65.5, a figure that rose to 66.18 in
2011 (CEES report, 2013) after the start of the improvements. Further,
TURKSAT has been identified as a centre of excellence due to the CEES
project (ICT RTD Technological Audit Project Potential in Turkey
report, 2012 page 48 table 9.8). This has enhanced the TURKSAT brand
and national credibility.
TURKSAT has shared CEES knowledge and results with 27 municipal partners
to more fully coordinate e-government services. These municipal partners
now offer services on the e- government gateway.
The CEES project formed the basis for I-MEET funded by Qatar National
Research Foundation which extended the COBRA framework to include
government evaluations and which is being applied to Qatar, Lebanon and
the UK. For the application of the extended COBRA framework, workshops
were organised in each country in 2012:
- an e-government service providers' engagement workshop was held in
Qatar on 19th Jan 2012 where the e-government service
providers (ictQATAR) followed a presentation of the Turkey case outlined
above and decided to adopt the COBRA framework for evaluating their
- a similar workshop was held in Lebanon on 14th April 2012
with representatives from the Office of the Minister of State for
Administrative Reform (OMSAR) who also decided to adopt the COBRA
framework for the improvement of their e-government services.
- the COBRA framework was presented to local authorities in the UK
during Transformational Government Workshop held in London on 13th
An Estonian research team used the COBRA framework to evaluate their
e-government services. This resulted in a postgraduate research thesis
called "Quality Assessment of Estonian e-government Services" by Hannaes
Lehemets. (Thesis, Institute of Computer Science, University of Tartu,
2012). The results of the research were communicated to the Estonian State
Information System Office (RISO) — http://www.riso.ee/en/.
Specifically, a copy of the thesis was given to the head of the IT
Architecture Office of RISO (Uuno Vallner) who acknowledged the
conclusions of the study.
Brunel research leading to the COBRA framework is highly significant in
that no reliable benchmarking index, that included citizens' and
government perspectives, was previously available to evaluate e-government
services. The research is therefore a step towards the development of a
holistic and hence more reliable benchmark. Its reach is global in that
the development of COBRA in the Middle Eastern context means that it has
universal applicability and can potentially be used all over the world.
Sources to corroborate the impact
- I-MEET research grant contract and research plan (to support the claim
that new project was funded based on the output of CEES project)
- CEES Report: Citizen Oriented Evaluation of e-Government Services: A
Reference process Model, February 2013 to corroborate improvement in
e-government provision and citizen satisfaction
- Professor of Software Engineering, University of Tartu, Estonia. The
Professor can provide factual statements on the use of COBRA framework
of CEES project in a Master Thesis of one of his students in Estonia.
- CEES project deliverable D12 Exploitation Strategy written by Turksat
(available on http://www.iapp-cees.eu
Dissemination page) — to support the claim that COBRA helped Turksat
improve their e-government services and citizen satisfaction
- Assistant Secretary General-Information Technology Sector, ICT
Government Programs, ictQatar, Qatar. This person can be contacted to
support the claim that ictQatar adopted COBTRA framework for the
evaluation of their eGov services.
- Head of the E-Government Unit, OMSAR, Lebanon. This person can be
contacted to support the claim that OMSAR adopted COBRA framework for
the evaluation of e- government services in Lebanon.
ICT RTD Technological Audit Project Potential in Turkey report, 2012
page 48 table 9.8 to corroborate CEES as CoE
- The thesis of Hannes Lehemets available at Brunel Business School.