Summary of the impact

The Technology Management Group (TMG) in the Department of Space and Climate Physics (also known as the Mullard Space Science Laboratory, or MSSL) at UCL has developed a range of professional training courses for industry that promote a forward-looking approach to the management of technology projects. Industrial customers have invested almost £2.4 million on the training within the REF impact period, greatly valuing its impact in helping their staff deal with the challenges of modern, complex projects, such as achieving high reliability in network-enabled systems that need to perform in the harshest environments. The training has improved engineering capability and organisational effectiveness for its customers, helping them to deliver excellent performance — to budget, on time and with the quality and functionality required. The TMG has also contributed to a systems engineering competency framework that is being used worldwide in the professional certification of systems engineers.

Underpinning research

Space agencies operate in a highly developed systems engineering environment where project failure of any form is not tolerated. Systems engineering as a discipline owes much to seminal work undertaken by NASA; however, in many cases the rigorous formality of a NASA approach is inappropriate to a particular system development and more pragmatic solutions have to be found. UCL's MSSL has gained a high level of competence in this area and has developed numerous insights on how to deliver successful projects in the most demanding of environments.

These insights into systems engineering have arisen from MSSL's programme of instrument development for space science applications, which has been running since 1959 and has particularly developed in this area since 1993. Each development programme involves the management and engineering of a complex instrument system, often as the Principal Investigator group within an international consortium of institutes and an industrial supply chain. Starting from the early 2000s, after many of these space projects UCL's TMG conducted interviews with the project managers and systems engineers to review their experiences. In January 2010, the TMG undertook a project to consolidate their findings formally, in order to provide a more coherent expression of best practice in project management and systems engineering. This project drew on the interviews and also involved an intensive three-day workshop in which TMG staff and programme managers identified the influences that made the greatest impact on the outcomes of MSSL projects. From an initial brainstorm of issues, a shortlist of common themes was identified, and from these themes it was found that a set of five orthogonal `principles' were needed to cover the most important issues: `principles govern process', `seek alternative systems perspectives', `understand the enterprise context', `integrate systems engineering and project management', and `invest in the early stages of projects'. Behind these principles is a will to anticipate and respond to a changing environment with a focus on achieving long-term value for the enterprise. They were presented at the 2012 International Systems Engineering Symposium and in a journal article [1], and are now applied in both space projects and non-space projects at MSSL.

An example of an experience that the TMG drew upon to formulate their five principles comes from the area of design qualification. In space projects, environmental qualification of a design is usually an essential part of the development process and involves subjecting a test item to higher levels of stress than it would be expected to experience during launch. Ideally, this involves building a special item purely for testing and then discarding it, since the testing process may have weakened it, i.e. the design rather than the test item is qualified. To save resources, an increasingly common ("protoflight") approach is to test the actual flight hardware, but to a lower level (still above flight levels), i.e. to qualify the item rather than the design. When an item fails either type of qualification test, the formal process demands repair and retest; however, repair would take time and resources and repetition of testing could degrade the strength below a flight-acceptable level. At this point, it is important to remember that the testing process is meant to reduce risk, not increase it. In a number of MSSL projects, items have failed testing; for example, during a protoflight vibration test, several components on a printed circuit board became detached. On analysis, it was discovered that all had a common and simple mounting problem. Rather than risk weakening other components with a full retest, the board was repaired and tested at a much lower level with the full agreement of the space agency involved (a second failure would have set the project back two months while a new item was built and tested). This is one of the examples that inspired the `principles govern process' principle.

Further insights into systems engineering have arisen from MSSL's research into (a) the impact of technology decisions in the instrumentation supply chain, with a focus on the pharmaceutical and agricultural sectors (2002-05) [2, 3]; (b) the costs and benefits of introducing new technology in transport (2008-11); and (c) improving the understanding across industry sectors of terminology used in systems engineering and systems architecting (2011) [4, 5]. Each project has led to significant insights that have been integrated into the training courses delivered by the TMG. For example, project (a) led to the development of a methodology for mapping future technology needs and capabilities, project (b) led to the insight that value for money (over the lifecycle) is often overlooked in technology development projects, and project (c) led to the insight that a lack of standard systems engineering terminology is hampering understanding in UK industry.

Key UCL researchers: Alan Smith (Professor of Detector Physics 1998-present) and Michael Emes (Research Associate 2002-2005; Senior Research Associate 2005-October 2013; Principal Research Associate October 2013-present).

References to the research

References [1], [2] and [4] best indicate the quality of the underpinning research.

Research grants: Research funding for the three projects described towards the end of section 2 included £190,000 from EPSRC in 2002-05, £350,000 from the Technology Strategy Board in 2008-11, and £20,000 from the International Council on Systems Engineering in 2011.

Details of the impact

Since 2008, TMG has disseminated its expertise in systems engineering through its portfolio of training courses, with resulting improvements in performance and competitiveness of a range of companies, including some of the world's largest aerospace, defence and engineering companies. This know-how has received an even wider audience with TMG's contribution to a systems engineering competency framework that has been adopted worldwide in the professional certification of systems engineers.

Improvement in the performance of organisations through the provision of training: The TMG has developed and delivers a range of tailored continuing professional development courses for industry in the areas of systems engineering, project management and technology management. Over the period 2008 to July 2013, 76 of these courses were delivered across five continents to 14 customer organisations: the European Space Agency (ESA), General Dynamics UK, SELEX ES, BAE Systems, General Electric, QinetiQ, EADS Astrium, Ultra Electronics, EADS Cassidian, National Air Traffic Services, Mahindra, Marshall, L3 Communications and Transport for London. The number of trainee-days per year (# courses x # days x average # trainees per course) showed a general increase (254 in 2008, 580 in 2009, 555 in 2010, 630 in 2011 and 1,020 in 2012), almost doubling between 2011 and 2012, and totalled 3,039 trainee-days in 2008-12. Teaching on these courses involves a mix of interactive lectures, discussions of case studies, group exercises and dynamic simulations of projects, and is underpinned by the principles derived from the TMG's experience in space instrument developments and their research into systems engineering practice. There is a common focus on delivering value in the face of unpredictable or changing requirements, as today's complex projects demand.

All of the courses seek to share theory and best practice in the management of technology projects, including technology decision-making, risk management, and ensuring projects are well-managed, maximising the chance of completion on time, to budget and with the quality envisaged. This training reduces the risk of major project failure (overspend, late delivery, inadequate functionality and failure in service). This is particularly important for the TMG's customer organisations, which operate in sectors where very large and complex systems are developed; the complexity of such systems and the nature of the development approach make them very susceptible to huge cost overruns. For example, in a project led by BAE Systems, the Royal Navy's Astute class nuclear submarine programme is now expected to finish over seven years late and £1.3 billion over budget; problems the company has had with projects like this are a key justification for investing in the TMG's training courses.

One indicator of impact is the overall cost of the training to the company, which includes direct costs (charged by UCL) and the staff-time of delegates (salary assumed to be £100,000 per delegate including overheads). The direct costs are £634,000 and the staff costs are £1.748 million giving a total over the period of almost £2.4 million. Companies have been prepared to invest this level of resource with an expectation that this will lead to an improved performance and greater profitability. For the type of training offered, providing proof of the link between training course delivery and company performance is difficult, but there is a large amount of repeat business, with several of the customers having been using the courses throughout the five-year period covered.

One company that has made increasing use of the training courses is Ultra Electronics. This company, with around 5,000 employees worldwide, identified the need for systems engineering training following a number of engineering `near misses' that could have resulted in fatalities. Initially (in 2008), Ultra just asked the TMG to provide systems engineering training courses in the UK. This service was subsequently extended to North America, and, since 2011, TMG has also provided project management courses in the UK and US, and occasional consultancy services. Over 200 Ultra engineers, managers and directors have now received training, and 26 courses have been delivered since 2008 [A]. The company's Chairman has commented on the value of the courses to his company, saying: "the feedback from Ultra's engineers has been extremely positive and in my view has had a significant beneficial effect on Ultra's engineering capabilities", and that "through the adoption of best practice, senior management has the confidence to compete for large engineering programmes without the necessity of including excessive contingencies in competitive bids. As a result the company has been successful in securing more business through competition and which it has executed successfully and continues to do so. A proof of the effectiveness and value of this activity is that Professor Smith's courses continue to be offered to Ultra's personnel" [A]. He also noted that the consultancy Professor Smith provided to support project reviews had led to "significant improvement in Ultra's Systems Engineering capability with consequent positive impact on the Group's competitiveness" [A]. Ultra also recommended the service to Mahindra [A], a major Indian partner of theirs, and the TMG has provided training and consultancy to them on two occasions in 2012/13. Repeat business and deepening of relationships like this are strong indicators that customers believe that the benefits to their organisations of using the training courses is greater than the costs they incur in doing so.

Selex ES, an international electronics company, has also benefited from the training. Eleven courses have been provided for 176 trainees in the last three years at sites in Luton, Basildon and Edinburgh. A range of courses in Engineering for Complex Systems has been developed and delivered at foundation, experienced and senior management levels. Selex's Head of Systems & Supportability Engineering (UK) stated: "It is clear that the training is adding huge benefits to the company in many ways" [B]. The impacts on Selex that he identified include "The message of Systems Thinking and using the principles to add value to the wider company is gradually percolating to senior managers. Last year we used such principles to actually redesign part of our organisation and processes"; "Engineers understanding that they have more than a technical role to play (leadership, schedule, cost, etc.)"; "There is a wider cross-discipline appreciation of SE principles and of each other's discipline too"; and "Engineers are becoming aware that the old way of thinking isn't enough today. Just that realisation is significant as they are asking questions or performing additional analysis that wouldn't normally be done" [B].

Another organisation within which the training has led to changes in knowledge, capabilities and behaviour is General Dynamics UK. Eleven courses have been provided to 143 trainees since 2009 in Newbridge, South Wales, in the subjects of Systems Engineering and Systems Design. The company's Learning and Organisational Development Consultant reported comments from employees who benefited from training, including "I now see the systems as a whole and how the small part I am working on fits into the bigger picture", "I understand that applying process and systems into daily working practice can improve work output" and "Understanding of process has meant that I have been able to work more efficiently" [C]. The improved knowledge and skills have benefited four General Dynamics UK projects: Joint System Integration Body, Scout SV Video Server, Bowman Long Term Support Services, and New Integrated Marines Communication and Information System.

A web-based training course in Technology Risk Management was developed for BP in 2008 and several evening lectures have been provided over the last five years to support a University of Manchester course. BP "found UCL's support to be of high quality and valuable", reporting that "over the past few years, BP has become much, much better at understanding technology risks in the context of the facilities and equipment it needs to develop. The importance of this cannot be overstated. As our industry has found many times, the costs associated with recovering from a failed piece of technology far outweigh the costs of the initial investment in planning the appropriate technology development and well-managed implementation. The many discussions and activities between UCL and BP over the years have demonstrated that UCL possesses unique skills and experience that are relevant to BP now and in the future. It is also able to work flexibly with our large organization, an important attribute in itself." [D]

Impacts on practitioners and professional bodies: Based on their knowledge and experience of systems engineering, underpinned by the research in references [1] to [5] above, three TMG members were selected to chair or otherwise take part in the International Council on Systems Engineering (INCOSE) working group on the core competencies of systems engineering, and contributed significant insights to this group. The Systems Engineering Competency Framework that was developed was adopted worldwide by INCOSE in 2010 [E], including in the professional certification of systems engineers. It has been rolled out across a number of companies as a basis for career development and standardisation of systems engineering practice; major organisations that have used the framework include Thales, General Dynamics, BAE Systems, Atego, Bombardier and the Ministry of Defence [F].

Sources to corroborate the impact

[A] Supporting statement from the Chairman of Ultra Electronics — corroborates the benefit of the training to Ultra. Available on request.

[B] Supporting statement from Head of Systems & Supportability Engineering (UK), Selex ES — corroborates the benefit of the training to Selex. Available on request.

[C] Supporting statement from Learning and Organisational Development Consultant, General Dynamics UK — corroborates the benefit of the training to General Dynamics. Available on request.

[D] Supporting statement from Engineering Manager (GPO Concept Development Engineering), BP — corroborates the impact of the training on BP. Available on request.

[E] The INCOSE webpage http://bit.ly/19fRHmK and the Systems Engineering Body of Knowledge webpage http://www.sebokwiki.org/wiki/Roles_and_Competencies corroborate INCOSE's worldwide adoption of the Systems Engineering Competency Framework in 2010.

[F] The former president of INCOSE UK (now Managing Director of Cleave Systems Limited) can corroborate UCL's contribution to the INCOSE Core Competencies Working Group and the use of the competencies framework by major organisations. Contact details provided separately.