Metrics and Performance Management
ATM-2003 Executive Report
Introduction
ATM-2003 in Budapest, Hungary, was the fifth in a series of international ATM R&D Seminars that began with a meeting in Saclay, France in 1997. Subsequent Seminars were held in Orlando, Florida (1998), Napoli, Italy (2000), and Santa Fe, New Mexico (2001). The Seminar has grown from 60 people and 40 invited papers in Saclay to 200 people and 117 refereed papers in Budapest. In Budapest, for the first time, there was a very successful session with nine student papers. The purpose of the Seminar has remained the same - to create or reinforce working and personal relationships between leading experts and researchers in the ATM R&D. The hope of the original organizers was to establish a true research community where available results are shared, where consensus on major issues is achieved, and where recommendations to decision makers about research directions and research result implementation can be developed.
So where do we stand after six years? Topics addressed at the Seminar have evolved and broadened; ATM is maturing as a research discipline; we are becoming more of an international R&D community; we are starting to see more cooperation/awareness of work of others – but not enough (references generally don’t reflect the work of many labs, in most areas, people not really focused on furthering the state of the art as one would expect from a mature science); there are more and more joint papers from several labs and across the ocean, but again, this could be furthered, perhaps by having more exchange of people that can lead to, not just collaboration, but to cooperative research on common problems with common objectives; finally, we still are not reaching, at each symposium, a common understanding of what the open issues are, what progress has been made, and where more research emphasis is needed.
Selected Research Conclusions
- Modeling and simulation are becoming more and more important, especially for assessment of new methods and concepts for ATM. There is a need for tools to help understand total system impact and the economic aspects of future changes. New tool sets must be comprehensive, although not necessarily integrated. An underlying metric framework, currently being developed under the FAA/Eurocontrol R&D cooperation, will be helpful in understanding and comparing results.
- Safety research and safety modeling are still not getting much attention except for the excellent work on safety models at NLR. This area of research requires more financial support – it will be particularly important as we begin to develop and evaluate significant changes to today’s ATM paradigm.
- Parts of ASAS are becoming mature. We see R&D progress on both sides of Atlantic and an accepted international framework for ASAS services. It is time to increase focus on the use of initial ASAS services in the context of specific holistic ATM operational concepts and subsequent operational implementation.
- There were only 3 papers in the track on Weather and Environment. ATM research related to the environment must be encouraged. There is a need to integrate weather understanding and prediction into our ATM tools.
- In airport related research the focus on safety should continue. Unless there is international standardization for use of tools, human computer interfaces, and operational procedures on the flight deck, there will be serious safety ramifications.
Research Recommendations
While the perennial concern about how few research results have found their way into operational systems remains, the plenary presentations from Europe and the U.S. reflected the internationally recognized need for a change in research direction. If our community is to overcome the “capacity wall”, the traditional research aimed at ATM modernization is not enough. A focused effort aimed at a longer-term view for transforming our system is essential.
In Budapest, only one third of the papers came from Europe (there were no papers from the DLR). This appears to be a reflection of the decreased funding of the national labs from their own governments and the resulting need to spend more and more time on proposals for EC funds, where many projects have a near term focus because of the specific interests of the industry research partners. This alarming trend, if not reversed, could well lead to an erosion of the research capabilities of the labs. In the U.S. this trend is not as apparent, but most of the research is still aimed at modernization. If our community is to step up to the challenge of ATM transformation, the research establishments must increase their focus on advanced research.
Future Seminars
History has shown that this Seminar series has played a major role in the maturation of our research community and it is recommended that it continue. While there are frequent technical interchange meetings on specific topics through the FAA/Eurocontrol R&D cooperation, this Seminar series represents an opportunity for people to engage in the broader research community and to get to better know researchers from other organizations, on both sides of the Atlantic. Student participation brings young people to this community - sponsoring student participation at future seminars is recommended.
To realize the potential benefit of the symposium, managers of research establishments must, when they sponsor one of their people for the Seminar, make it clear that this is not just a meeting to “give a paper”, but that this is opportunity to become more engaged in the community and, as such, should be viewed as a special reward for excellence.
To take the next steps in the evolution of the Seminar series, the organizers must find ways to go beyond information exchange and set the stage for a focus on understanding open issues and furthering the state of the art.
ATM-2003 Session Report
Air/Ground Co-operation
The history of the Seminars has evolved from concentrating on principles of ASAS (Saclay 1997, Orlando 1998, Napoli 2000) and full 4D operations separately to looking at integrated air/ground co-operation (Santa Fe 2001, Budapest 2003). Many of the issues identified in Santa Fe 2001 (time-based versus distance based spacing, difficulties with autonomous operations in mixed aircraft equipment environment, requirement for aircraft intent data) were treated in work presented in Budapest 2003.
Recognizing that neither 4D trajectory based operations nor ASAS alone can provide a solution for the future of ATM, a consensus is slowly building that a mix of these two concept streams could be the best solution. With ASAS strictly focusing on the separation process, 4D trajectories would be used mainly to address strategic and tactical flow management and multi-sector functions like traffic complexity reduction.
The EUROCONTROL and the Boeing ATM concept analysis papers showed very similar operational flow diagrams to explain such a concept. Serious experimental work both in the U.S. (e.g. DAG/TM) and Europe (e.g. former PHARE and current Cospace) starts to show some evidence that combining the two modes is technically and operationally feasible. Much more work is needed in assessing the benefits and safety aspects of this type of mixed concepts. It can be concluded that U.S. and European research streams start getting closer together.
Human Factors
Human Factors has been an important subject in all R&D Seminars from a few presentations emphasising the role of HF in ATC systems design (Saclay 1997, Orlando 1998), through full sessions in Napoli 2000 and Santa Fe 2001 concentrating on human machine interfaces and human performance assessment, identifying the need for cognitive models to accommodate the variability in controller behavior.
This was confirmed in Budapest 2003 where the need to predict controller performance and workload through modeling and measuring was still mentioned as a problem. Surprisingly to some, there is still work on paper flight strips, although some European ATC centres already function satisfactorily without and some argue that only socio-cultural reasons keep them alive.
An interesting new aspect was that new technologies introduced simultaneously e.g. Airborne Separation Assistance Systems (ASAS), computer assistance tools for the Air Traffic Controller, and data link of important parameters between airborne and ground systems, can alter the control environment, thereby affecting system stability, with safety and efficiency implications. This warrants further study. Continuing studies exploring using mixed media (i.e. data link and voice) for communications have found effects upon safety and workload, this having an effect upon system efficiency. As we near implementation of these technologies, new insight into the operational practicalities are being revealed, and require considerably more research.
Promising new innovative/creative techniques exist to develop & elicit HF requirements (RESCUE) and although these are resource intensive, they may resolve some problems, like scenario walkthroughs if applied universally.
In Europe a multi/disciplinary creative design methodology applied by non-ATM experts has shown to lead to promising results for ATC centre environment definition in a relatively short time.
From the presentations and some discussions it can be concluded that that the U.S. and European work is going on quite independently and co-operation could be improved.
Safety
Safety was not mentioned as a separate research topic in the Saclay 1997 and Orlando 1998 Seminars, but got a separate session in Napoli 2000 where the need for a joint European-U.S. approach to safety of integrated air/ground operations was emphasised. In Budapest 2003 a separate track was devoted to safety but only 4 papers (out of 13 submitted) were accepted. It puts into question if the ATM research community really understands just how important safety R&D is.
The papers presented interesting relations between safety and operational culture by showing that different operational practices were used between Air Traffic Service provider Units (ATSUs) in the U.S. and between different operational units in Sweden. It would be interesting to extend this type of work to other nations, as especially Sweden may be non-representative in cultural respect.
A good example of a common methodology across the Atlantic for investigating ATM incidents is the JANUS Incident reporting scheme: the extension to a joint timeline for air and ground events is a very promising future path.
The papers showed that there are diverse methods for safety assessments from field trials through to mathematical models – they are complementary and should be!
Analysis of the safety aspects of the Capstone trials showed that safety benefits can be obtained in unexpected ways through the imagination of pilots and controllers (e.g. some pilots used the traffic display to identify aircraft that already had been where they intended to go and called them by radio to get weather and runway conditions).
Again the papers showed that researchers on both sides of the Atlantic are not fully aware of the capabilities of their counterparts, access and dissemination of information is essential.
As in other areas, several speakers mentioned the need for commonly accepted metrics.
Decision Support Tools
Since its first appearance in Saclay in 1997, the Decision
Support Tools track has been one of the most active in the Seminar series. At that time the emphasis was mainly focused
on conflict prediction techniques and new concept support systems such as
ERATO, CTAS and HIPS. However a significant element was the report on the
recently deployed URET prototype at Indianapolis. Over the Seminars, against a backdrop of progressive URET
implementation in the U.S. and the introduction of a number of stripless
systems in Europe, we have seen a growth in understanding and confidence on the
possibilities of developing acceptable tools for controllers and other key
roles in the ATM system. In 2003 we
saw an emphasis on medium
term, needs driven, evolutions of current concepts and on opportunities based
on recent improvements. Decision tools
are foreseen for an increasing number of different functions and individual
decision support capabilities are starting to be integrated to build larger
scale solutions.
As a
significant step in international co-operation, a Trajectory Prediction Action
Plan proposed to the EUROCONTROL/FAA R&D Committee appeared to have good
support from amongst Seminar participants and could play an important role in
charting future joint R&D.
In contrast
to the keynote addresses there was little attention on the long term, or future
tool concepts, although it is noteworthy that data link appears to be central
all those which were discussed. The
most significant theme was uncertainty; its impact, management and expression
and how to ensure that it is accounted for in decision support tool design
(e.g. weather forecasts, demand, capacity, etc.).
Weather and Environment
Weather and Environment first became the subject
of a track at the Santa Fe Seminar (2001) where it was linked with
Metrics. At Budapest there were two
papers addressing each of the sub-themes. In the environmental area it was
clear that there was good transatlantic co-operation in working towards a
single flight movement database. This has the potential to become an invaluable
resource for environmental impact and climatic modeling but direct links to the
modeling processes were not yet evident. Surprisingly, no papers on noise were
presented.
On the weather topic it was clear that there are
important gains to be derived from the availability of storm height estimation
creating opportunities for over flying routes. There was also a strong emphasis on the need to integrate weather information into
decision support tools from their inception.
A flow management system, which ignores weather, does not address the
‘real’ operational requirement (e.g. only 3 summer days NOT effected by storms
in U.S. East Coast area). There was
good evidence in the Decision Tools sessions that this message is beginning to
be heard. Discussion centered on the types
of uncertainty model suitable for weather modelling and the integration of
weather, noise and emission forecasting into decision support.
Traffic Flow Optimization
This track title also emerged for the first time
at the Seminar in Santa Fe. Prior to
that, many of these papers could have found themselves included in sessions
addressing Collaborative Decision Making, Sector Design and Optimization,
Airspace Modeling or Conflict Detection and Resolution (Saclay).
- In Budapest a number of diverse applications
were linked by an approach seeking optimization through a wide range of
mathematical modeling techniques.
- Interestingly, optimization is now considered
for a very wide range of criteria. In
addition to sector capacity and minimization of conflicts, we find models that
seek to minimize the impact of weather on re-routing, schedule recovery, and on
equity in ground delay.
- A wide variety of algorithms are employed,
e.g. Genetic Algorithms, Lagrangian Relaxation, Constraint Satisfaction Programming,
etc.
- Two main themes may be identified. Longer-term research, coming primarily from
Europe, tends to be seeking global optimization and to deal with multiple
constraints. Shorter-term research
tends to be concerned with dealing with a single constraint and providing
support for dynamic decision making in a particular operational context.
- Discussion focused on the difficulties
of providing local actors with a ‘global’ view; the balance between global and
local optimisation; the need to share data, benchmarks, etc., and once again,
dealing with uncertainty – management versus elimination.
Airport Management
Discussions of airport management in the ATM Seminar series date back to Saclay, but, over the years, research in dealing with safety and efficiency of surface operations has increased and the resulting number of papers dealing with airport issues at the Budapest Seminar was up to nine.
- We are starting to gain experience with surface management systems through major demonstrations, but much research remains before we can get mature surface management systems that are integrated with arrival and departure planning. International cooperation is essential, lest safety be compromised by confusing differences in the ways pilots interact with these systems.
- There is consensus that we must move to time based separation for aircraft arrivals and that there is a need for re-examining the ICAO separation standards.
- Three of the papers dealt with issues surrounding airport capacity, particularly in the light of wake vortices. The discussion surrounding these papers pointed to a need for a robust safety case, to a clear, internationally accepted wake vortex hazard definition, and to the continued need for broad data collection and analysis. The opportunity of cross Atlantic cooperation, particularly in the latter area, needs to be exploited.
- Finally, we are starting to see research dealing with
broader airport issues, such as airport layout, innovative landside operations,
and inter-modal interoperability.
Research in these areas must be encouraged if we want an air
transportation system that is friendly and convenient for the flying public.
Metrics and Performance Management
This has been a topic of major interest at all ATM Seminars and, from the Budapest Seminar, it is clear that it is of increasing importance, especially to demonstrate the benefits of near term developments and new longer-term concepts. Metrics need to reflect what we are trying to achieve, to have the ability to measure our progress year on year, and to be understood by decision makers at high levels in all parts of the industry. The workshop consensus was that metrics work for ATM is not yet mature.
- Progress is being made in measuring the benefits of new decision support tools and procedures (e.g. sector complexity), but it was clear from the presentations at the symposium that there is a need, not necessarily for common metrics, but for a common framework for metrics so that alternative proposals can be easily compared. It is encouraging that there is cooperative FAA/Eurocontrol activity in establishing a common framework.
- We are getting
better at understanding incremental improvements to the current CNS/ATM system,
but there is not enough work on model development and assessment methodology
for evaluating the potential of future, long term concept that change the
underlying CNS/ATM paradigm.
- There is a lack of
good understanding of the non-linear, dynamic adaptive behavior of the CNS/ATM
system. Such understanding is essential
if we are to make significant progress in system performance improvements.
- Better performance
assessment will require:
- a currently lacking common understanding of the degree of aggregation
(short term peaks versus longer term averages) necessary for analyses.
- data availability is improving (especially in the U.S.), but more and
better data are needed.
- sensitivity analysis has to become a common practice.
- Our performance
assessment tool set must be expanded, especially for cost/economic studies and
safety and human performance analysis. As more and better tools become
available, our analyses must include more tradeoffs of capacity – delay –
safety – efficiency - and cost.
- Finally, we would like to see more work in Europe. Of the 15 papers presented, 11 were from the U.S. and one was a joint U.S./European paper.