National Missile Defense Review Committee Report

Text of the "Welch Report"

In November, the National Missile Defense Review Committee, an independent panel headed by retired Air Force Chief of Staff General Larry Welch, currently president of the Institute for Defense Analyses, issued an update of its February 1998 report evaluating the U.S. national missile defense (NMD) program. While acknowledging that improvements had been made since its first report, the committee was still highly critical of the NMD program and recommended that the June 2000 deployment review be considered a "feasibility" rather than a "readiness" decision. (See news article.)

The report, as presented, was a briefing that used both visual aids and text. What follows is the full text of the report with selected slides that the editors felt were particularly useful. Because the report uses numerous technical terms, readers are encouraged to use the extensive list of acronyms and abbreviations. Readers who simply want an overview of the report should skip to the "Findings and Recommendations" section.

The full report with all accompanying slides can be found online at http://www.acq.osd.mil/bmdo/bmdolink/pdf/welch.pdf (requires Adobe Acrobat reader, and may require some time to download depending on your connection speed).

The Panel on Reducing Risk in BMD [Ballistic Missile Defense] Flight Test Programs met originally in late 1997 and early 1998 to assess risk in the flight test programs for the hit-to-kill (HTK) ballistic missile defense programs [i.e., National Missile Defense (NMD), Theater High Altitude Area Defense (THAAD), Navy Theater-Wide Defense System, and the Patriot System with the new Patriot Advanced Capability (PAC)-3 interceptor].

Since the release of the panel's report in February 1998, the program for the NMD system has been restructured to allow more time and testing before critical decision points. The NMD Joint Project Office (NMD JPO) has also selected the Boeing Company as the lead System Integrator (LSI). These two major changes have prompted requests for the panel to reconvene and reexamine the NMD program with regard to the features listed above.

The Office of the Secretary of Defense (OSD) program guidance to the NMD JPO and the LSI continues to call for a system to defend against a limited attack and for a Deployment Readiness Review (DRR) in FY2000 [fiscal year]. However, the decisions that were originally to be made at the DRR under the earlier 3 + 3 program are now to be made in phases between 2000 and 2003. The initial operating capability (IOC) has been moved from 2003 to 2005.

Still, OSD guidance requires that the program offices take no programmatic action that would preclude an emergency deployment in 2003 should conditions require and technology permit such a deployment.

This last requirement will be difficult to convert to realistic program actions, and there is the potential that this guidance could lead to emphasis on near-term deployment readiness at the expense of properly completing the development activities needed for a system that meets the stated operational need.

The outline of the report is shown above [not shown]. In Sections 2-6, we outline the recommendations of the Panel on Reducing Risk in BMD Flight Test Programs and note how these recommendations have been implemented-adding information from this review.

The NMD program has characteristics that are fundamentally different from normal development programs. One such unique characteristic is that national leaders have made a conscious decision that the significance of the potential threat to the security of the United States warrants pursuing the program on a high-risk schedule.

The original 3 + 3 program schedule was widely regarded as very high risk, with no reasonable expectation of meeting that schedule. The restructure of the NMD program was intended to return the program to an acceptable expectation of execution.

However, even at this early point in execution of the restructured program, delays in key events-ground tests and flight tests-and delays in the development of key simulation and test facilities are compressing the schedule. Continued compression will require that the decision points be moved commensurate with event delays in order not to increase risk further. While the need for an emergency deployment may arise in the future, the possibility of such a decision should not be allowed to drive the restructured program. The restructured program, as currently defined, is still highly demanding, and any additional burdens for program management can seriously compromise it.

While the DRR date of June 2000 is unchanged from the 3 + 3 program structure, the nature of the decision expected from the initial DRR is significantly changed. There is no intent to provide detailed information leading to production of the radar or the GBI at that time. Instead, the decision will be limited to a site selection and award of the construction contract-both essential long-lead steps. Subsequent Defense Acquisition Boards (DABs) will be held to approve the remaining elements of the program. Hence, the DRR should be regarded more as a feasibility decision with some long-lead deployment actions rather than a readiness decision.

One performance criterion now specified for achievement before DRR includes two successful intercepts, with at least one occurring in an Integrated System Test (IST). The criterion clearly makes the DRR a performance milestone rather than a calendar milestone Still, the program guidance, in various forms, places great emphasis on the calendar milestone of June 2000. It will take strong program management and top-level support to ensure that the performance requirements and basic system engineering and design functions are not sacrificed to the calendar since no decision will produce successful deployment until the system can be shown to perform as required.

The panel noted that the authority of the JPO, the JPO system engineer and the program managers (PMs) are unclear. It does seem clear that the program's government managers do not have authority commensurate with the responsibility of running the program. The Contracting Officer's Representatives (CORs)-two from the Ballistic Missile Defense Organization (BMDO), two from the Army, and one from the Air Force-are told they are responsible to "ensure the contractor performs the requirements of the contract," but "may not make any commitments or changes that affect price, quality, delivery...of the contract."

Although the LSI reports directly to the JPO, three of the five CORs are from the Service program offices, and they report to their respective Service Program Executive Officers (PEOs). Thus, the authority of the JPO and the role of the JPO system engineer are unclear.

This lack of integrated authority and responsibility at the JPO is inconsistent with the complexity of the integration task that the JPO is overseeing.

At the time of this review, the relationship between the LSI and the individual program element program management was also unclear.

With the selection and start-up of the LSI, the JPO plans for all the element contractors to become subcontractors to the LSI. Hence, the LSI will run the EKV [exoatmospheric kill vehicle], XBR [X-Band Radar], Upgraded Early Warning Radar (UEWR), and BM/C3 [Battle Management/Command, Control and Communications] subcontracts. In the case of the EKV, the panel found that the LSI had not yet penetrated into the details of that program deeply enough to exercise oversight. The panel did not delve into the relationship with the other elements.

The LSI heavily emphasized the very challenging task of integrating the elements into a system-of-systems. However, the lack of government control, combined with the current state of LSI oversight of the subcontractors, leads to risks that can be mitigated by focusing more management on the performance of the elements to be integrated.

Given the set of challenges and the phased decision process, the JPO and LSI have formulated a sensible, phased, incremental approach to the development and deployment decision-while managing the risk. While the information required for decisions will come from ground testing, simulation, and flight testing, it is useful to describe the three major increments of the flight test program.

Through IFT-6 [integrated flight test], the program will focus on technology demonstration with the central objective of developing and demonstrating HTK capability. One requirement for passing DRR is two successful intercepts in four attempts on IFT-3 through IFT-6. IFT-3 and IFT-4 will rely heavily on Kwajalein Missile Range (KMR) radars and equipment to support the EKV in flight. However, beginning with IFT-5, prototype versions of the BM/C3, UEWR, and XBR [using the surrogate ground-based radar (GBR-P)] will be used in the test, thus making this test the first IST.

From IFT-7 through IFT-13, development and testing and demonstrations of system integration will continue. IFT-7 is the earliest that the operational booster can be part of the system for test. IFT-13 is planned to be the first flight test with the operational kill vehicle.

The remaining program will complete Engineering and Manufacturing Development (EMD) and increase confidence in the deployable system until a decision is made to deploy.

The current NMD effort is focused on being ready to deploy a system that meets the minimum requirements to meet the C1 threat-a challenging task using even the best available technology. Still, whether the decision is to continue development or to commit to deployment, technology development and testing will need to continue to meet future needs in expanding to a more sophisticated existing or developing set of threats. The panel found that the JPO and LSI were highly focused on developing a system to counter the C-1 threat with an IOC in 2005 but with minimum attention to planning for system growth and enhanced test capabilities that will be needed to support any enhanced system.

Many of these issues are discussed elsewhere in this report. We discuss only two of them here, and both are related to the facilities at KMR. The first issue is at KMR that there is currently only a single GBI [ground-based interceptor] launcher, yet the system will have to operate with two or more interceptors in flight. This dual-salvo feature cannot be verified with a flight test unless a second launcher is built. A second launcher should be planned and funded now. The second issue at KMR is the degradation of the GBR-P. There is reason for concern that it might be unable to support the test program by 2005 unless action is taken to maintain the capability. Taking a long-term view of NMD, the system will be undergoing tests for 10 to 20 years; hence, the problems with the GBR-P must be addressed now, and a long-term solution must be put in place.

The visit to the Raytheon facility in Tucson highlighted the impacts of the "hardware-poor" nature of the EKV program. There were no spares, no development articles, and no articles available for parallel activities that could significantly reduce development and test risk. The first article built appears to be the one that will fly.

Being hardware-poor has forced the EKV program to forego activities that would have reduced risk. For example, the early flight models in the EKV program will have hundreds of hours of use on them since these flight articles are also the developmental articles. The EKV program is unable to support important development activity, such as improving the test capability of the facility at Arnold Engineering Development Center (AEDC), since the flight test article had to be returned quickly to Tucson to meet schedules. Raytheon has also been unable to provide a test-article to the KKV [kinetic kill vehicle] Hardware-in-the-Loop Simulation (KHILS) facility for facility development and testing. The upcoming acquisition of the 1,000-Hz [hertz] rate table at KHILS could make that facility of high value in future NMD EKV development.

Finally, the lack of spare hardware is driving flight test delays. The inertial measurement unit (IMU) on the test article for IFT-3 failed. The EKV for IFT-4 could not be substituted. So, the IMU was removed from the IFT-4 EKV and installed into the EKV for IFT-3. The defective IMU was returned to the vendor for repairs, but it is not likely to be available for the planned IFT-4 launch date.

The NMD program is characterized by a combination of challenges that add up to severe demands on management. They include the urgent need (leading to a planned high-risk schedule), first use advanced technology, complex system architecture, compressed and concurrent schedules, long development times and long operational life span, complications from arms control agreements, high cost, and participation by a multitude of government organizations and the Services. These program characteristics generate some unique management issues that demand a high degree of clarity in assigning authority and responsibility. As noted in this report, instead of unusual clarity, there is unusual fragmentation and confusion about authority and responsibility.

While the flight tests are high visibility, the vital information needed for development and decisions (i.e., the DRR) will come primarily from the LSI Integrated Distributed Simulation (LIDS) and the IGTs [integrated ground test] anchored with data from the flight tests. However, the panel considers the resources currently available for simulation development and IGTs as inadequate to provide the information needed for the DRR on the planned date.

In fact, the LIDS' development and the execution of the IGTs are currently behind schedule. The plan to mitigate the risks associated with these delays did not provide much confidence to the panel. The Integrated System Test Capability (ISTC) is in heavy demand for the IGTs and the IFTs, and it cannot adequately support both. This is a cause for particular concern in the near term, where the IGT and IFT events have been compressed into a very small time window to support the June 2000 DRR. An expanded capability or some unidentified workaround will be required to support both.

The panel believes that the government and the integrating contractor continue to underestimate the challenge of reliably performing exoatmospheric HTK. Exoatmospheric HTK of ICBM reentry vehicle (RV)-like targets has been successfully demonstrated only twice [Homing Overlay Experiment (HOE) and Exoatmospheric Reentry Intercept Subsystem (ERIS)] with at least 8 failures to date [HOE, ERIS, and Lightweight Exoatmospheric Projectile (LEAP)]. The panel noted a lack of urgency to test the EKV in an end-game hardware-in-the-loop (HWIL) facility, given that most other BMDO HTK programs view this need as critical.

The panel did not think that including decoys on the first intercept attempt was a wise choice, and BMDO has recently elected to remove two of them. The panel views a successful HTK engagement as being difficult enough that the first attempts should not be unnecessarily complicated.

Finally, the panel was concerned about the single test geometry for IFT-7 through IFT-13. The LSI's plan to use the data from the flight tests to anchor the simulations is likely to be compromised seriously by the level of extrapolation that will be required to assess capability over the required flight envelope.

This reinforced the panel's judgment that the government and the LSI were more optimistic than warranted about the state of capability to achieve reliable exoatmospheric HTK capability.

This chart [not shown] provides some additional detail regarding the phased incremental approach described in the overarching observations. IDR, the Interim Design Review, is a prelude to the DRR. SCDR, the System Critical Design Review, is where the configuration of the system is fixed. The 2001 DAB is the decision to go forward with the procurement of the UEWR upgrades and the XBR. The 2003 DAB is the decision on the GBI-the final system decision.

The fact that the system configurations is not fixed until 2001 and the operational version of the EKV will not be tested until FY2003 lead the panel to conclude that the DRR should more properly be considered a Deployment Feasibility Review under the restructured program. If all goes according to the restructured plan, by June 2000, BMDO will have demonstrated the "feasibility" of a NMD system but not the "readiness to deploy" of the system. The demonstration of readiness to deploy will not come until 2003 at the earliest, when the integrated GBI (i.e., operational version of the booster and EKV) is to be demonstrated.

The extra time for more flight tests before the decision to deploy, which resulted in the phased approach to decision-making, lowered the risk in the restructured program.

This chart [available in print edition] shows the change in schedule from the restructure. IFT-lA and IFT-2 were seeker fly-by tests and were completed successfully before the restructure. Three intercepts are to be attempted before the DRR, with IFT-5 being the required IST.

The RRFs [risk reduction flight] mentioned in the chart are tests in which targets of opportunity are tracked, allowing the system to operate, collect data, and simulate intercepts. These can occur on targets launched from Vandenberg AFB [air force base] and Kodiak Island.

After the three key IFTs, there are three key decision points. Following a successful IST, a commit to deployment resulting from the DRR would consist of a decision to select the interceptor site and award the construction contract. IFT-7 is the first flight of the interceptor using the operational booster. The subsequent DAB could approve the start of site construction and approve the early warning radar upgrades, the XBR, and the BM/C3 integration program.

There will be six additional intercept attempts through IFT-13, the first test flight with the operational EKV. The subsequent DAB is to approve the GBI build.

The decision criteria for the DRR specify performance requirements for design development, deployment and program cost. Details regarding the criterion of demonstration of integrated system/element functionality are shown on the next chart [not shown].

The program workload to generate the information for meeting these criteria is formidable. Numerous agencies, including NMD JPO, USD(A&T) [Under Secretary of Defense (Acquisition and Technology)], and the Director of Operational Test and Evaluation (DOT&E) in OSD, must work together to produce the reports demonstrating that the criteria have been met.

[The preceding two paragraphs are repeated for the next slide in the report, which is not shown.]

The information needed to develop and verify the system will come from an orderly combination of simulation, test, and analytic activities. Testing includes ground testing of individual system elements and of subsets of the integrated system. RRFs use targets of opportunity, such as Minuteman reliability launches. IFTs are designed to collect information that cannot be adequately addressed in ground testing. This combination of testing provides validation for the simulations and, in turn, provides confidence for development and deployment decisions.

Given the nature of the NMD mission and the inability to conduct full-scale end-to-end tests in a realistic environment, simulations and analyses will provide much of the needed design and decision information. The review group has urged-and continues to urge-extensive use of HWIL simulations to increase the validity of the simulations and confidence in the validity of the simulations.

As discussed earlier, the program has experienced significant ground and flight test compression since the restructure. IFT-3, the first flight test since award of the lead system integration contract, will be directed by the government and has slipped several months. IFT-4 will be the first IFT fully directed by the LSI, and it has also slipped several months. Since a successful intercept on IFT-5 is a prerequisite for DRR, any significant slip of that test must affect the DRR date.

Similarly, the IGTs are slipping as indicated on the chart [not shown], again compressing the schedule. Given the amount of analysis that must be accomplished based on these tests and the simulations they anchor, this compression is cause for major concern.

Both IGTs and IFTs require the use of the ISTC facility. With both slipping, the demand for the ISTC facility is likely to exceed its capabilities. This could result in even more schedule slips.

The schedule in this chart [not shown] shows that the system design will not be firm until the SCDR in FY01, yet prototypes of the EKV, UEWR, XBR, and BM/C3 are participating in the flight tests now. This high level of concurrency adds risk to the program by raising the possibility of future design and/or requirement changes. Such changes may not cause great harm, but they work against getting a true baseline system for production. Although block changes were said to be planned to resolve this challenge, we found no freeze points to catch specific numbers of flights and intercepts before decision points.

As part of the test program, the NMD program is conducting opportune RRFs. These are occasions in which the program takes advantage of a target that most likely might be launched from Vandenberg AFB toward KMR but possibly from other Pacific launch ranges (e.g., Kodiak Island). In these flights, the elements of the NMD system are turned on and operating. The tracking, classification, and BM/C3 element will operate along with the radars and portions of the interceptor element. However, the interceptor will not launch.

The JPO has been able to influence the payload selection for some of these flights to try to provide a realistic target set for the test. Five RRFs have occurred to date, and 11 are planned before the FY03 DAB decision.

The LIDS is vital to the approach for verifying the system design. It is a simulation that includes models of the individual elements of the system and uses inputs from the threat environment, and lethality models. It is designed to be fast-running so that it can perform thousands of runs to support the DRR. LIDS will use data from the IFTs and IGTs to verify its system descriptions.

Build 4 of the LIDS, despite its importance, is substantially behind schedule. Further, there is serious doubt that the LIDS, as currently configured, will have the capacity to support both IGTs and IFTs in the near term in the role of test-planning. Moreover, the data from the completed IFTs and IGTs must feed back into LIDS to anchor it to the test observables. Since the IFTs and IGTs are slipping, there will be less time before DRR for this process. Hence, the development and reliance on LIDS is rapidly becoming a high-risk portion of the program.

Raytheon performed extensive HWIL testing of the EKV. They used numerous facilities for calibration and characterization of the EKV sensor. They also used the Hover Test facility to test the propulsion system. However, no plans have been made to use the KHILS facility at Eglin AFB.

KHILS is acquiring new capabilities that could provide the opportunity for more realistic testing of the seeker and guidance on the EKV (e.g., a 1,000-Hz rate table that will provide the response necessary to model the EKV). These new capabilities would provide realistic, dynamic scenes and allow the seeker to operate and close the guidance loop all the way to intercept. This type of testing would complement the other HWIL tests being conducted by Raytheon and verify EKV performance in a variety of conditions that cannot be tested at KMR.

As discussed previously, environmental testing of the GBI is a major concern.

Extensive discussions with Raytheon did not provide the panel confidence as to when and how it will be determined whether the EKV will be able to withstand the environments on the new GBI booster. We view this, as does Raytheon, as a high risk to the program. We will discuss this topic later in the report.

The top program risks identified by the LSI [chart available in print edition] are dominated by schedule concerns-reinforcing our conclusion that further compression of events leading to the DRR must be avoided. Instead, the panel recommends slipping the DRR date to accommodate any further delays in key events leading up to the DRR.

The panel was also concerned that the LSI list is dominated by large-scale integration issues. While integration is a challenging task, very difficult element challenges also remain at this stage of development, and the orientation of this list adds to our concerns that the LSI has not yet had time to penetrate the element programs fully.

As expected, the EKV makes the list of high risks for technical and other reasons. However, the panel, in response to questions, was concerned about the lack of identified particulars concerning why the risk was thought to be so high. The panel concluded that no firm risk mitigation plan could be implemented at present to drive down the high risk areas.

For several key risk areas, the planned risk mitigation effort is the flight test that will be impacted by the risk. Once again, the panel strongly urges a greater effort to minimize and close the risks by using ground testing. Flight tests should be used to close only those risks that cannot be tested and verified on the ground.

This chart [not shown] shows the medium- and high-risk areas in the EKV program, as identified by Raytheon. The risk management program at Raytheon appears to be well conceived, well structured and well executed.

On the next three charts [not shown], we discuss further the risks associated with the IMU, the high environmental loads produced by the new GBI booster, and the sensor schedules for EKV integration.

One of the highest EKV risk areas is its ability to withstand the environmental loads of the new booster that will fly on IFT-7 This chart [not shown] shows that the expected shock loads are over an order of magnitude greater than those imparted from the PLV [payload launch vehicle]. We are not certain that the EKV will be able to withstand these loads, and we will not know with any degree of certainty until IFT-13, the flight on which the EKV and new booster are first mated together. This flight is planned for 2003.

The recently failed IMU that was planned for the EKV on IFT-3 has led to schedule slips in IFT-3 and possibly IFT-5. Because no immediate spare was available, the IMU in the EKV planned for IFT-5 was removed and installed into the EKV planned for IFT-3. The defective IMU was returned to the manufacturer for repairs. Because the manufacturer has discontinued making IMUs, a new one cannot be substituted. Rather, Raytheon must wait for the repairs to be made. If delays occur, this might result in delays to IFT-5. This risk is directly attributable to inadequate developmental and spare hardware.

This chart [not shown] addresses the EKV sensor delivery schedule challenge. Deliveries were behind schedule even before the IMU-related schedule slips were announced in late June 1999.

There is still a possibility that sensor delivery can support the current schedule for IFT-4 and IFT-5. However, the schedule margin has been consumed. Any further schedule slip or compression will jeopardize the DRR date.

On this chart [available in print edition], we show the risk mitigation plan for items in IFT-5. In the event that any of the items on theThe Panel on Reducing Risk in BMD [Ballistic Missile Defense] Flight Test Programs met originally in late 1997 and early 1998 to assess risk in the flight test programs for the hit-to-kill (HTK) ballistic missile defense programs [i.e., National Missile Defense (NMD), Theater High Altitude Area Defense (THAAD), Navy Theater-Wide Defense System, and the Patriot System with the new Patriot Advanced Capability (PAC)-3 interceptor].

Since the release of the panel's report in February 1998, the program for the NMD system has been restructured to allow more time and testing before critical decision points. The NMD Joint Project Office (NMD JPO) has also selected the Boeing Company as the lead System Integrator (LSI). These two major changes have prompted requests for the panel to reconvene and reexamine the NMD program with regard to the features listed above [see chart, next page].

The Office of the Secretary of Defense (OSD) program guidance to the NMD JPO and the LSI continues to call for a system to defend against a limited attack and for a Deployment Readiness Review (DRR) in FY2000 [fiscal year]. However, the decisions that were originally to be made at the DRR under the earlier 3 + 3 program are now to be made in phases between 2000 and 2003. The initial operating capability (IOC) has been moved from 2003 to 2005.

Still, OSD guidance requires that the program offices take no programmatic action that would preclude an emergency deployment in 2003 should conditions require and technology permit such a deployment.

This last requirement will be difficult to convert to realistic program actions, and there is the potential that this guidance could lead to emphasis on near-term deployment readiness at the expense of properly completing the development activities needed for a system that meets the stated operational need.

The outline of the report is shown above [not shown]. In Sections 2-6, we outline the recommendations of the Panel on Reducing Risk in BMD Flight Test Programs and note how these recommendations have been implemented-adding information from this review.

The NMD program has characteristics that are fundamentally different from normal development programs. One such unique characteristic is that national leaders have made a conscious decision that the significance of the potential threat to the security of the United States warrants pursuing the program on a high-risk schedule.

The original 3 + 3 program schedule was widely regarded as very high risk, with no reasonable expectation of meeting that schedule. The restructure of the NMD program was intended to return the program to an acceptable expectation of execution.

However, even at this early point in execution of the restructured program, delays in key events-ground tests and flight tests-and delays in the development of key simulation and test facilities are compressing the schedule. Continued compression will require that the decision points be moved commensurate with event delays in order not to increase risk further. While the need for an emergency deployment may arise in the future, the possibility of such a decision should not be allowed to drive the restructured program. The restructured program, as currently defined, is still highly demanding, and any additional burdens for program management can seriously compromise it.

While the DRR date of June 2000 is unchanged from the 3 + 3 program structure, the nature of the decision expected from the initial DRR is significantly changed. There is no intent to provide detailed information leading to production of the radar or the GBI at that time. Instead, the decision will be limited to a site selection and award of the construction contract-both essential long-lead steps. Subsequent Defense Acquisition Boards (DABs) will be held to approve the remaining elements of the program. Hence, the DRR should be regarded more as a feasibility decision with some long-lead deployment actions rather than a readiness decision.

One performance criterion now specified for achievement before DRR includes two successful intercepts, with at least one occurring in an Integrated System Test (IST). The criterion clearly makes the DRR a performance milestone rather than a calendar milestone Still, the program guidance, in various forms, places great emphasis on the calendar milestone of June 2000. It will take strong program management and top-level support to ensure that the performance requirements and basic system engineering and design functions are not sacrificed to the calendar since no decision will produce successful deployment until the system can be shown to perform as required.

The panel noted that the authority of the JPO, the JPO system engineer and the program managers (PMs) are unclear. It does seem clear that the program's government managers do not have authority commensurate with the responsibility of running the program. The Contracting Officer's Representatives (CORs)-two from the Ballistic Missile Defense Organization (BMDO), two from the Army, and one from the Air Force-are told they are responsible to "ensure the contractor performs the requirements of the contract," but "may not make any commitments or changes that affect price, quality, delivery...of the contract."

Although the LSI reports directly to the JPO, three of the five CORs are from the Service program offices, and they report to their respective Service Program Executive Officers (PEOs). Thus, the authority of the JPO and the role of the JPO system engineer are unclear.

This lack of integrated authority and responsibility at the JPO is inconsistent with the complexity of the integration task that the JPO is overseeing.

At the time of this review, the relationship between the LSI and the individual program element program management was also unclear.

With the selection and start-up of the LSI, the JPO plans for all the element contractors to become subcontractors to the LSI. Hence, the LSI will run the EKV [exoatmospheric kill vehicle], XBR [X-Band Radar], Upgraded Early Warning Radar (UEWR), and BM/C3 [Battle Management/Command, Control and Communications] subcontracts. In the case of the EKV, the panel found that the LSI had not yet penetrated into the details of that program deeply enough to exercise oversight. The panel did not delve into the relationship with the other elements.

The LSI heavily emphasized the very challenging task of integrating the elements into a system-of-systems. However, the lack of government control, combined with the current state of LSI oversight of the subcontractors, leads to risks that can be mitigated by focusing more management on the performance of the elements to be integrated.

Given the set of challenges and the phased decision process, the JPO and LSI have formulated a sensible, phased, incremental approach to the development and deployment decision-while managing the risk. While the information required for decisions will come from ground testing, simulation, and flight testing, it is useful to describe the three major increments of the flight test program.

Through IFT-6 [integrated flight test], the program will focus on technology demonstration with the central objective of developing and demonstrating HTK capability. One requirement for passing DRR is two successful intercepts in four attempts on IFT-3 through IFT-6. IFT-3 and IFT-4 will rely heavily on Kwajalein Missile Range (KMR) radars and equipment to support the EKV in flight. However, beginning with IFT-5, prototype versions of the BM/C3, UEWR, and XBR [using the surrogate ground-based radar (GBR-P)] will be used in the test, thus making this test the first IST.

From IFT-7 through IFT-13, development and testing and demonstrations of system integration will continue. IFT-7 is the earliest that the operational booster can be part of the system for test. IFT-13 is planned to be the first flight test with the operational kill vehicle.

The remaining program will complete Engineering and Manufacturing Development (EMD) and increase confidence in the deployable system until a decision is made to deploy.

The current NMD effort is focused on being ready to deploy a system that meets the minimum requirements to meet the C1 threat-a challenging task using even the best available technology. Still, whether the decision is to continue development or to commit to deployment, technology development and testing will need to continue to meet future needs in expanding to a more sophisticated existing or developing set of threats. The panel found that the JPO and LSI were highly focused on developing a system to counter the C-1 threat with an IOC in 2005 but with minimum attention to planning for system growth and enhanced test capabilities that will be needed to support any enhanced system.

Many of these issues are discussed elsewhere in this report. We discuss only two of them here, and both are related to the facilities at KMR. The first issue is at KMR that there is currently only a single GBI [ground-based interceptor] launcher, yet the system will have to operate with two or more interceptors in flight. This dual-salvo feature cannot be verified with a flight test unless a second launcher is built. A second launcher should be planned and funded now. The second issue at KMR is the degradation of the GBR-P. There is reason for concern that it might be unable to support the test program by 2005 unless action is taken to maintain the capability. Taking a long-term view of NMD, the system will be undergoing tests for 10 to 20 years; hence, the problems with the GBR-P must be addressed now, and a long-term solution must be put in place.

The visit to the Raytheon facility in Tucson highlighted the impacts of the "hardware-poor" nature of the EKV program. There were no spares, no development articles, and no articles available for parallel activities that could significantly reduce development and test risk. The first article built appears to be the one that will fly.

Being hardware-poor has forced the EKV program to forego activities that would have reduced risk. For example, the early flight models in the EKV program will have hundreds of hours of use on them since these flight articles are also the developmental articles. The EKV program is unable to support important development activity, such as improving the test capability of the facility at Arnold Engineering Development Center (AEDC), since the flight test article had to be returned quickly to Tucson to meet schedules. Raytheon has also been unable to provide a test-article to the KKV [kinetic kill vehicle] Hardware-in-the-Loop Simulation (KHILS) facility for facility development and testing. The upcoming acquisition of the 1,000-Hz [hertz] rate table at KHILS could make that facility of high value in future NMD EKV development.

Finally, the lack of spare hardware is driving flight test delays. The inertial measurement unit (IMU) on the test article for IFT-3 failed. The EKV for IFT-4 could not be substituted. So, the IMU was removed from the IFT-4 EKV and installed into the EKV for IFT-3. The defective IMU was returned to the vendor for repairs, but it is not likely to be available for the planned IFT-4 launch date.

The NMD program is characterized by a combination of challenges that add up to severe demands on management. They include the urgent need (leading to a planned high-risk schedule), first use advanced technology, complex system architecture, compressed and concurrent schedules, long development times and long operational life span, complications from arms control agreements, high cost, and participation by a multitude of government organizations and the Services. These program characteristics generate some unique management issues that demand a high degree of clarity in assigning authority and responsibility. As noted in this report, instead of unusual clarity, there is unusual fragmentation and confusion about authority and responsibility.

While the flight tests are high visibility, the vital information needed for development and decisions (i.e., the DRR) will come primarily from the LSI Integrated Distributed Simulation (LIDS) and the IGTs [integrated ground test] anchored with data from the flight tests. However, the panel considers the resources currently available for simulation development and IGTs as inadequate to provide the information needed for the DRR on the planned date.

In fact, the LIDS' development and the execution of the IGTs are currently behind schedule. The plan to mitigate the risks associated with these delays did not provide much confidence to the panel. The Integrated System Test Capability (ISTC) is in heavy demand for the IGTs and the IFTs, and it cannot adequately support both. This is a cause for particular concern in the near term, where the IGT and IFT events have been compressed into a very small time window to support the June 2000 DRR. An expanded capability or some unidentified workaround will be required to support both.

The panel believes that the government and the integrating contractor continue to underestimate the challenge of reliably performing exoatmospheric HTK. Exoatmospheric HTK of ICBM reentry vehicle (RV)-like targets has been successfully demonstrated only twice [Homing Overlay Experiment (HOE) and Exoatmospheric Reentry Intercept Subsystem (ERIS)] with at least 8 failures to date [HOE, ERIS, and Lightweight Exoatmospheric Projectile (LEAP)]. The panel noted a lack of urgency to test the EKV in an end-game hardware-in-the-loop (HWIL) facility, given that most other BMDO HTK programs view this need as critical.

The panel did not think that including decoys on the first intercept attempt was a wise choice, and BMDO has recently elected to remove two of them. The panel views a successful HTK engagement as being difficult enough that the first attempts should not be unnecessarily complicated.

Finally, the panel was concerned about the single test geometry for IFT-7 through IFT-13. The LSI's plan to use the data from the flight tests to anchor the simulations is likely to be compromised seriously by the level of extrapolation that will be required to assess capability over the required flight envelope.

This reinforced the panel's judgment that the government and the LSI were more optimistic than warranted about the state of capability to achieve reliable exoatmospheric HTK capability.

This chart [not shown] provides some additional detail regarding the phased incremental approach described in the overarching observations. IDR, the Interim Design Review, is a prelude to the DRR. SCDR, the System Critical Design Review, is where the configuration of the system is fixed. The 2001 DAB is the decision to go forward with the procurement of the UEWR upgrades and the XBR. The 2003 DAB is the decision on the GBI-the final system decision.

The fact that the system configurations is not fixed until 2001 and the operational version of the EKV will not be tested until FY2003 lead the panel to conclude that the DRR should more properly be considered a Deployment Feasibility Review under the restructured program. If all goes according to the restructured plan, by June 2000, BMDO will have demonstrated the "feasibility" of a NMD system but not the "readiness to deploy" of the system. The demonstration of readiness to deploy will not come until 2003 at the earliest, when the integrated GBI (i.e., operational version of the booster and EKV) is to be demonstrated.

The extra time for more flight tests before the decision to deploy, which resulted in the phased approach to decision-making, lowered the risk in the restructured program.

This chart [see above] shows the change in schedule from the restructure. IFT-lA and IFT-2 were seeker fly-by tests and were completed successfully before the restructure. Three intercepts are to be attempted before the DRR, with IFT-5 being the required IST.

The RRFs [risk reduction flight] mentioned in the chart are tests in which targets of opportunity are tracked, allowing the system to operate, collect data, and simulate intercepts. These can occur on targets launched from Vandenberg AFB [air force base] and Kodiak Island.

After the three key IFTs, there are three key decision points. Following a successful IST, a commit to deployment resulting from the DRR would consist of a decision to select the interceptor site and award the construction contract. IFT-7 is the first flight of the interceptor using the operational booster. The subsequent DAB could approve the start of site construction and approve the early warning radar upgrades, the XBR, and the BM/C3 integration program.

There will be six additional intercept attempts through IFT-13, the first test flight with the operational EKV. The subsequent DAB is to approve the GBI build.

The decision criteria for the DRR specify performance requirements for design development, deployment and program cost. Details regarding the criterion of demonstration of integrated system/element functionality are shown on the next chart [not shown].

The program workload to generate the information for meeting these criteria is formidable. Numerous agencies, including NMD JPO, USD(A&T) [Under Secretary of Defense (Acquisition and Technology)], and the Director of Operational Test and Evaluation (DOT&E) in OSD, must work together to produce the reports demonstrating that the criteria have been met.

[The preceding two paragraphs are repeated for the next slide in the report, which is not shown.]

The information needed to develop and verify the system will come from an orderly combination of simulation, test, and analytic activities. Testing includes ground testing of individual system elements and of subsets of the integrated system. RRFs use targets of opportunity, such as Minuteman reliability launches. IFTs are designed to collect information that cannot be adequately addressed in ground testing. This combination of testing provides validation for the simulations and, in turn, provides confidence for development and deployment decisions.

Given the nature of the NMD mission and the inability to conduct full-scale end-to-end tests in a realistic environment, simulations and analyses will provide much of the needed design and decision information. The review group has urged-and continues to urge-extensive use of HWIL simulations to increase the validity of the simulations and confidence in the validity of the simulations.

As discussed earlier, the program has experienced significant ground and flight test compression since the restructure. IFT-3, the first flight test since award of the lead system integration contract, will be directed by the government and has slipped several months. IFT-4 will be the first IFT fully directed by the LSI, and it has also slipped several months. Since a successful intercept on IFT-5 is a prerequisite for DRR, any significant slip of that test must affect the DRR date.

Similarly, the IGTs are slipping as indicated on the chart [not shown], again compressing the schedule. Given the amount of analysis that must be accomplished based on these tests and the simulations they anchor, this compression is cause for major concern.

Both IGTs and IFTs require the use of the ISTC facility. With both slipping, the demand for the ISTC facility is likely to exceed its capabilities. This could result in even more schedule slips.

The schedule in this chart [not shown] shows that the system design will not be firm until the SCDR in FY01, yet prototypes of the EKV, UEWR, XBR, and BM/C3 are participating in the flight tests now. This high level of concurrency adds risk to the program by raising the possibility of future design and/or requirement changes. Such changes may not cause great harm, but they work against getting a true baseline system for production. Although block changes were said to be planned to resolve this challenge, we found no freeze points to catch specific numbers of flights and intercepts before decision points.

As part of the test program, the NMD program is conducting opportune RRFs. These are occasions in which the program takes advantage of a target that most likely might be launched from Vandenberg AFB toward KMR but possibly from other Pacific launch ranges (e.g., Kodiak Island). In these flights, the elements of the NMD system are turned on and operating. The tracking, classification, and BM/C3 element will operate along with the radars and portions of the interceptor element. However, the interceptor will not launch.

The JPO has been able to influence the payload selection for some of these flights to try to provide a realistic target set for the test. Five RRFs have occurred to date, and 11 are planned before the FY03 DAB decision.

The LIDS is vital to the approach for verifying the system design. It is a simulation that includes models of the individual elements of the system and uses inputs from the threat environment, and lethality models. It is designed to be fast-running so that it can perform thousands of runs to support the DRR. LIDS will use data from the IFTs and IGTs to verify its system descriptions.

Build 4 of the LIDS, despite its importance, is substantially behind schedule. Further, there is serious doubt that the LIDS, as currently configured, will have the capacity to support both IGTs and IFTs in the near term in the role of test-planning. Moreover, the data from the completed IFTs and IGTs must feed back into LIDS to anchor it to the test observables. Since the IFTs and IGTs are slipping, there will be less time before DRR for this process. Hence, the development and reliance on LIDS is rapidly becoming a high-risk portion of the program.

Raytheon performed extensive HWIL testing of the EKV. They used numerous facilities for calibration and characterization of the EKV sensor. They also used the Hover Test facility to test the propulsion system. However, no plans have been made to use the KHILS facility at Eglin AFB.

KHILS is acquiring new capabilities that could provide the opportunity for more realistic testing of the seeker and guidance on the EKV (e.g., a 1,000-Hz rate table that will provide the response necessary to model the EKV). These new capabilities would provide realistic, dynamic scenes and allow the seeker to operate and close the guidance loop all the way to intercept. This type of testing would complement the other HWIL tests being conducted by Raytheon and verify EKV performance in a variety of conditions that cannot be tested at KMR.

As discussed previously, environmental testing of the GBI is a major concern.

Extensive discussions with Raytheon did not provide the panel confidence as to when and how it will be determined whether the EKV will be able to withstand the environments on the new GBI booster. We view this, as does Raytheon, as a high risk to the program. We will discuss this topic later in the report.

The top program risks identified by the LSI [see chart above] are dominated by schedule concerns-reinforcing our conclusion that further compression of events leading to the DRR must be avoided. Instead, the panel recommends slipping the DRR date to accommodate any further delays in key events leading up to the DRR.

The panel was also concerned that the LSI list is dominated by large-scale integration issues. While integration is a challenging task, very difficult element challenges also remain at this stage of development, and the orientation of this list adds to our concerns that the LSI has not yet had time to penetrate the element programs fully.

As expected, the EKV makes the list of high risks for technical and other reasons. However, the panel, in response to questions, was concerned about the lack of identified particulars concerning why the risk was thought to be so high. The panel concluded that no firm risk mitigation plan could be implemented at present to drive down the high risk areas.

For several key risk areas, the planned risk mitigation effort is the flight test that will be impacted by the risk. Once again, the panel strongly urges a greater effort to minimize and close the risks by using ground testing. Flight tests should be used to close only those risks that cannot be tested and verified on the ground.

This chart [not shown] shows the medium- and high-risk areas in the EKV program, as identified by Raytheon. The risk management program at Raytheon appears to be well conceived, well structured and well executed.

On the next three charts [not shown], we discuss further the risks associated with the IMU, the high environmental loads produced by the new GBI booster, and the sensor schedules for EKV integration.

One of the highest EKV risk areas is its ability to withstand the environmental loads of the new booster that will fly on IFT-7 This chart [not shown] shows that the expected shock loads are over an order of magnitude greater than those imparted from the PLV [payload launch vehicle]. We are not certain that the EKV will be able to withstand these loads, and we will not know with any degree of certainty until IFT-13, the flight on which the EKV and new booster are first mated together. This flight is planned for 2003.

The recently failed IMU that was planned for the EKV on IFT-3 has led to schedule slips in IFT-3 and possibly IFT-5. Because no immediate spare was available, the IMU in the EKV planned for IFT-5 was removed and installed into the EKV planned for IFT-3. The defective IMU was returned to the manufacturer for repairs. Because the manufacturer has discontinued making IMUs, a new one cannot be substituted. Rather, Raytheon must wait for the repairs to be made. If delays occur, this might result in delays to IFT-5. This risk is directly attributable to inadequate developmental and spare hardware.

This chart [not shown] addresses the EKV sensor delivery schedule challenge. Deliveries were behind schedule even before the IMU-related schedule slips were announced in late June 1999.

There is still a possibility that sensor delivery can support the current schedule for IFT-4 and IFT-5. However, the schedule margin has been consumed. Any further schedule slip or compression will jeopardize the DRR date.

On this chart [see below], we show the risk mitigation plan for items in IFT-5. In the event that any of the items on the left fails during the test, the backup is shown on the right. For example, if the GBR-P failed during the test, the use of the Global Positioning System (GPS) and other GBRs would still allow the EKV to fly and attempt an engagement. Thus, some of the test objectives could still be salvaged, although the goal of demonstrating the integrated system would not be met.

The EKV and the command launch equipment are both high risk and represent single points of failure. The acquisition of a second launcher at KMR would mitigate some of this risk.

The panel identified several risks in addition to the risks raised in the earlier review and those raised by either BMDO or the LSI. The first risk results from the state of contract negotiations between the LSI and some of its major subcontractors. The JPO is in the process of turning over the contracts for all the elements to the LSI. However, some contracts have not yet been finalized. This leads to higher risk because the LSI does not have detailed insights into some of the elements or full control.

While BMDO is currently responsible for the conduct of IFT-3, continued slips could lead to a decision to assign responsibility to the LSI. A hand-over this close to the flight test carries added risk.

The NMD Test and Evaluation Master Plan (TEMP) is still in draft form. The document should be expeditiously completed and approved.

The panel is also concerned about the limited flight test exploration of the flight envelope. Range constraints at KMR limit testing to very few geometries and intercept conditions. For example, a single intercept geometry will be used for IFT-3 through IFT-6, and a second geometry will be used for IFT-7 through IFT-13. From this limited set of data, the LSI hopes to anchor its simulations to prove the system's ability to handle the required threat envelope.

The panel is concerned about the scope of extrapolations required.

Findings and Recommendations

The program has been restructured from the earlier 3 + 3 with a DRR for all elements of the system in July 2000 to a series of phased deployment readiness decisions leading to a most likely capability to deploy in 2005. This restructure significantly reduced program risk.

• However, key event slips have compressed the planned restructured schedule.

RECOMMENDATION: Do not allow further compression of the schedule. If there are additional slips in key events, adjust the DRR date as needed to avoid regressing to a very high risk schedule.

• Several key program issues had not been adequately addressed at the time of this review.

RECOMMENDATION: LSI (Boeing) contract should be updated to reflect clearly LSI responsibility and accountability.

RECOMMENDATION: An updated TEMP should be approved and implemented.

• The information from the events planned up to the DRR could provide confidence that deployment in the planned time frame (2005) is feasible and that the nation could continue on the path to deployment. Readiness to deploy will be determined in phases, based upon events following the DRR. Key components are to be tested just before each phased milestone decision and are the decision drivers.

RECOMMENDATION: Consider DRR event as a system development feasibility review rather than a deployment readiness review.

• High concurrency exists in the requirements, program definition, and system engineering planning-driven by the DRR date.

RECOMMENDATION: Recognize that the first DRR milestone is a deployment feasibility milestone since the decision process is phased over decision milestones through 2003.

• The LSI-described system for risk assessment and mitigation is extensive and detailed but seems to focus primarily on the higher level integration issues.

RECOMMENDATION: In addition to the current emphasis, the LSI should penetrate more deeply into the element technical performance risks to identify those risks that can be most damaging to cost, schedule, and performance. Specific actions for mitigating these risks should also be identified.

• The IFTs are essential events, but the IGTs and the LIDS will be the principal sources of comprehensive information and understanding needed for program development and milestone decisions. IGT and LIDS' development are behind schedule and do not appear to be resourced adequately to ensure timely support.

RECOMMENDATION: Increase the priority given to resourcing IGTs and LIDS' development.

• The ISTC is overscheduled, and the IFTs will inevitably out-prioritize the IGTs for ISTC resources. Full ISTC support for the IGTs is essential.

RECOMMENDATION: Consider expanding the ISTC capability now.

• LSI penetration into the details of the EKVnbsp program seemed inadequate.

RECOMMENDATION: LSI should audit all EKV ground test results to ensure that EKV meets all flight environments-with the PLV (BV) [boost vehicle] and the operational booster-with substantial margins.

• The flight test envelope is small relative to that of the operational system.

RECOMMENDATION: Consider using additional target platforms (air-dropped or sea-launched) to gather expanded operational testing data on the XBR and variations in terminal conditions.

• The lack of representative targets and single end-game geometry raises questions about the ability of the flight-test program to verify system performance.

RECOMMENDATION: Accelerate the development of a HWIL facility to allow credible testing of the EKV in a variety of end-game geometries against a variety of threats. Strongly consider either KHILS with the 1,000-Hz rate table or expanded capabilities at AEDC.

• The panel believes added long-term focus is needed for this long-term evolutionary system.

RECOMMENDATION: Allocate resources for the short term and the long term. For the longer term:

• Ensure GBR-P will maintain performance.

• Erect a second launcher at KMR to allow dual-salvo testing.

• Immediately design and invest in an adequate HWIL test facility.

• Aggressively pursue technology development for upgrades and follow-ons.

• Determine the need to invest in a second or expanded ISTC now.

• Develop facilities and/or capabilities to evaluate C2-level [command and control] threats now.

• Procure sufficient EKV hardware to support needed facility development, ground testing, and flight test backups.

Acronyms and Abbreviations

<!--#include virtual="/linked/foot.inc"-->