Medium-class Explorers (MIDEX)
Lessons-Learned Workshop
Hampton, Virginia
June 26 -27, 1996
Final Report
August 1996
Table of Contents
Introduction............................................................................................. 1
Introduction An Announcement of Opportunity (AO) for the first Medium-class Explorer (MIDEX) missions was issued in March, 1995 (AO-95-OSS-02). This AO included, for the first time in the Explorer program, several features that were intended to facilitate the preparation and review of flight investigation proposals: two-step procurement process, an option for a Principal Investigator (PI)-mode, and strict adherence to cost caps. In addition, the AO addressed new programmatic initiatives for technology, education and outreach, and small and disadvantaged businesses. After the selections under the AO, the Associate Administrator for Space Flight, Dr. Wesley Huntress, requested that a Workshop be conducted to solicit feedback on the process from the community it served. Participants in the MIDEX AO process, including proposal team members, reviewers, and NASA program management personnel were invited to participate. Other interested parties were also invited and did attend. The Workshop addressed various aspects of the solicitation and selection cycle and resulted in a set of recommendations for NASA to consider for future Explorer AO cycles. Four areas were identified by Workshop participants as requiring focused attention. These were the two-step selection process, the dual-mode spacecraft options offered in the AO, the AO selection criteria, and cost assessment. The recommendations in these four areas follows the Agenda in this report. These recommendations were first formulated by separate splinter groups, then presented to a plenary session and endorsed by all attending the workshop. Following the recommendations are the Chairman’s comments. Kenneth Lang, Professor of Physics and Astronomy at Tufts University, served as the very able Chair of the Workshop. Professor Lang, who is an experienced space scientist, had no prior involvement in this particular AO process. Including in Appendices are the list of attendees, a copy of the letter of invitation to the Workshop, and Workshop minutes. The Workshop was conducted on June 26 and 27, 1996 at the Holiday Inn Conference Center in Hampton Virginia. It was sponsored by the NASA Langley Research Center with support from Jorge Scientific Corporation.
Wednesday, June 26, 1996
8:30-8:45 Introduction and Purpose D. Bohlin, NASA
8:45-9:15 Agenda, Scope of Workshop K. Lang, Tufts
9:15-9:30 MIDEX AO Genesis J. Lintott, NASA
9:30 Science Community Perspectives M. Janssen, JPL P. Evenson, Del N. Johnson, NRL R. Paulos, Wisc. B. Mauk, APL J. Raitt, Utah State D. Rust, APL B. Dennis, GSFC
Industry Perspectives S. Savarino, TRW L. Di Biasi, OSC W. Gail, Ball
Govt./FFRDC Perspectives Mike Janssen, JPL Larry Crawford, APL
1:00 Summary of Written Comments J. Lintott
1:30 Splinter Group Objectives and Assignments K. Lang
2:00 Splinter Group Sessions 1. Two-Step Process 2. Dual-Mode Spacecraft 3. Selection Criteria 4. Cost Accounting
Thursday June 27, 1996
8:30 Plenary Session: Splinter Group Reports and Discussion
12:00 Conclusions K. Lang D. Bohlin
Chairman's Conclusions
Representatives from academic, industrial, and government institutions met to discuss the strengths and weaknesses of the recently completed Medium-class Explorer (MIDEX) Announcement of Opportunity (AO-95-OSS-02). Participants in this workshop were experienced and well-informed, but many felt that they had not been well-served by the recently completed MIDEX AO process. In the course of the meeting it became clear that there had been many misunderstandings and a good deal of frustration. NASA representatives indeed accepted the need for comment and community input, hence the workshop took place.
The workshop participants were divided amongst themselves as to the best approach, resulting in some final recommendations that could be contradictory. Nonetheless, everyone agreed that the exercise was valuable and all the final recommendations are being seriously considered by NASA.
Participants reaffirmed their support and enthusiasm for the MIDEX program, and focused on methods for improving the evaluation process.
The meeting was carried out in three stages:
Stage 1. Presentations by twelve participants who were team members of proposals that did not win the recent MIDEX competition. Their comments, which are reproduced in this report (pages C-2 to C-21), provide an objective assessment of the positive and negative elements of the evaluation process, from the perspectives of the science, industry, and government communities. This was followed by a summary of written comments sent to NASA before the workshop, and also reproduced in this report (page C-22).
Stage 2. Four focal points were identified for further discussion, and chairpersons assigned. The assignments and objectives for these splinter groups are provided in the meeting minutes (on pages C-24 and C-25 of this report). The four groups met individually for several hours, and then met collectively in a first plenary session. The four chairpersons subsequently met late into the first night to identify the primary issues, remove overlap, and agree upon recommendations.
Stage 3. The suggestions resulting from Stage 2 were further distilled and endorsed by all attending in another plenary session during the morning of the second day. Each splinter group chairperson presented the relevant ideas, which were corrected, discussed, and eventually confirmed in an open group process. The resulting recommendations are also reproduced in this report (on pages 6-13).
Concluding Synopsis of Suggestions
An abridged overview of the four splinter group suggestions for further improving the MIDEX selection process follows; this reflects the Chairman's interpretation of the events and is not meant to portray all of the group perspectives.
A. Two-Step Selection Process.
The initial evaluation should depend only on science and the technology required to do that science. NASA's interpretation of community scientific priorities should be introduced during this initial evaluation, and the number of proposals continuing to further evaluation of cost and management should be reduced to no more than four per final selection. This will minimize the discontent and total cost to the proposing community. Changes in science goals and instrumentation should not be permitted after the initial evaluation.
B. Dual-Mode Selection Process.
Only one mode of proposing should be allowed, in which all proposals are treated completely equally on a level playing field. To avoid inequities, NASA should provide proposal support for cost, management, and technical issues when proposers need it; such support should not depend on inclusion of a NASA Center in the proposal. Discontent can be avoided by clearly stating, in the AO, the expected extent of Goddard Space Flight Center (GSFC) involvement in all aspects of the MIDEX program. NASA should be viewed as the contractor of last resort, providing key links to make otherwise feasible missions possible.
C. Selection Criteria.
The initial evaluation should include an assessment of existing and new technology by qualified experts. Some form of interactive feedback should be established between panel reviewers and proposers to avoid misinterpretations and misunderstandings. A common complaint was that previous evaluation of technical risk and science objectives could have been misunderstood by scientific reviewers who were not experts in the relevant area.
D. Cost Assessment.
There is overwhelming support for full cost accounting of all NASA contributions to, and participation in, flight programs. This should be accomplished without grandfathering or additional costs to the Explorer program. NASA should consider foreign, non-NASA domestic, and non-Explorer NASA contributions to the MIDEX program, with a cap to the level of foreign contribution.
To summarize, some of the most qualified members of the science, industry, and government communities have carried out a democratic and honest evaluation of the MIDEX selection process. They have identified seminal issues, and provided recommendations that will undoubtedly lead to improvements in the MIDEX program in general and the MIDEX AO in particular.
We are grateful to NASA for its open responsiveness to the external communities that it serves, and for providing this opportunity to help retain the utmost integrity in its selection processes.
Signed,
____________________________ Professor Kenneth R. Lang Tufts University Chairman, MIDEX Lessons-Learned Workshop
Recommendations of the MIDEX Lessons Learned Workshop
I. Two-Step Process
Chair: Michael Janssen, Jet Propulsion Laboratory
The overall goal of the selection process must be to obtain the best science for missions chosen within MIDEX program constraints. To effectively attain this goal, however, two conditions must be met:
The consensus of the workshop was that the two-step process that was used fell short of these conditions. It was felt that much unnecessary effort was spent in the development of proposals. Dissatisfaction was expressed by both industry representatives and proposers on several other aspects of the process as well.
Specific problems that were identified are:
The importance of and requirement for cost evaluation expressed in the AO for Step One and as interpreted by proposers were treated differently in the Step One
reviews.
Reviewers seemed to misunderstand key elements of some proposals.
Specific complaints were directed toward the evaluation of technical risk and cost realism by the Step One review panels.
Recommendations:
Overall, a two-step process is favored, with suggested improvements:
• It is recommended that NASA allow reviewers to ask questions of PIs during the review process to clarify issues.
A common complaint of the MIDEX selection process was that reviewers missed or misinterpreted key aspects of the proposals and
that no mechanism was available for correcting this review deficiency. An oral or written exchange between reviewers and proposers
would allow the reviewers to determine whether perceived weaknesses are true weaknesses, or just matters of misinterpretation. The
workshop members recommend such interactions for the Step Two process. Some workshop members recommend that such interactions be
carried out during the Step One process. The objections that some other members have to the latter recommendation are related to
implementation; i.e., how onerous would such a process be for some 40 or so proposals and how would panel member anonymity be
maintained?
One suggestion follows as to how the proposer/reviewer interactions would be carried out (for either Step One or Step Two). All proposers would be provided by e-mail drafts of the respective panel reviews, including all perceived deficiencies, and the proposers would be asked to respond with a less than one page (TBD) response within 24 (TBD) hours.
• It is recommended that the Step One cost proposal be replaced by a simple assertion by the PI that the mission costs,
including contingency, will be less than the cost cap of $70m. Discriminations between proposals based on the total costs would occur
only during the Step Two evaluation.
The Not-To-Exceed (NTE) cost criterion of the present MIDEX AO requires a robust and perhaps comprehensive costing activity on the part of the Step One proposers. Thus, not much money is saved by the two-step process. Also, during the previous selection activity, there is concern that the Step One selection panel passed judgments on costs that the panel did not have the resources or the expertise to make. Thus, we recommend that cost be taken out of the Step One selection criteria. If NASA cannot accommodate this recommendation, then the Step One review process must include the time, expertise, and resources to adequately evaluate cost credibility.
• The Step One review process should strongly emphasize science and the AO should be rewritten to reflect this
emphasis.
As the Step One panel searches for the proposals that have the absolutely highest ranked science goals, there is the temptation to trade the science goals off in favor of missions that are perceived to have somewhat lower technical or cost risks. The AO in fact (p. 13) gives heavy weight to the latter in the Step One review criteria. Because the science panel does not necessarily have the expertise or resources to fully evaluate cost and technical issues, it is recommended that the science panel evaluate only the science merit and that the science merit grade be decoupled from the technical and cost evaluations to the extent possible. The AO should be written to reflect this change of emphasis. To the extent that it is deemed necessary to evaluate technical and cost realism during Step One, adequate resources should be made available to the review team.
• The number of candidates selected to submit proposals for Step Two be limited to a maximum of four per final
selection.
Once the number of viable proposals is reduced to this level, all are in the category of excellent science, have a well-defined management approach, and are doable within the cost and schedule constraints. Four per final selection provides sufficient margin to account for unforeseen programmatic circumstances that may occur during the evaluation period that would be cause for the elimination of a proposal while still leaving excellent proposals that would vie for the selection as primary or backup missions. The selection of the finalist then becomes a matter of finding the very slightest discriminators. It will be difficult enough finding these discriminators. Any more than four places unnecessary cost, time and resource burdens upon other PIs and their teams.
• Apply the programmatic filter in Step One to achieve this.
The selection of the best science among discipline panels should be done as part of the Step One review rather than at the end of Step Two. This appears to be an essential step if the Step One down-selection recommended above is to be achieved. The specific way in which this recommendation may be implemented is beyond the purview of this workshop and is put forth as an objective for NASA to achieve.
• Speed up the process to 6 to 8 months from AO release to selection.
Shortening the duration of the selection process forces a streamlining of the proposal development effort, which should reduce overall costs while promoting continuity between the Step One and Step Two reviews.
• Provide written evaluations.
It was deemed highly desirable that written instead of oral debriefings be presented at the end of both Step One and Step Two to both winners and losers.
• NASA should consider providing nominal funding for proposals that are advanced to Step Two.
A suggested amount is $100K. Currently some proposers are at a disadvantage relative to others at well-supported
institutions and cannot supply the detail necessary to compete in the Step Two evaluation. Nominal funding will further help level
the playing field.
II. Dual Mode Option
Chair: Paul Evenson, University of Delaware
Issues:
Recommendation:
A binary choice of investigation mode should be replaced by a single mode. A management approach should be proposed by the investigator as part of the mission design.
Comments:
A detailed list of services available from NASA should be included in the AO. This should include management services in addition to more traditional items such as tracking, integration, test, spacecraft systems, or even complete spacecraft.
Detailed consultation with NASA on the use and costing of these items should be part of the preparation of the Step Two proposal. GSFC should develop a point of contact for these discussions and a standard procedure for costing these products and services.
The name of the single but flexible investigation mode should be chosen with care. The term "PI mode" carries with it a certain connotation within NASA, such that continued use of the term might cause confusion.
It is the express intent of this recommendation that a proposer could specify any component of a mission (such as a particular commercially provided spacecraft) but could at the same time request NASA management of the mission.
It would be appropriate for the AO to contain examples of management approaches, but the proposer should not be required to adopt one of the examples.
Issue Relating to the Scope of the Explorer Program:
As expertise in building and operating spacecraft has improved, the provision of space platforms is being transferred to commercial ventures. This has already happened at geosynchronous orbit, which now accommodates many sophisticated communication satellites. Commercialization of space is about to occur at LEO altitudes as the funding is established to implement large constellations of communication satellites for telephony and data transfer between any two or more points or the earth’s surface. Commercial platforms provide an opportunity to perform certain types of experiments at low cost and with unique features.
By purchasing space and resources from the commercial operators NASA could provide opportunities for certain experiments that could not be economically done without the leveraged advantage of the commercial money invested in the program. On the other hand the funding provided by NASA would represent a government investment in systems to help keep the leadership role the USA holds in space activities.
Recommendation:
NASA should find a means of selecting and funding instrument investigations to take advantage of launch targets of opportunity. This should include the possibility of purchasing space on commercial satellites.
Science Recommendations:III. Selection Criteria
Chair: Shirley Savarino, TRW
New Technology:
Recommendations:
General/Evaluation Criteria:
General Recommendations:
IV. Cost Assessment
Chair: Larry Crawford, Johns Hopkins University Applied Physics Laboratory
Issue: All NASA center contributions to flight programs should require full cost accounting for all personnel, services, and hardware.
Comment: It has been stated by NASA that full cost accounting for all NASA contributions to flight programs will be implemented soon. The question is: How much grandfathering will be permitted? (i.e. Will missions designed under the old rules be able to avoid full cost accounting?)
Recommendation: Full cost accounting for all NASA contributions to and participation in flight programs should be implemented. Further, this should not be accomplished at the expense of science (i.e. the amount of dollars available to the Explorer Program should not be correspondingly reduced). No grandfathering should be permitted.
Issue: Why doesn’'t NASA permit foreign contributions outside of the program cap?
Recommendation: NASA should permit foreign contributions to be included in proposed efforts, but require proposer to provide contingency for foreign risk. NASA to provide cap for foreign contributions. Further, don’t subtract the dollar value of the foreign contribution from the program cap.
Issue: The AO cost evaluation approach does not allow for different costing methodologies to be proposed (e.g. it is required to explicitly list cost reserves in proposals).
Recommendation:Allow proposers freedom to justify their handling of contingency (cost reserves); don’t force them to explicitly carry contingency, but require them to explain their approach.
Issue: NASA’s current approach to cost assessment during proposal evaluations relies primarily on cost model outputs. Why doesn’'t NASA adopt the new government guidelines based on the contractor’s past cost performance?
Recommendation: NASA should review the contractor past cost performance-based Federal Procurement Policy guidelines (Federal Report 0014-9063/95; Vol. 63, No. 18; page 573) and issue a NASA policy and position on their use in the source selection process.
Comment: A possible approach that combines cost estimating model outputs and a contractor’s past cost performance in the cost assessment evaluation process is shown on the Figure below.
Appendices
APPENDIX A
Attendees
Jay T. Bergstralh NASA Headquarters Code SR Washington, DC 20546 phone: 202-358-0313 fax: 202-358-3097 e-mail:
Gregory D. Berthiaume Massachusetts Institute of Technology Lincoln Laboratory 244 Wood Street Lexington, MA 02173-9108 phone: 617-981-4903 fax: 617-981-0969 e-mail:
J. David Bohlin NASA Headquarters Code SR Washington, DC 20546-0001 phone: 202-358-0880 fax: 202-358-3987 e-mail:
Michael L. Cherry Louisiana State University Department of Physics and Astronomy Baton Rouge, LA 70803-4001 phone: 504-388-8591 fax: 504-388-1222 e-mail: phcher@lsuvax.sncc.lsu.edu
Larry Crawford Applied Physics Laboratory Space Department 11100 Johns Hopkins Road Laurel, MD 20723 phone: 301-953-5193 fax: e-mail:
Brian R. Dennis NASA/Goddard Space Flight Center Code 682 Greenbelt Road Greenbelt, MD 20771 phone: 301-286-7983 fax: 301-286-1617 e-mail: brian.dennis@gsfc.nasa.gov
Lamont Di Biasi Orbital Sciences Corporation Business Development 20301 Century Boulevard Germantown, MD 20874 phone: 301-428-6610 fax: 301-428-6641 e-mail:
Mary S. Di Joseph NASA/Goddard Space Flight Center Code 490 Greenbelt, MD 20771 phone: 301-286-0118 fax: e-mail:
Paul A. Evenson University of Delaware Bartol Research Institute 222 S Chapel Street Newark, DE 19716-4793 phone: 302-831-2988 fax: 302-831-1843 e-mail: penguin@bartol.udel.edu
Cynthia Faulconer Lockheed Martin Astronautics Mail Stop S8110 PO Box 179 Denver, CO 80201 phone: 303-977-9277 fax: e-mail:
Orlando Figueroa NASA/Goddard Space Flight Center Code 701 Greenbelt, MD 20771 phone: 301-286-4489 fax: e-mail:
Dennis Finley TRW Space Technology Division 515B 14320 Sullyfield Circle Chantilly, VA 22021 phone: 703-802-1918 fax: e-mail:
Peter G. Friedman California Institute of Technology MS 424-47 Pasadena, CA 91125 phone: 818-395-6786 fax: e-mail: peterf@valkyrie.columbia.edu
William B. Gail Ball Aerospace Systems Engineering Communication Systems Division PO Box 1062 Broomfield, CO 80038 phone: 303-939-4418 fax: 303-939-4430 e-mail: bgail@ball.com
Tim Gehringer NASA/Goddard Space Flight Center Code 790.4 Greenbelt, MD 20771 phone: 301-286-6831 fax: e-mail:
David Gilman NASA Headquarters Astrophysics Division Washington, DC 20546-0001 phone: 202-358-0349 fax: e-mail:
Lisa Guerra NASA Headquarters Code B Washington, DC 20546 phone: 202-358-2513 fax: e-mail:
John B. Hall, Jr. NASA/Langley Research Center MS 328 Hampton, VA 23681-0001 phone: 804-864-1742 fax: 804-864-1975 e-mail: john.b.hall.jr@larc.nasa.gov
F. Rick Harnden, Jr. NASA Headquarters Code SR Washington, DC 20546-0001 phone: 202-358-0351 fax: 202-358-3096 e-mail: frh@hq.nasa.gov
Andrew Hunter NASA Headquarters Code B Washington, DC 20546 phone: 202-358-2514 fax: e-mail:
Michael A. Janssen Jet Propulsion Laboratory MC 169-506/Space Sciences Division 4800 Oak Grove Pasadena, CA 91109-8001 phone: 818-354-7247 fax: 818-354-8895 e-mail: michael.a.janssen@jpl.nasa.gov
W. Neil Johnson Naval Research Laboratory Code 7651/Space Science Division 4555 Overlook Avenue, SW Washington, DC 20375-5352 phone: 202-767-6817 fax: 202-767-6473 e-mail: johnson@osse.nrl.navy.mil
Kenneth J. Johnston US Naval Observatory Code SD 3450 Massachusetts Avenue, NW Washington, DC 20392-5420 phone: 202-762-1513 fax: 202-762-1461 e-mail: kjj@astro.usno.navy.mil
Thomas C. Jones NASA/Langley Research Center MS 431 Hampton, VA 23681-0001 phone: 804-864-7037 fax: 804-864-7202 e-mail: t.c.jones@larc.nasa.gov
Kenneth R. Lang Department of Physics and Astronomy Robinson Hall Tufts University Medford, MA 02155 phone: 617-627-3390 fax: 617-627-3878 email:
John A. Lintott NASA/Langley Research Center Code S Support Office Mail Stop 160 Hampton, VA 23681-0001 phone: 804-864-9864 fax: 804-864-8894 e-mail: j.a.lintott@larc.nasa.gov
Barry H. Mauk Applied Physics Laboratory Space Department 11100 Johns Hopkins Road Laurel, MD 20723-6099 phone: 301-953-6023 fax: 301-953-6670 e-mail: barry_Mauk@jhuapl.edu
Arlene A. Moore NASA/Langley Research Center MS 365 Hampton, VA 23681-0001 phone: 804-864-4407 fax: 804-864-8671 e-mail: a.a.moore@larc.nasa.gov
Jayant Murphy Johns Hopkins University Department of Physics Baltimore, MD 21218-2695 phone: 410-516-7027 fax: e-mail:
Paul J. Ondrus NASA/Goddard Space Flight Center Code 510.1 Greenbelt Road Greenbelt, MD 20771 phone: 301-286-9858 fax: e-mail:
Pete Partridge Applied Physics Laboratory Space Department 11100 Johns Hopkins Road Laurel, MD 20723 phone: 301-953-5623 fax: e-mail:
Robert J. Paulos University of Wisconsin, Madison Space Science and Engineering 1225 West Dayton Street Madison, WI 53706 phone: 608-263-6729 fax: e-mail:
Cynthia A. Peslen NASA/Goddard Space Flight Center Code 406 Greenbelt, MD 20771 phone: 301-286-5917 fax: e-mail:
W. John Raitt Utah State University Center for Atmospheric and Space Sciences SER Building Room 250A Logan, UT 84322-4405 phone: 801-797-2849 fax: 801-797-2492 e-mail: raitt@cc.usu.edu
R. W. Richie NASA/Langley Research Center Code S Support Office Mail Stop 160 Hampton, VA 23681-0001 phone: 804-864-9863 fax: e-mail:
Chris Roberts CTA Incorporated 1521 W. Branch Drive McLean, VA 22102 phone: 703-883-1017 fax: e-mail:
David M. Rust Applied Physics Laboratory Johns Hopkins University Johns Hopkins Road Laurel, MD 20723-6099 phone: 301-953-5414 fax: 301-953-6670 e-mail: david.rust@jhuapl.edu
Shirley A. Savarino TRW Spacecraft Technology Division Civil & International Systems One Space Park R9/1076 Redondo Beach, CA 90278 phone: 310-814-6366 fax: 310-813-3457 e-mail: shirleysavanno@gmail4.sp.trw.com
James Thieman NASA/Goddard Space Flight Center Code 633.2 National Space Science Data Center Greenbelt, MD 20771-0001 phone: 301-286-9790 fax: 301-286-1771 e-mail:
Melville P. Ulmer Northwestern University Department of Physics and Astronomy 2145 Sheridan Road Evanston, IL 60208-2900 phone: 847-491-5633 fax: 847-491-3135 e-mail: m-ulmer2@nwu.edu
Peter W. Vedder Omitron Incorporated 6411 Ivy Lane, Suite 600 Greenbelt, MD 22209 phone: 301-474-1700 fax: 301-345-4594 e-mail: peter.vedder@omitron.gsfc.nasa.gov
Richard I. Weiss NASA/Goddard Space Flight Center Code 410 Greenbelt, MD 20771 phone: 301-286-7493 fax: e-mail:
Theodore B. Williams Rutgers University Physics and Astronomy Department Piscataway, NJ 08855 phone: 908-445-2516 fax: 908-445-4343 e-mail: williams@fenway.rutgers.edu
APPENDIX B
Letter of Invitation
SR
Dear Colleague:
You are cordially invited to attend a Medium-class Explorer (MIDEX) Lessons-Learned Workshop to be held June 26 and 27, 1996, at an appropriate hotel in Hampton, Virginia. Dr. Wesley Huntress, Associate Administrator for the Office of Space Science, has requested that this Workshop be conducted to focus on the recently completed Announcement of Opportunity (AO-95-OSS-02) solicitation and selection cycle. The Workshop is open to all MIDEX proposers, including their proposal teams, science and technical peer reviewers, and NASA program management. The Workshop will begin at 8:30 am on both days. Prof. Kenneth R. Lang of the Department of Physics, Tufts University, will serve as the Workshop Chair. The results of the Workshop, both positive and negative, will be captured, adjudicated and approved by the Chairman.
The objective is to identify both the strengths and weaknesses of the process up to this point, with the goal of improving the process for the future Explorer AO cycles, including those for Small Explorers (SMEX) and MIDEX. NASA plans to release a SMEX AO this fall and the next MIDEX AO in the spring.
The Workshop is scheduled for a full day on June 26 and the morning of June 27. Given this tight schedule, we want to focus the Workshop on the areas of highest interest. Therefore, we are soliciting your written comments, suggestions, and lessons learned about the recently completed MIDEX AO process in advance. Your comments, which will be held anonymously, will be used to establish a final agenda that will be distributed prior to the Workshop. Your written comments should be submitted by June 10 to:
Mr. John Lintott Code S Support Office Mail Stop 160 Langley Research Center Hampton, VA 23681-0001 Phone: (804) 864-9850 FAX: (804) 864-8894 email: j.a.lintott@larc.nasa.gov
It is anticipated that after introductory remarks, the first day will be devoted to open presentations followed by splinter working group sessions — into the evening if necessary. The splinter groups would report the results of their deliberations at a plenary session on the morning of the second day. Specific areas for consideration include, but are not limited to, the following:
• Overall clarity and content of the Announcement of Opportunity
• Evaluation Criteria
• Preproposal Conference
• Evaluation Process including two-step vs. a one-step process
• Debriefings
You may, if you wish, submit private communications in addition to, or in lieu of, your personal attendance. These comments should be addressed to Mr. Lintott at the address above or to me at the following address:
Dr. J. David Bohlin Science Program Executive for Review and Evaluation Code SR NASA Headquarters Washington, DC 20546 Fax: (202) 358-3097> email: jbohlin@hq.nasa.gov
Funds to support this meeting are limited to those from academic institutions who were either proposing as MIDEX PI’s or those who served as science and technical reviewers. Government per diem at the local Hampton rate and reimbursement of travel expenses will be provided to these individuals only. A block of rooms will be reserved for this Workshop. Detailed information on logistics, travel, accommodations, and reimbursements will follow shortly from Ms. Bonita Hawkins of Jorge Scientific Corporation. She can be reached on (202) 554-2775 if you have any immediate questions.
Sincerely,
J. David Bohlin Science Program Executive for Review and Evaluation Office of Space Science
APPENDIX C
Minutes of the Meeting
Wednesday, 26 June, 1996
Dr. J. David Bohlin opened the meeting and described his role in the Medium-class Explorer Mission (MIDEX) process. He had not been the selecting official but was now in charge of ensuring the integrity of the process, as a keeper of the corporate knowledge. The purpose of the meeting is to identify the strengths and weaknesses of the process to see how things might be improved in the future. We need to be forward looking. NASA is not ignorant of the imperfections in the MIDEX AO. It was clear as we went along and requirements changed that there were impacts on the AO. The results of this meeting will be applied to the upcoming Small Explorers (SMEX) AO due for release next fall. However, there is no guarantee that all inputs can be taken into account; NASA is working within certain boundary conditions.
Professor Ken Lang then outlined the purpose of the workshop: to review the strengths and weaknesses of the solicitation and selection process for the MIDEX Announcement of Opportunity (AO-95-OSS-02). He asked the participants to review what went right, what went wrong, to ask themselves how "right" and "wrong" might depend on a particular perspective, such as certain scientific disciplines, academia in general, industry, and government. This also needed to be viewed in light of NASA's stated objectives of achieving low-cost research opportunities in space.
On the basis of written comments submitted prior to the workshop, Prof. Lang suggested splinter groups should consider the following aspects of the process: the two-step process; the dual-mode selection process; the selection criteria; and full cost accounting. He asked that these groups return to a plenary session the following morning with the objective of providing an integrated set of findings for NASA to consider to improve future SMEX and MIDEX AOs.
John Lintott then described the genesis of the MIDEX AO, which was released in March 1995. It had been based on the most recent SMEX issued in 1992 and previous Explorer AOs, with the incorporation of some Discovery-like features. Only charges to the Explorer program were counted, not the launch vehicle, nor civil service manpower and facilities. The Principal Investigator (PI)-mode was added as an option during the development of the AO but because Discovery had not yet flown it was regarded as an experiment, and thus proposals had to show a compelling rationale for such an option. NASA anticipated selecting one Goddard Space Flight Center (GSFC)-built spacecraft and one procured from industry, as stated on pages 1 and 2 of the AO.
Lintott said that the issue of NASA civil service manpower and facility charges had been addressed at the April 1995 MIDEX AO preproposal conference. He said that NASA had in fact selected, on the basis of the evaluation and selection criteria in the AO, two missions, one with a GSFC-built spacecraft and one with the spacecraft procured from industry.
Mike Janssen, JPL, asked Lintott to elaborate on the consistency between GSFC spacecraft and one from industry: was that consistent with PI-mode? Lintott said that the industry spacecraft could have been PI-mode.
Barry Mauk, APL, asked about the inconsistency of not allowing foreign contributions to count in reducing the cost cap, whereas free civil service input was allowed. Lintott replied that the intent was to avoid having disproportionately-sized proposals. At the extreme, for example, this could have led to simply having instruments proposed for launch on major foreign missions, and so the approach on foreign contributions had been to maintain the traditional scope of the Explorer program.
Chris Roberts, CTA Space Systems, questioned the result in which all four missions finally selected proposed a GSFC-provided spacecraft. The discussion illustrated that there was also confusion in the meaning of a "GSFC-provided" spacecraft. Did that mean "built" or "procured"?
Mel Ulmer, Northwestern University, questioned the tension between cutting-edge technology and risk assessment. He said that two and a half years was not enough time to bring a new technology to a risk-free state of development.
The Chairman noted that these issues would be addressed in the splinter groups and then introduced the first speaker in the next session.
Science Community Perspectives
(i) Mike Janssen, JPL
Janssen briefly described the Primordial Structures Investigation (PSI) mission, which he believed set a good example for the PI-mode of operation. The proposal exhibited strong science, the team had put in a major effort, and he believed that the proposal was a real contender.
He had two areas of concern with the AO. It was inherently biased because the full costs of GSFC and other civil service were not to be taken into account, amounting to an unfair competition in a cost-constrained environment. His recommendation would be to adopt the PI-mode and let everyone compete equally. As currently set up, PI-mode proposals had to explain a lot more than those using the NASA-provided spacecraft.
He had taken the cost cap in Step One seriously into account (a lesson learned from Discovery) and had worked hard to achieve a cost below $70m. This meant that he had to do a comprehensive costing at Step One. Most people had the same experience. The two-step process did not solve the problems it had been meant to solve.
Janssen felt that a down-select to 13 was too many. There was no interaction between reviewers and proposers and he felt it important to have oral presentations at that stage. The science review of PSI got sidetracked at Step One and he would have liked the opportunity to correct what he described as misunderstanding on the part of the reviewers. Finally, he commented that the evaluation criteria were not consistent with the goals as stated in the AO.
The presentation given at the meeting by Mike Janssen follows.
Title: The Primordial Structures Investigation (PSI) Spacecraft Option: PI Mode Status: Passed Step One Review. Category 1 in Step Two Principal Investigator: Michael A. Janssen JPL |
PI-mode is a wonderful addition in MIDEX; however, the two-mode structure was seriously biased:
TWO-STEP PROCESS SUCCESSES:
PROBLEMS:
EVALUATION CRITERIA NOT CONSISTENT WITH GOALS GOALS STATED IN AO:
INSTEAD:
|
(ii) Paul Evenson, Bartol Institute
Paul Evenson noted that he was from a small university group and so had a rather different perspective from the previous speaker. The Positron Electron Magnet Spectrometer (POEMS) investigation has a long history; it was originally selected for EOS, but was then down-selected. It was also selected for the last SMEX: it completed Phase B and was judged flyable. The team saw MIDEX as a golden opportunity, ideally suited for the experiment. The team interpreted the GSFC-provided spacecraft as one procured from industry by GSFC. They did not think GSFC could provide the kind of spacecraft they needed, so they had no option other than PI-mode.
Criticisms of the proposals were focused on management issues. Being a small group (but with an industrial partner who did a good job), Evenson pled guilty as charged to the criticisms of the POEMS proposal. But he felt the criticisms were based on issues that could be worked out in Phase B, e.g., a Program Manager could have been hired in Phase B once they had a budget. The team was using successful SMEX experience which had not requested the kind of Work Breakdown Statement (WBS) that MIDEX AO had. They had depended on their SMEX heritage.
Evenson thought that Step Two was an excellent idea but that the down-select was not stringent enough. Bartol did not have deep enough pockets to adequately prepare the proposal without being guaranteed a flight. The AO demanded a high standard of management for a small scientific group. Such standards should be fixed, that is, if a WBS is acceptable for one Explorer AO it should be acceptable for another. Once, the POEMS experiment had been defined for flight; NASA had once thought that its WBS process was acceptable.
He said that NASA may not want groups like the Bartol group and they will not be proposing again to NASA. They will propose to groups that have enough resources to help them propose to NASA.
Kenneth Johnston, Naval Observatory, suggested that resources be made available to PIs after the down-select in order to help them meet the stringent demands of the AO in the management and cost areas.
Mel Ulmer from Northwestern University asked whether there should not be more justification of cost at Step One, with oral briefings, a Phase A, and then a down-select. Evenson replied there was not enough time for a Phase A study. The two-step process should be modified to address the technical, management, and resource issues.
(iii) Neil Johnson, NRL
Johnson was the PI on the Burst Locations with an Arc Second Telescope (BLAST) proposal, and was presenting from a high-energy perspective. Many of these high-energy missions are rather large and opportunities for flight are restricted although there are some sub-orbital opportunities. How can one get from SR&T level to a flight instrument? One needed to take more risk.
The team needed more information on the NASA spacecraft and ground-support. It was very difficult to get information. Johnson asked why free foreign participation was not a plus. Was the purpose of the program to do science or to create jobs in the U.S.?
Michael Cherry, Louisiana State University, noted that the Step One process set up proposals for failure. Step One proposals did not contain enough information on the cost and management issues. The science review was looking for exciting science; they identified potential technical problems. There was no feedback from Step One. He thought it would have been helpful to have engineering and cost reviewers in the Step One review.
Kenneth Johnston, Naval Observatory, commented that there should be more input from science reviewers at Step Two to see whether the science was good enough to justify whatever risks there may be. Brian Dennis asked whether this was a case of simply not being high-energy’s turn this time.
Johnston commented that in the Solar Max review process, the two-step approach had worked well. The first selection had been followed by a study, then an oral presentation, followed by a second-step selection.
The presentation given at the meeting by Neil Johnson follows.
High Energy Astrophysics Perspective (MIDEX Step Two Proposal: BLAST)
High Energy Astrophysics (HEA) Context
MIDEX Two-Step Competition is a good idea, but
RECOMMENDATIONS:
Dual Mode—PI or NASA-Provided Spacecraft—is bad
RECOMMENDATIONS:
The Explorer Program Summary/Recommendations:
|
(iv) Robert Paulos, University of Wisconsin
Paulos presented the perspective of the Hot Interstellar Medium (HIMS) PI, Dr. Wilton Sanders. HIMS had chosen the NASA-provided spacecraft mode and also had a difficult time communicating with GSFC. Paulos suggested that the basic bus with a cost breakdown should be described in the AO. He felt it was also difficult to address all the management points within the given page limit. On the two-step process, he suggested that reviewers, or NASA, should simply assign a ranking at the end of Step One, and then leave it to proposer to decide whether to go ahead. He endorsed finding some way of getting feedback to reviewers to avoid misunderstanding. This might be oral or submitted written questions from reviewers making reference to material already in the proposal.
Mel Ulmer, Northwestern University, asked Paulos whether they had subtracted anything from the spacecraft bus: they had in fact done that. Johnston said that there were clearly points in the proposal that the reviewers missed and suggested that questions could have been sent out. Paulos said a down-select to four proposals would prevent overworking the reviewers and so avoid their missing information in proposals. Evenson had learned from the debriefing that there had been misunderstanding on the reviewers' part. Friedman, CalTech, also suggested that, had there been feedback, misunderstandings could have been cleared up. The presentation given at the meeting by Robert Paulos follows.
Hot Interstellar Medium Spectrometer (HIMS) The AO
The Two-Step Process
Debriefings
|
(v) Barry Mauk, APL
Paulos asked whether if Mauk had had a choice of going to Step Two, would he have done so? The answer was yes. Neil Johnson, NRL, noted that Step Two reviewers did not have the results of the Step One review. Janssen asked whether anyone from NASA could elaborate on the comment that the magnetosphere panel did not reflect the discipline, as this would be a serious flaw. (See Mauk presentation below). The Chairman commented that he had looked at the list of reviewers and assured the audience that the reviewers were among the finest in the country. Janssen noted that he would have liked to have seen broader participation by the community. Mauk commented that his remarks about the reviewers were based on information gained in the proposal debriefing. The presentation given at the meeting by Barry Mauk follows.
Barry H. Mauk Character of Phase 1 Evaluations
MIDEX as the Vehicle for Discipline Science
Full Cost Accounting
-Q: Some of our Co-Investigators are U.S. Government employees. The stated reason for including foreign contributions in the cost cap constraints is to make sure that all proposed investigations are of the same scope (or no bigger than a certain scope). For the same reason, are the full costs of the participation of U.S. Government employees to be included in the proposals (i.e., must one include the true costs of utilizing the "free" labors of U.S. Government employees)? -A: The proposer needs to negotiate the appropriate rate with the government agency providing the civil service personnel, and include that cost in the proposal. The rate chargeable to the proposal is included in the cost cap. Recommendations
|
(vi) John Raitt, Utah State University
Raitt was the PI on COPIES (a Constellation of Polar Ionospheric Electrodynamic Satellites), which needed 8 spacecraft. He did not think that NASA could provide these spacecraft and so opted for the PI-mode. In general, he thought the two-step approach a good concept that needed more work. He was concerned that PIs overestimated costs at Step One so as not to exceed that cap at Step Two. At Step One NASA should do a proper cost review so as to ensure that a PI can stay within the limit. He suggested that the AO not request a cost definition, but simply allow the PI to state that he can stay within the program cost cap.
Raitt was relieved not to have been selected for Step Two. A relatively large number of proposals had been selected for Step Two so there was a great risk of spending resources and not winning. He suggested that funding should be provided before Step Two to enable good costing to be done, like a Preliminary Design Review. Although a small percentage of the total cost of mission, it would be a good investment.
He was not sure whether the PI-mode put some at a disadvantage but his experiment required that choice. He did understand from the AO that they could work with GSFC to have GSFC procure what was needed but that would have required too much interaction. He felt that they were too far away, and so went with local industry.
He would have liked some feedback on why the experiment had not been selected for Step Two. He felt that the science was good as the obvious next step, but the limited page count meant the proposal omitted some points that could have been clarified with interaction with the reviewers. Perhaps NASA should consider a three-step approach? It would be easy to cut from a large number of proposals to dismiss some non-contenders, so as to arrive at some number that needed serious consideration. Then there might be an interview process with this smaller group of PIs, followed by a down-select to a smaller number for a Step Three process which might be funded at $100K level to produce high-level proposals for a better selection.
Ulmer asked how Raitt expected reviewers to assess his costs if the proposal did not give them any cost information. Raitt said that the proposal has to give enough technical information on implementation to support the contention that the mission can stay within the cost limit. Confidence in the estimate comes from experience; Raitt said his costs are mostly manpower, not hardware. But there was too much work involved at present in documenting that the estimate was good.
Janssen suggested the use of cost models, accepting a level of uncertainty.
Raitt asked whether NASA was really ready to spend $70m on a mission, or were they really looking for a cheaper mission?
Jayant Murphy, Johns Hopkins University, said he was very happy with the science review but said that costing should be in Step One proposal. Not enough information was required at Step One for reviewers to judge costs. The MIDEX program requires both science and other resources to produce a good proposal. The Step One proposals each took a different approach to costing; one could be penalized for too little information and also penalized for providing too much information. Neil Johnson added that it was made clear that the cap was $70m; proposals had to be very carefully costed so as not to come above $70m. Raitt noted, however, that that was only true where mission costs were near the cap. His experiment was not that costly.
(vii) Dave Rust, APL.
Rust was the PI on the Heliospheric Links Explorer (HELIX) experiment that, at the beginning of the process, used a very new approach that had now become very well-known. The MIDEX process was inherently conservative and encouraged division by discipline. The Step One selection, with one review panel per discipline, produced one proposal for each discipline. He felt that programmatics dominated the process. NASA should decide one way or another on the conception of program: was this an Explorer program? Did NASA have to select the highest priority science or was it discipline-divided?
He queried the cost accounting. The science panel had said they did not believe this experiment could be done for this price. But APL always rigorously costs everything. The proposal was thoroughly cost accounted, so it was outrageous for scientists to judge the cost inadequate. He suggested the need to either do real cost accounting, or check boxes for cost ranges, e.g.. $30-$60m; $50m-$70m.
He asked whether MIDEX was eating up the Solar-Terrestrial Probe line and suggested that the appropriate level of science effort in the overall program was not being achieved.
Feedback was very important. It was not a question of sealed bids. The SOHO selection had engaged in dialogue with PIs. Evenson suggested that it was ESA that had introduced the dialogue. But Rust said he did talk to GSFC during that process.
(viii) Brian Dennis, GSFC
Dennis made a couple of unique points. He was a Co-I on the High Energy Solar Spectroscopic Imager (HESSI) proposal, which had been chosen as runner up. He was also a free civil servant at GSFC. Dennis noted that full cost accounting was being introduced and that one had to wait to see how it was implemented. He had also had difficulty getting information about the spacecraft. Once selected at Step Two, they, like all NASA-mode proposers, had been invited to talk to GSFC, but then got the message that GSFC could not handle the demand for information. He preferred a single step approach.
He said that, in the final analysis, the evaluation was ignored. NASA wanted to do certain experiments (e.g., cosmic background). NASA did not want to select a solar mission. The team was told that the proposal was perfect and could not have been improved.
HESSI had been studied for a long time, compared to HELIX which was a new, unstudied idea. He suggested that Explorer money should not be used to study new mission concepts as such money is available elsewhere. The two-step process levels the playing field for those whose ideas have not been studied; the other study routes should be used so as to get on with the selection of MIDEX.
He felt that debriefing must be face-to-face. The team could not get the written evaluations at the technical debriefing; it was based only on the notes of those who happened to be there. There should be direct contact with reviewers and PIs should have to go forward to answer reviewers' questions. There were misunderstandings of technical questions and thus communication with reviewers was needed. Too much of the process was driven by lawyers.
Contributions from foreign partners should be encouraged. The objective is to achieve the maximum science per US dollar. It was very hard to satisfy requirements for detailed costs, and the 10-month and the 22-month definition phase merely doubled the work.
Neil Johnson did not want to see the program reduced to only long-studied proposals. He noted that there had been questions about how much heritage the winning proposals had.
Bohlin addressed the question of a discipline-driven selection. He said the selection team had not been predisposed, although everyone in the community knew that there were high-priority science areas. At the ultimate selection between several excellent proposals, programmatics had to come into play. The presentation given at the meeting by Brian Dennis follows.
Brian Dennis — HESSI Lead Co-I
Clarity and Content of AO
Evaluation Criteria
Proposal Conference
Evaluation Process
General Comments
|
That concluded the presentations from the science community. The next item on the agenda began presentations on perspectives from industry.
Industry Perspectives
(i) Shirley Savarino, TRW Civil & International Systems Division, Space & Electronics Group
Savarino noted that no PI-mode missions won, which she thought sent a message as to what NASA wanted. It had been made very clear that PI-mode was not wanted. Five proposals had industry participation. Now that the IMAGE (Imager for Magnetopause-to-Aurora Global Exploration) team was to procure a spacecraft, industry now had to bid again on the same competition.
Ulmer asked whether, as the new NASA structure encourages disciplines merge, there were fewer chances of discipline-specific missions? Bohlin responded that the community has to work within disciplines, but that there may be opportunities for cross-discipline science. However, line item funding from Congress for Discovery (planetary) and MIDEX (space physics and astrophysics) still continued. Jay Bergstrahl, NASA Headquarters, noted that the next Discovery AO includes Origins science.
The presentation given at the meeting by Shirley Savarino follows.
Industry Perspective Three key issues relating to MIDEX procurement
Issue #1: Win Probability Low
Issue #2: Spacecraft Supplier Provides Low Leverage on Outcome
Issue #3: Uncomfortable to Compete Against Our Customers
Thoughts for Future Procurements
|
(ii) Lamont Di Biasi, Orbital Sciences Corporation
Di Biasi noted that it was not easy to understand the AO process. He said there were certain areas of science that Orbital Sciences felt NASA wanted to do, and so the company pursued certain investigators in certain science disciplines. He noted that the actual expenditures had been closer to $600K, rather than the $500K noted below. He was now having trouble convincing his management to invest in the next SMEX and MIDEX proposals.
He noted the inconsistency between the value of high technology and risk. At one extreme is a mere concept, others have some heritage, testbed experience, or flight history, perhaps on the Shuttle.
Citing his experience of the Earth System Science Pathfinder (ESSP) program, he said that the various Centers direct their PIs to procure industry participation in very different ways. These can vary from a one- or two- page proposal to a comprehensive effort. This is something NASA could address. In addition, universities have their own differing rules about how to procure industry partnerships, but this is not something that NASA fix.
Di Biasi noted that when some PIs found they could not get information from GSFC on the NASA-mode spacecraft, they wanted to switch to PI-mode, but the AO did not allow this. Criteria were different in the two modes: the PI-mode proposals were evaluated on their Program Managers, but the NASA-mode proposals were not. The presentation by Lamont Di Biasi follows.
PERSPECTIVE - 1 - EXPENSIVE
PERSPECTIVE - 2 - PROCESS
PERSPECTIVE - 3 - INCONSISTENT EVALUATION
PERSPECTIVE - 4 - INDUSTRY PARTNER SELECTION PROCESS
SUMMARY
|
(iii) Bill Gail, Ball Aerospace.
The AO process should help improve the science being done through cross-pollination. The cost of the whole process is probably about 10% of the total program value, if everybody’s cost is included. Gail asked what was meant by science/dollar; does that mean low return for low cost? The down-select should be based on issues of science, not resources. Industry has limited resources and cannot support many proposals. Therefore Step One should be judged only on science; reviewers should be relied on to competently assess relative costs. Gail also supported the concept of giving Step One proposals a ranking and letting the PI judge whether to go forward. This is the approach in the Code Y ESSP program.
Mauk commented that if cost were not taken into account at Step One, then those proposals could result in "undoable" science, and beat "doable" science. Cindy Faulconer, Lockheed Martin Astronautics, echoed that sentiment. John Raitt said that he supported describing a technology approach in Step One to show how the science will be done, but without detailed costing. Step Two is really the definition phase of the proposal and people who do the costing need the resources at that point to do it well. Gail, however, emphasized that funds should be available to help people to do costing, not to define the mission. The presentation given at the meeting by Bill Gail follows.
Objectives
Issues
Suggestions
|
That concluded the presentations from the industry community. The next item on the agenda was perspectives from government/FFRDC.
Government/FFRDC Perspectives
(i) Mike Janssen, JPL
Mike Janssen spoke briefly, noting that he was not really equipped to speak for JPL. He did suggest that NASA eliminate the NASA-mode option.
(ii) Larry Crawford, Applied Physics Laboratory
Crawford emphasized that he did not like to see APL costs compared to model costs. Federal regulations allow NASA to strongly emphasize past performance to evaluate cost estimates. He had made same point at the Discovery Lessons Learned Workshop but the current draft AO still does not include that feature. He urged OSS to consider the ESSP 7-month two-step process.
Evenson commented that scientists can be qualified to provide a technical review; many were experimenters not theorists. Crawford felt that there should be engineers on the review panel. Cherry confirmed that his panel had consisted of scientists (including experimenters) but not engineers nor cost experts. When asked whether there had been 13 selected for Step Two due to lack of expertise to hone the numbers, Cherry said that the 13 was simply a result of each panel having been asked to come up with their best 2 or 3 proposals. The presentation given at the meeting by Larry Crawford follows.
JHU/APL MIDEX Proposals
Major Shortcomings: S = Science T = Technical merit TC = Total NTE Cost DR = Data Reduction & Analysis plan S + T + TC + DR = 100% -- Evaluation Formula?? S + T = TC If DR = 20%, DR <S + T Then TC = 40%, DR <TC and T = 20% S = 20% i.e. Science Value Only 20-–40%
Cost Evaluation
Recommendations
Spacecraft Procurement
SMEX Cost Cap Launch vehicle for the last SMEX AO selection process was standard Pegasus, since replaced with the Pegasus XL, with greater payload weight and volume capability. To optimize NASA’s return on its investment in SMEX, PI’s should be able to propose payloads that utilize full capacity of the Pegasus XL. The next SMEX AO should provide proportionately larger payload cost cap, after adjusting for inflation, than the last SMEX AO. |
The meeting then adjourned for lunch. Reconvening in the afternoon, John Lintott, in response to the prior discussion, noted that the cost evaluation in Step One had not been an in-depth evaluation, but was only intended to provide a reality check. NASA had relied on the threat of elimination if exceeding Step One costs in Step Two to keep PIs in line on Step One costs. Lintott summarized written comments received prior to the meeting. Comments were received from:
3 Step Two proposersVerbal comments received during debriefings were not included. Several compliments had also been received. Some suggested that the two-step process had considerable merit, enabling the proposer to concentrate on scientific and technological issues in Step One. The preproposal conference was thought to be beneficial.
The presentation given at the meeting by John Lintott follows.
Criticisms and suggestions for improvement The two-step process Modify the two-step process
-enormous effort on the part of the proposer. -expensive - some institutions do not have deep pockets. -reduce the number of proposals going into Step Two. -fund preparation of Step Two proposals. -reduce the length of time for submission of proposals. Two-step process a disadvantage
-requires a repetition in description of instrument. -possibility of changes in NASA operating rules and capabilities between steps. Eliminate the two-step process
-select 4 missions. -conduct a Phase A, then down-select.
The AO
Costing requirements of the AO
Preproposal conference
|
Options for spacecraft development/full-cost accounting
Technology and risk
Debriefings
|
The Chairman then invited participants to divide into groups, and circulate between groups if so desired. His charge to the groups follows.
Two-Step Selection Process: Chair, Mike Janssen, JPL. This group was to consider:
Dual-Mode Selection Process: Chair, Paul Evenson, Bartol. This group was to consider:
Selection Criteria: Shirley Savarino, TRW. This group was to consider:
Cost Assessment: Chair, Larry Crawford, APL. This group was to consider:
The splinter groups met individually for several hours on Wednesday afternoon, and then reconvened briefly in the late afternoon for a review of progress and potential overlap. It was suggested that the groups continue to work into the evening, and that the plenary session should begin at 8:30 am the following morning, so that all participants could review all recommendations.
Thursday 27 June, 1996
The plenary session began with Mike Janssen, JPL, presenting the recommendations of his group (Two-Step Process). There was wide-ranging discussion and a suggestion that there be a survey of PIs to see how many felt dissatisfied with the review process. There was consensus that too many proposals had been down-selected after Step One. This made it particularly difficult and costly for industry to support the Step Two effort, with statistically unattractive odds of winning. The need for feedback from reviewers was reiterated. In addition, it was felt that those PIs selected for Step Two should have received the evaluations from the Step One review.
It was also felt that there was a disconnect between the emphasis in the AO on the Not-to-Exceed cost cap required at Step One, and the less than rigorous review of costs at that stage. The criteria in the AO gave cost as equal in importance to scientific and technical merit, and so many PIs had worked very hard at getting the correct costs. There was a long discussion about the approach of the review panels to cost feasibility, particularly in the light of the dictates of the AO. It was again suggested that science only should be reviewed and that technical merit and cost should be separated from the Step One phase. However, Brian Dennis, GSFC, suggested that one could not and should not separate science and technical merit, only costs.
Shirley Savarino, TRW, presented her group's recommendations on selection criteria. There were points of clarification discussed but there was general agreement on the need to separate science and programmatics out at Step One, and that the science proposed should not change from Step One to Step Two. There had not been much discussion of the role of the minimum science mission, although some PIs had been confused as to what was meant by the minimum science mission.
Larry Crawford presented his group's findings on cost assessment. There was little discussion, except on the issue of foreign contributions. NASA's goal had been to confine the costs of the mission within the MIDEX range, within NASA's control, and to avoid a proposal that simply made a contribution to a foreign mission, over which NASA would have no control. It was agreed that by imposing a cap on foreign contributions, NASA could achieve that goal without penalizing the PI.
Paul Evenson, Bartol, gave the last presentation. This group recommended a single-mode approach that would be flexible enough to include a menu of services, including management services, that NASA could provide. The terminology in the program should be changed to reflect the fact that a PI could ask NASA to manage the mission, which might incorporate a commercially provided spacecraft.
Meeting participants then unanimously endorsed the recommendations given on pages 6-13 of this report and the plenary session was adjourned.