MIL-HDBK-61A: Configuration Verification and Audit

8.2 Configuration Verification and Audit Concepts and Principles

There is a functional and a physical attribute to both configuration verification and configuration audit. Configuration verification is an on-going process. The more confidence the Government has in a contractor's configuration verification process, the easier the configuration audit process becomes. The reward for effective release, baselining and configuration/change verification is delivery of a known configuration that is consistent with it's documentation and meets its performance requirements. These are precisely the attributes needed to satisfy the ISO-9000 series requirements for design verification and design validation as well as the ISO 10007 requirement for configuration audit.

Configuration verification is a process that is common to configuration management, systems engineering, design engineering, manufacturing, and quality assurance. It is the means by which a contractor verifies his design solution. The functional aspect of configuration verification encompasses all of the test and demonstrations performed to meet the quality assurance sections of the applicable performance specifications. The tests include verification/qualification tests performed on a selected unit or units of the CI, and repetitive acceptance testing performed on each deliverable CI, or on a sampling from each lot of CIs, as applicable. The physical aspect of configuration verification establishes that the as-built configuration is in conformance with the as-designed configuration. The contractor accomplishes this verification by physical inspection, process control, or a combination of both.

Once the initial configuration has been verified, approved changes to the configuration must also be verified. Figure 8-2 illustrates the elements in the process of implementing an approved change.

Change verification may involve a detailed audit, a series of tests, a validation of operation, maintenance, installation, or modification instructions, or a simple inspection. The choice of the appropriate method depends upon the nature of the CI, the complexity of the change, and upon the support commodities that the change impacts. If the change is being introduced into a production line, and all future units will have the change incorporated via the production process, it is normally sufficient to ensure that:

Manufacturing instructions contain the change and are released for use (as with a work order), and
The first articles produced are inspected for compliance.

However, if support elements are impacted, or the change requires incremental retrofit to many units, complete implementation and verification of the change can be a lengthy process. Under these circumstances, implementation planning must define the extent to which the change to each unit and support commodity is to be verified; and the records to be maintained. When materials, parts, or retrofit kits are ordered in incremental stages (e.g., per year, per month), the incremental ordering and supply actions should also be verified.

Retrofit changes to organically supported items are verified and reported to the Government's status accounting system by the activity given installation and checkout responsibility for the retrofit. Changes retrofit by the contractor for contractor supported items are verified by the contractor.

The dictionary definition of the word "audit" as a final accounting gives some insight into the value of conducting configuration audits. As has been discussed earlier in this handbook, configuration management is used to define and control the configuration baselines for the CIs and the system. In general, a performance specification is used to define the essential performance requirements and constraints that the CI must meet. When a performance specification is baselined by the Government, those requirements are contractual, so it is prudent for the Government to ascertain that the contractor has provided the expected performance capabilities. For complex systems and CIs, a "performance" audit is necessary to make this determination. Also since development of an item involves the generation of product documentation, it is prudent to ascertain that the documentation is an accurate representation of the design being delivered. To the extent that the Government is buying the CIs to approved detail specifications, the Government would perform this kind of "design" audit. However, the design activity should perform both performance and design audits on all CIs in the deliverable product, especially if the government does not intend to conduct audits on those particular CIs (usually because the applicable configuration documentation is not (will not be) under government configuration control). The operation and life cycle support of the CI is based on this documentation. To fail to assure its accuracy can result in acceptance of items that will not perform as specified, or to greatly complicate future logistics support of the CI.

Configuration audits provide the framework, and the detailed requirements, for verifying that the contractor's development effort has successfully achieved all of the requirements specified in the configuration baselines. If there are any problems, it is the auditing activity's responsibility to ensure that all action items are identified, addressed and closed out before the design activity can be deemed to have successfully fulfilled the requirements.

There are three phases to the audit process, and each is very important. The pre-audit part of the process sets the schedule, agenda, facilities and the rules of conduct and identifies the participants for the audit. The actual audit itself is the second phase, and the third is the post-audit phase in which diligent follow-up of the audit action items must take place. For complex products such as major weapon systems, the configuration audit process is a series of sequential/parallel audits of various CIs and the system to Government-controlled System and CI performance specifications conducted over a period of time to verify all relevant elements in the weapon system product structure.

Audit of a CI may include incremental audits of lower-level items to assess the degree of achievement of requirements defined in specifications/documentation not controlled by the government. The process will normally involve audits conducted by prime contractors on subcontracted items at subcontractor facilities with or without Government participation (at Government option) and audits of prime contractor developed items conducted by the Government at the contractor's facility. Each item may be subjected to separate functional and physical audits, or both audits may be conducted at the same time.

8.2.2.1 Functional Configuration Audit.

The Functional Configuration Audit (FCA) is used to verify that the actual performance of the CI meets the requirements stated in its performance specification and to certify that the CI has met those requirements. For systems, the FCA is used to verify that the actual performance of the system meets the requirements stated in the system performance specification. In some cases, especially for very large, complex CIs and systems, the audits may be accomplished in increments. Each increment may address a specific functional area of the system/CI and will document any discrepancies that are found in the performance capabilities of that increment. After all of the increments have been completed, a final (summary) FCA may be held to address the status of all of the action items that have been identified by the incremental meetings and to document the status of the FCA for the system or CI in the minutes and certifications. In this way, the audit is effectively accomplished with a minimum of complications.

Although an FCA is only required once for each CI or system, a number of FCA-like activities may be accomplished at other times during the life cycle of the CI or system.

a. Many Class I ECPs incorporate a new design into the baselined design. The performance of each new design element must be verified to ensure that it will not degrade performance of the CI or system below the performance specified by it's Government-controlled performance specification. The degree and type of verification will be included as part of the ECP; it may vary from a simple analysis of the similarity to the old design to a lengthy program of testing similar to the original verification testing accomplished during the EMD phase. However, it is important to understand that a complete retest and FCA are not required for each ECP; only the verifications specified in the ECP are required.

b. If the Government is controlling the detailed design, a production contract may require a "first article" inspection to be accomplished. This would include more comprehensive "testing" than the normal production acceptance tests, and the test data resulting from the "first article" would be subject to a review process not unlike an FCA.

c. An ECP or a new contract may call for the development of a new CI(s) and incorporation of the new CI into the system via a modification program. The expected performance of the new CI would commonly be defined in a performance specification, and the results of the verification testing of the CI would be checked at an FCA for the new CI. In addition, some retesting of the existing system elements with the new CI incorporated would normally be required, and those results would also be subject to a review similar to an FCA.

8.2.2.2 Physical Configuration Audit.

The Physical Configuration Audit (PCA) is used to examine the actual configuration of the CI that is representative of the product configuration in order to verify that the related design documentation matches the design of the deliverable CI. It is also used to validate many of the supporting processes that the contractor uses in the production of the CI. The PCA is also used to verify that any elements of the CI that were redesigned after the completion of the FCA also meet the requirements of the CI's performance specification. In cases where the Government does not plan to control the detail design, it is still essential that the contractor conduct an internal PCA to define the starting point for controlling the production design and to establish a product baseline. Additional PCAs may be accomplished later during CI production if circumstances such as the following apply:

The original production line is "shut down" for several years and then production is restarted
The production contract for manufacture of a CI with a fairly complex, or difficult-to-manufacture, design is awarded to a new contractor or vendor.

This re-auditing in these circumstances is advisable regardless of whether the contractor or the government controls the detail production design.

8.2.2.3 Application of Audits during Life Cycle.

It is extremely unlikely that FCAs or PCAs will be accomplished during the Concept Exploration and Definition phase or the Program Definition and Risk Reduction phase of the life cycle. Audits are intended to address the acceptability of a final, production-ready design and that is hardly the case for any design developed this early in the life cycle. [NOTE: An activity similar to the FCA (and sometimes the PCA) might be accomplished during the PD&RR phase as a part of the completion of a competitive prototyping effort to facilitate the evaluation of the results of the competition.]

It is during the Engineering and Manufacturing Development (EMD) phase that the final, production, operationally ready design is developed. Thus, this phase is normally the focus for the auditing activity. Either the Government or the contractor will conduct a PCA for each HW CI that has completed the FCA process to "lock down" the detail design by establishing a product baseline. Hardware CIs built during this phase are sometimes "pre-production prototypes" and are not necessarily representative of the production hardware. Therefore, it is very common for the PCAs to be delayed until early in the Production phase of the program.

Requirements to accomplish FCAs for systems and CIs are included in the Statement of Work (SOW) tasking. The FCA is accomplished to verify that the requirements in the system and CI performance specifications have been achieved in the design. It does not focus on the results of the operational testing that is often accomplished by operational testing organizations in the services, although some of the findings from the operational testing may highlight performance requirements in the baselined specification that have not been achieved. Deficiencies in performance capability, as defined in the baselined specification, result in FCA action items requiring correction without a change to the contract. Deficiencies in the operational capability, as defined in user-prepared need documents, usually result in ECPs and/or contract changes to incorporate revised requirements into the baselined specifications or to fund the development of new or revised designs to achieve the operational capability. Since the final tested software design verified at the FCA normally becomes the production design, the PCAs for CSCIs are normally included as a part of the SOW tasking for the EMD phase. CSCI FCAs and PCAs may be conducted simultaneously to conserve resources and to shorten schedules.

It is normal that the first production units in the Production, Fielding/Deployment and Operational Support Phase would be subjected to a PCA, which, depending on whether the acquisition strategy was performance or detail design based, would be conducted by the contractor or by the Government, respectively. This PCA allow the establishment of a Product Baseline for the CI reflecting the design that will be delivered to the field and will require support. From a logistics support standpoint, it is essential that the support activity have an accurate picture of the exact configuration. If it does not, it is likely that the wrong spares will be acquired or redesign of the CI will be based on inaccurate information, leading to problems in the operation and/or support of the CI.

During a PCA, the deliverable item (hardware or software) is compared to the product configuration documentation to ensure that the documentation matches the design. This ensures that the exact design that will require support is documented. The intent is that an exact record of the configuration will be maintained as various repair and modification actions are completed. The basic goal is sometimes compromised in the actual operation and maintenance environment. Expediency, unauthorized changes, cannibalization, overwork, failure to complete paperwork, and carelessness can cause the record of the configuration of operational software or hardware to become inaccurate. In some situations, a unit cannot be maintained or modified until its configuration is determined. In these kind of circumstances, it is often necessary to inspect the unit against approved product configuration documentation, as in a PCA, to determine where differences exist. Then the unit can be brought back into conformance with the documentation, or the records corrected to reflect the actual unit configuration.

8.2.2.4 Auditing in the Performance-based Acquisition Environment

As discussed above, configuration audits address two major concerns:

The ability of the developed design to meet the specified performance requirements. The FCA addresses this concern.
The accuracy of the documentation reflecting the production design. This concern is addressed by the PCA.

Over the years prior to acquisition reform, the DoD developed hardware and software audit topics that were to be addressed by the FCA and the PCA, respectively. To document acceptability of a contractor's accomplishments in the FCA topic area, a series of certifications were established. Similarly another series of certifications were established for the PCA topic areas. The audit teams completed the certifications that were applicable to the type of audit they were performing. Because the Government typically took control of the detail design, it conducted both FCA and PCA for each CI. The Government teams eventually addressed all the audit topic areas that were applicable to the type of item (hardware or software) being audited.

Acquisition reform policy requires acquisition of deliverable products based on performance specifications, rather than detail specifications unless it is essential to buy an identical item. Using the certifications as they existed before acquisition reform would mean that:

The Government would normally conduct FCAs for the System and CIs with Government controlled performance specifications and would thus address (and certify) the FCA topics

Or by an "acquiring" contractor.

For correct application of this information, see NOTE on Contents page