Appendix D - Appendix D to Part 236—Independent Review of Verification and Validation
(a) This appendix provides minimum requirements for independent third-party assessment of product safety verification and validation pursuant to subpart H or subpart I of this part. The goal of this assessment is to provide an independent evaluation of the product manufacturer's utilization of safety design practices during the product's development and testing phases, as required by any mutually agreed upon controlling documents and standards and the applicable railroad's:
(1) Railroad Safety Program Plan (RSPP) and Product Safety Plan (PSP) for processor based systems developed under subpart H or,
(2) PTC Product Development Plan (PTCDP) and PTC Safety Plan (PTCSP) for PTC systems developed under subpart I.
(b) The supplier may request advice and assistance of the reviewer concerning the actions identified in paragraphs (c) through (g) of this appendix. However, the reviewer shall not engage in any design efforts associated with the product, the products subsystems, or the products components, in order to preserve the reviewer's independence and maintain the supplier's proprietary right to the product.
(c) The supplier shall provide the reviewer access to any and all documentation that the reviewer requests and attendance at any design review or walkthrough that the reviewer determines as necessary to complete and accomplish the third party assessment. The reviewer may be accompanied by representatives of FRA as necessary, in FRA's judgment, for FRA to monitor the assessment.
(d) The reviewer shall evaluate the product with respect to safety and comment on the adequacy of the processes which the supplier applies to the design and development of the product. At a minimum, the reviewer shall compare the supplier processes with acceptable validation and verification methodology and employ any other such tests or comparisons if they have been agreed to previously with FRA. Based on these analyses, the reviewer shall identify and document any significant safety vulnerabilities which are not adequately mitigated by the supplier's (or user's) processes. Finally, the reviewer shall evaluate and document the adequacy of the railroad's
(1) RSPP, the PSP, and any other documents pertinent to a product being developed under subpart H of this part; or
(2) PTCDP and PTCSP for systems being developed under subpart I of this part.
(e) The reviewer shall analyze the Hazard Log and/or any other hazard analysis documents for comprehensiveness and compliance with applicable railroad, vendor, supplier, industry, national, and international standards.
(f) The reviewer shall analyze all Fault Tree Analyses (FTA), Failure Mode and Effects Criticality Analysis (FMECA), and other hazard analyses for completeness, correctness, and compliance with applicable railroad, vendor, supplier, industry, national and international standards.
(g) The reviewer shall randomly select various safety-critical software, and hardware modules, if directed by FRA, for audit to verify whether the requirements of the applicable railroad, vendor, supplier, industry, national, and international standards were followed. The number of modules audited must be determined as a representative number sufficient to provide confidence that all unaudited modules were developed in compliance with the applicable railroad, vendor, supplier, industry, national, and international standards.
(h) The reviewer shall evaluate and comment on the plan for installation and test procedures of the product for revenue service.
(i) The reviewer shall prepare a final report of the assessment. The report shall be submitted to the railroad prior to the commencement of installation testing and contain at least the following information:
(1) Reviewer's evaluation of the adequacy of the PSP in the case of products developed under subpart H, or PTCSP for products developed under subpart I of this part, including the supplier's MTTHE and risk estimates for the product, and the supplier's confidence interval in these estimates;
(2) Product vulnerabilities, potentially hazardous failure modes, or potentially hazardous operating circumstances which the reviewer felt were not adequately identified, tracked, mitigated, and corrected by either the vendor or supplier or the railroad;
(3) A clear statement of position for all parties involved for each product vulnerability cited by the reviewer;
(4) Identification of any documentation or information sought by the reviewer that was denied, incomplete, or inadequate;
(5) A listing of each applicable vendor, supplier, industry, national, or international standard, procedure or process which was not properly followed;
(6) Identification of the software verification and validation procedures, as well as the hardware verification validation procedures if deemed appropriate by FRA, for the product's safety-critical applications, and the reviewer's evaluation of the adequacy of these procedures;
(7) Methods employed by the product manufacturer to develop safety-critical software;
(8) If deemed applicable by FRA, the methods employed by the product manufacturer to develop safety-critical hardware by generally acceptable techniques;
(9) Method by which the supplier or railroad addresses comprehensiveness of the product design which considers the safety elements listed in paragraph (b) of appendix C to this part.