Commission Implementing Regulation (EU) 2022/1426 of 5 August 2022 laying down rules for the application of Regulation (EU) 2019/2144 of the European Parliament and of the Council as regards uniform procedures and technical specifications for the type-approval of the automated driving system (ADS) of fully automated vehicles (Text with EEA relevance)

• A list shall be provided, collating all the units of the ADS and mentioning the other vehicle systems as well as off-board hardware/software and remote capabilities that are needed to achieve specified performance of the ADS to be approved according to its ODD.

• An outline schematic showing these units in combination, shall be provided with both the equipment distribution and the interconnections made clear.

• This outline shall include:

• (a) perception and objects/events detection including mapping and positioning

• (b) Characterisation of Decision-making

• (c) The ADS data elements.

• (d) links and interface with other vehicle systems, off-board hardware/software and remote capabilities

• 3.4.2.Functions of the units

• The function of each unit of the ADS shall be outlined and the signals linking it with other units or with other vehicle systems shall be shown. It shall include off-board systems supporting the ADS and other vehicle systems. This may be provided by a labelled block diagram or other schematic, or by a description aided by such a diagram.

• 3.4.3.Interconnections within the ADS shall be shown by a circuit diagram for the electric transmission links, by a piping diagram for pneumatic or hydraulic transmission equipment and by a simplified diagrammatic layout for mechanical linkages. The transmission links both to and from other systems shall also be shown.

• 3.4.4.There shall be a clear correspondence between transmission links and the signals carried between units. Priorities of signals on multiplexed data paths shall be stated wherever priority may be an issue affecting performance or safety.

• 3.4.5.Identification of units

• 3.4.5.1.Each unit shall be clearly and unambiguously identifiable (e.g. by marking for hardware, and by marking or software output for software content) to provide corresponding hardware and documentation association. Where a software version can be changed without requiring the replacement of the marking or component, the software identification must be by software output only.

• 3.4.5.2.Where functions are combined within a single unit or indeed within a single computer, but shown in multiple blocks in the block diagram for clarity and ease of explanation, only a single hardware identification marking shall be used. The manufacturer shall, by the use of this identification, affirm that the equipment supplied conforms to the corresponding document.

• 3.4.5.3.The identification defines the hardware and software version and, where the latter changes such as to alter the function of the unit as far as this Regulation is concerned, this identification shall also be changed.

• 3.4.6.Installation of sensing system components

• The manufacturer shall provide information on the installation options for the individual components that comprise the sensing system. These options shall include, but are not limited to, the location of the component in/on the vehicle, the material(s) surrounding the component, the dimensioning and geometry of the material surrounding the component, and the surface finish of the materials surrounding the component, once installed in the vehicle. The information shall also include installation specifications that are critical to the ADS’s performance, e.g. tolerances on installation angle.

• Changes to the individual components of the sensing system, or the installation options, shall be notified to the type-approval authority and be subject to further assessment.

3.5.   Safety concept of the manufacturer and validation of the safety concept by the manufacturer

• 3.5.1.The manufacturer shall provide a statement which affirms that the ADS is free from unreasonable risks for the vehicle occupants and other road users.

• 3.5.2.In respect of software employed in the ADS, the outline architecture shall be explained and the design methods and tools used shall be identified (see 3.5.1). The manufacturer shall show evidence of the means by which they determined the realisation of the ADS logic, during the design and development process.

• 3.5.3.The manufacturer shall provide the type-approval authority with an explanation of the design provisions built into the ADS so as to ensure functional and operational safety. Possible design provisions in the ADS are for example:

• (a) fall-back to operation using a partial system.

• (b) redundancy with a separate system.

• (c) diversity of systems performing the same function.

• (d) removal or limitation of the automated driving function(s).

• 3.5.3.1.If the chosen provision selects a partial performance mode of operation under certain fault conditions (e.g. in case of severe failures), then these conditions shall be stated (e.g. type of failure) and the resulting limits of effectiveness defined (e.g. immediate initiation of a minimal risk manoeuvre) as well as the warning strategy to the operator/remote operator, occupants and other road users (when applicable).

• 3.5.3.2.If the chosen design provision selects a second (back-up) or diverse means to realise the performance affected by the fault, the principles of the change-over mechanism, the logic and level of redundancy and any built-in checking features shall be explained and the resulting limits of effectiveness defined.

• 3.5.3.3.If the chosen design provision selects the removal of the automated driving function(s), this shall be done in compliance with the relevant provisions of this regulation. All the corresponding output control signals associated with this function shall be inhibited.

• 3.5.4.The manufacturer shall also provide the type-approval authority with an explanation of the operational safety measures to be put in place for the safe operation of the ADS such as an on-board operator or a remote intervention operator, supporting off-board infrastructure, transport and physical infrastructure requirements, maintenance measures, etc.

• 3.5.5.The documentation shall be supported, by an analysis that shows, how the ADS will behave to mitigate or avoid hazards that can have a bearing on the safety of vehicle occupants and other road users.

• 3.5.5.1.The chosen analytical approach(es) shall be established and maintained by the manufacturer and shall be made open for inspection by the type-approval authority at the time of the type-approval and afterwards.

• 3.5.5.2.The type-approval authority shall assess the application of the analytical approach(es):

• (a) Inspection of the safety approach at the concept level. This approach shall be based on a Hazard analysis/Risk assessment appropriate to system safety.

• (b) Inspection of the safety approach at the ADS level including a top down (from possible hazard to design) and bottom-up approach (from design to possible hazards). The safety approach may be based on a failure mode and effect analysis (FMEA), a fault tree analysis (FTA) and a System-theoretic process analysis (STPA) or any similar process appropriate to system functional and operational safety.

• (c) Inspection of the validation/verification plans and results including appropriate acceptance criteria. This shall include testing appropriate for validation, for example, hardware in the Loop (HIL) testing, vehicle on-road operational testing, testing with real end users, or any other testing appropriate for validation/verification. Results of validation and verification may be assessed by analysing coverage of the different tests and setting minimum coverage thresholds for various metrics.

• 3.5.5.3.The analytical approach under 3.5.5.2. shall confirm that at least each of the following items is covered:

• (i) Issues linked to interactions with other vehicle systems (e.g. braking, steering);

• (ii) Failures of the automated driving system and system risk mitigation reactions;

• (iii) Situations within the ODD where the ADS may create unreasonable safety risks for the vehicle occupants and other road users due to operational disturbances (e.g. lack of or wrong comprehension of the vehicle environment, lack of understanding of the reaction from the operator/remote operator, vehicle occupants or other road users, inadequate control, challenging scenarios)

• (iv) Identification of the relevant scenarios within the boundary conditions and management method used to select scenarios and validation tool chosen.

• (v) Decision making process resulting in the performance of the dynamic driving tasks (e.g. emergency manoeuvres), for the interaction with other road users and in compliance with national traffic rules

• (vi) Reasonably foreseeable misuse by the vehicle occupants/other road users, mistakes or misunderstanding by the operator/remote operator/occupants/other road users (e.g. unintentional override) and intentional tampering of the ADS.

• (vii) Cybersecurity threats on the safety of the ADS (to be covered by the analysis done in accordance with UN Regulation No 155 on Cyber Security and Cyber Security Management System).

• (viii) Operational safety issues: problems with the supporting off-board infrastructure, problem with the remote intervention operator, loss of connectivity, lack of maintenance, etc.

• 3.5.5.4.The assessment by the type-approval authority shall consist of spot checks to establish that argumentation supporting the safety concept is understandable and logical and implemented in the different functions of the ADS. The assessment shall also check that validation plans are robust enough to demonstrate safety (e.g. reasonable coverage of chosen scenarios testing by the validation tool chosen) and have been properly completed.

• 3.5.5.4.1.It shall demonstrate that the operation of fully automated vehicle is free from unreasonable risks for the vehicle occupants and other road users in the operational design domain, i.e. through:

• (a) an overall validation target (i.e. overall validation acceptance criteria) supported by validation results, demonstrating that the entry into service of the ADS will overall not increase the level of risk for the vehicle occupants and other road users compared to a manually driven vehicles; and

• (b) a scenario specific approach (i.e. scenario based validation acceptance criteria) showing that the ADS will overall not increase the level of risk for the vehicle occupants and other road users compared to a manually driven vehicles for each of the safety relevant scenarios;

• 3.5.5.5.The type-approval authority shall perform or shall require to perform tests as specified in point 4. of this Annex to verify the safety concept.

• 3.5.5.6.This documentation shall itemize the parameters being monitored and shall set out, for each failure condition of the type defined in point 3.5.4. of this annex, the warning signal to be given to the operator/remote operator/vehicle occupants/other road users and/or to service/technical inspection personnel.

• 3.5.5.7.This documentation shall also describe the measures in place to ensure the ADS is free from unreasonable risks to vehicle occupants, and other road users when the performance of the ADS is affected by environmental conditions e.g. climatic, temperature, dust ingress, water ingress, ice packing, inclement weather.

4. Verification and tests

Taking into account the results of the analysis of the manufacturer’s documentation package, the type-approval authority shall request the tests to be performed or witnessed by the Technical Service to check specific points arising from the assessment.

• 4.1.The functional operation of the ADS, as laid out in the documents required in point 3., shall be tested as follows:

• 4.1.1.Verification of the function of the ADS

• The type-approval authority shall verify the ADS under non-failure conditions by testing on a track a number of selected functions, as deemed necessary by the type-approval authority, from those described by the manufacturer, and by checking the overall behaviour of the ADS in real driving conditions including compliance with traffic rules.

• These tests shall include scenarios whereby the ADS is overridden by the remote intervention operator (if applicable).

• These tests can be based on test scenarios listed in Part 3 of this Annex and/or on additional scenarios not covered by Part 3.

• 4.1.1.1.The test results shall correspond with the description, including the control strategies, provided by the manufacturer in point 3.2. and shall comply with the performance requirements of this regulation.

• 4.1.2.Verification of the ADS safety concept

• The reaction of the ADS shall be checked under the influence of a fault in any individual unit by applying corresponding output signals to electrical units or mechanical elements in order to simulate the effects of internal failure within the unit.

• The type-approval authority shall verify that these tests include aspects that may have an impact on vehicle controllability and user information (HMI aspects e.g. interaction with the operator/remote operator).

• 4.1.2.1.The type-approval authorities shall also check a number of scenarios that are critical for the Object and Event Detection and Response (OEDR) and Characterisation of the decision-making and HMI functions of the ADS (e.g. object difficult to detect, when the ADS reaches the ODD boundaries, traffic disturbance scenarios, connectivity issue, problem with off-board systems, remote capabilities issues e.g. the absence of the remote intervention operator) as defined in this regulation.

• 4.1.2.2.The verification results shall correspond with the documented summary of the hazard analysis, to a level of overall effect such that the safety concept and execution are confirmed as being adequate and in compliance with the requirements of this regulation.

• 4.2.Simulation tool and mathematical models to verify of the safety concept may be used in accordance with Annex VIII to Regulation (EU) 2018/858, in particular for scenarios that are difficult on a test track or in real driving conditions. Manufacturers shall demonstrate the scope of the simulation tool, its validity for the scenario concerned as well as the validation performed for the simulation tool chain (correlation of the outcome with physical tests). To demonstrate the validity of the simulation toolchain, the principles of Part 4 of this Annex shall apply. Simulation shall not be a substitute for physical tests in Part 3 of this Annex.

• 4.3The manufacturer shall have a valid certificate of compliance for the safety management system (SMS) relevant to the vehicle type being approved.

5. Safety management system (SMS)

• 5.1.In respect of the ADS, the manufacturer shall demonstrate to the type-approval authority in terms of a safety management system (SMS) that effective processes, methodologies, training and tools are in place, up to date and being followed within the organization to manage the safety and continued compliance throughout the ADS lifecycle.

• 5.2.The design and development process shall be established and documented including safety management system, requirements management, requirements’ implementation, testing, failure tracking, remedy and release.

• 5.3.The manufacturer shall ensure effective communication channels between manufacturer departments responsible for functional/operational safety, cybersecurity and any other relevant disciplines related to the achievement of vehicle safety.

• 5.4.The manufacturer shall have processes aimed at collecting vehicle data, and data from other sources to monitor and analyse safety-relevant incidents/accidents caused by the engaged automated driving system. The manufacturer shall report to type-approval authorities, market surveillance authorities and the Commission the relevant occurrences in accordance with Part 5 of this Annex.

• 5.4.1.The manufacturer must enable the transport service operator to provide the type-approval authorities, market surveillance authorities or other authorities designated by the Member States with the vehicle data in accordance with paragraph 5.4 above, as well as with the ADS data and the specific data elements for event data recorder collected in accordance with Section 9 of Annex II.

• 5.5.The manufacturer shall have processes to manage potential safety-relevant gaps post-registration and to update the vehicles if necessary.

• 5.6.The manufacturer shall demonstrate that periodic independent internal process audits (e.g. every 2 years) are carried out to ensure that the processes established in accordance with points 5.1 to 5.5. are implemented consistently.

• 5.7.The manufacturer shall ensure that the organisation safety management system complies with the requirements of points 5.1. (except for vehicle related aspects like ‘operation’ and ‘decommissioning’), 5.2, 5.3 and 5.6 by means of arrangements with any organization involved in the development, manufacturing, or in-use deployment of its ADS and its features (including, where relevant, contracted suppliers, service providers, or manufacturers’ sub-organizations). Such arrangements may include, where appropriate, contractual arrangements, clear interfaces or a quality management system.

• 5.8.Certificate of compliance for safety management system

• 5.8.1.An application for a Certificate of Compliance for Safety Management System shall be submitted by the manufacturer or by their duly accredited representative to the type-approval authority.

• 5.8.2.It shall be accompanied by the undermentioned documents in triplicate, and by the following particular:

• (a) Documents describing the Safety Management System.

• (b) A signed Declaration of Compliance of the SMS with all the requirements for safety management according to this Regulation, using the model as defined in Appendix 3 to this Annex.

• 5.8.3.When this audit of the SMS has been satisfactorily completed and in receipt of a signed declaration from the manufacturer according to the model as defined in Appendix 3, a certificate named Certificate of Compliance for SMS as described in Appendix 4 (hereinafter the Certificate of Compliance for SMS) shall be granted to the manufacturer.

• 5.8.4.The Certificate of Compliance for SMS shall remain valid for a maximum of three years from the date of deliverance of the certificate unless it is withdrawn.

• 5.8.5.The type-approval authority may at any time verify that the requirements for the Certificate of Compliance for SMS continue to be met. The type-approval authority shall withdraw the Certificate of Compliance for SMS if major non-conformities in the compliance with the requirements laid down in this Regulation are discovered and not immediately addressed.

• 5.8.6.The manufacturer shall inform the type-approval authority or its technical service of any change that will affect the relevance of the certificate of compliance for SMS. After consultation with the manufacturer, the type-approval authority or its Technical Service shall decide whether new checks are necessary.

• 5.8.7.In due time, the manufacturer shall apply for a new or for the extension of the existing Certificate of Compliance for SMS. The type-approval authority shall, subject to a positive audit, issue a new Certificate of Compliance for SMS or extend its validity for a further period of three years. The type-approval authority shall verify that the SMS continue to comply with the requirements of this Regulation. The type-approval authority shall issue a new certificate in cases where changes have been brought to the attention of the type-approval authority or its Technical Service and the changes have been positively re-assessed.

• 5.8.8.The expiry or withdrawal of the manufacturer’s Certificate of Compliance for SMS shall be considered, with regard to the vehicle types to which the SMS concerned was relevant, as modification of approval, which may include the withdrawal of the approval if the conditions for granting the approval are not met anymore.

6. Reporting provision

• 6.1.The reporting of the safety assessment of the ADS safety concept as well as the audit of the safety management system of the manufacturer shall be performed in such a manner that allows traceability, e.g. versions of documents inspected are coded and listed in the records of the Technical Service.

• 6.2.An example of layout for the report on the assessment of the ADS safety concept from the Technical Service to the type-approval authority is provided in Appendix 1 to this part. The listed items in this Appendix are outlined as minimum set of items that need to be covered.

• 6.3.The granting type-approval authority shall issue the safety assessment results to be annexed to the type-approval certificate based on the documentation provided by the manufacturer, the report of the assessment of the ADS safety concept by the technical service and on the outcomes of the verification and test campaigns performed in accordance with Part 3 of this Annex. An example of a possible layout for the safety assessment results is given in Appendix 4.

7. Competence of the auditors/assessors

• 7.1.The assessment of the ADS safety concept and the audit of the safety management system under this part shall only be conducted by assessors/auditors with the technical and administrative knowledge necessary for such purposes. They shall in particular be competent as auditor/assessor for ISO 26262-2018 (Functional Safety – Road Vehicles), and ISO/PAS 21448 (Safety of the Intended Functionality of road vehicles); and shall be able to make the necessary link with cybersecurity aspects in accordance with UN Regulation No 155 and ISO/SAE 21434). This competence shall be demonstrated by appropriate qualifications or other equivalent training records.

Appendix 1

Model for the assessment report of the ADS safety concept

Safety assessment report No:

• 1.Identification

• 1.1.Vehicle make

• 1.2.Vehicle type

• 1.3.Means of identification of vehicle type if marked on the vehicle

• 1.4.Location of that marking

• 1.5.Manufacturer’s name and address

• 1.6.If applicable, name and address of manufacturer’s representative

• 1.7.Manufacturer’s formal documentation package

• Documentation reference No:

• Date of original issue:

• Date of latest update:

• 2.Assessment method

• 2.1.Description of the assessment processes and methodologies

• 2.2.Acceptability criteria

• 3.Results of the review of the documentation package

• 3.1.Review of the ADS description

• 3.2.Review of Manufacturer’s safety concept and the manufacturer safety analysis

• 3.3.Review of the Verification and Validation performed by the manufacturer in particular coverage of the different tests and setting minimum coverage thresholds for various metrics

• 3.4.Review of the methods and tools (software, laboratory, others) and the credibility assessment

• 3.5.Review of ADS data requirements and specific data elements for event data recorder for fully automated vehicles

• 3.6.Checks of the Cyber Security and Software Updates certificates are covering he ADS

• 3.7.Review of the information provided in the Operating Manual

• 3.8.Review of the provisions for the periodic roadworthiness tests of the ADS

• 3.9.Review of additional information not included in the Information Document

• 4.Verification of ADS functions under non-failure conditions (referred to in point 4.1.1. of Annex III Part 2 to Commission Implementing Regulation (EU) 2022/1426 of 5 August 2022 laying down rules for the application of Regulation (EU) 2019/2144 of the European Parliament and of the Council as regards uniform procedures and technical specifications for the type-approval of the automated driving system (ADS) of fully automated vehicles ([^E0013])

• 4.1.Rationale for the selection of test scenarios

• 4.2.Selected test scenarios

• 4.3.Test reports

• 4.3.1.Test No (add as many as the tests performed)

• 4.3.1.1.Objectives of the test

• 4.3.1.2.Test conditions

• 4.3.1.3.Measured quantities and measuring devices

• 4.3.1.4.Acceptability criteria

• 4.3.1.5.Test results

• 4.3.1.6.Comparison with the manufacturer’s supplied documentation

• 5.Verification of ADS safety concept under failure (referred to in point 4.1.2. of Annex III Part 2 to Implementing Regulation (EU) 2022/1426

• 5.1.Rationale for the selection of test scenarios

• 5.2.Selected test scenarios

• 5.3.Test reports

• 5.3.1.Test No (add as many as the tests performed)

• 5.3.1.1.Objectives of the test

• 5.3.1.2.Test conditions

• 5.3.1.3.Measured quantities and measuring devices

• 5.3.1.4.Acceptability criteria

• 5.3.1.5.Test results

• 5.3.1.6.Comparison with the manufacturer’s supplied documentation

• 6.Safety management system certificate (shall be appended to this test report)

• 7.Date of the assessment

• 8.Final judgement on the safety assessment outcome

• 9.This assessment has been carried out and the results reported in accordance with Implementing Regulation (EU) 2022/1426

• Technical Service carrying out the assessment

• Signed: … Date: …

• 10.Comments:

Appendix 2

Model of the ADS assessment results to be annexed to the type-approval certificate

• 1.Identification

• 1.1.Vehicle make

• 1.2.Vehicle Type

• 1.3.Means of identification of vehicle type if marked on the vehicle

• 1.4.Location of that marking

• 1.5.Manufacturer’s name and address

• 1.6.If applicable, name and address of manufacturer’s representative

• 1.7.Manufacturer’s formal documentation package

• Documentation reference No:

• Date of original issue:

• Date of latest update:

• 2.Assessment method

• 2.1.Description of the assessment processes and methodologies

• 2.2.Acceptability criteria

• 3.Verification of ADS functions under non-failure conditions (referred to in point 4.1.1. of Annex III Part 2 to Implementing Regulation (EU) 2022/1426

• 3.1.Rationale for the selection of test scenarios

• 3.2.Selected test scenarios

• 4.Verification of ADS safety concept under single failure (referred to in point 4.1.2. of Annex III Part 2 to Implementing Regulation (EU) 2022/1426

• 4.1.Rationale for the selection of test scenarios

• 4.2.Selected test scenarios

• 5.Assessment results

• 5.1.Results of the review of the Information Document

• 5.2.Results of the verification of ADS functions under non-failure conditions

• 5.3.Results of the verification of ADS safety concept under single failure

• 5.4.Results of the assessment of the Safety Management System

• 5.5.Results of the verification of provisions for the periodic roadworthiness tests

• 6.Final judgement on the safety assessment outcome

Appendix 3

Model of Manufacturer’s Declaration of Compliance for SMS

Manufacturer’s declaration of compliance with the requirements for the Safety Management System

Manufacturer’s Name:

Manufacturer’s Address:

…(Manufacturer’s Name) attests that the necessary processes to comply with the requirements for the Safety Management System laid down in Implementing Regulation (EU) 2022/1426 are installed and will be maintained.

Done at: … (place)

Date:

Name of the signatory:

Function of the signatory:

(Stamp and signature of the manufacturer’s representative)

Appendix 4

Model of Certificate of Compliance for SMS

Certificate of Compliance for Safety Management System

With Implementing Regulation (EU) 2022/1426

Certificate number [Reference number]

[… Type-approval authority]

Certifies that

Manufacturer: …

Address of the manufacturer:

complies with the provisions of Implementing Regulation (EU) 2022/1426

Checks have been performed on:

by (name and address of the type-approval authority or Technical Service):

Number of report:…

The certificate is valid until […Date]

Done at […Place]

On […Date]

[…Signature]

Attachments: description of the Safety Management System by the manufacturer.

PART 3

TESTS

1. General provisions

Pass- and fail-criteria to assess ADS safety shall be based on the requirements set out in Annex II and the scenario described in Part 1 of this annex. The requirements are defined in such a way that the pass/fail criteria can be derived not only for a specific set of test parameters, but also for all safety-relevant combinations of parameters that may occur in the operating conditions covered by the type approval and the specified operating range (e.g., speed range, longitudinal and transverse acceleration range, radii of curvature, brightness, number of lanes). For conditions not tested but that may occur within the defined ODD of the system, the manufacturer shall demonstrate as part of the assessment described in Part 2 to the satisfaction of the type-approval authority, that the vehicle is safely controlled.

These tests shall confirm the minimum performance requirements described in Annex II and the functionality of the ADS and the safety concept of the manufacturer as described in Part 2 of this Annex. Test results shall be documented and reported in accordance with point 6 of Part 2 of this annex.

These tests shall also confirm, that the ADS complies with the traffic rules, adapts its operations to environmental conditions, avoids disruption to the flow of traffic (e.g. blocking the lane because of too many MRMs), does not show unpredictable behaviour and shows reasonable cooperative and anticipatory behaviour in relevant situations (i.e. merging in dense traffic or in vicinity of vulnerable road users).

2. Test site

The test site shall comprise characteristics (example: friction value) that correspond to the specified ODD of the ADS. As necessary to apply the specific conditions of the ODD of the ADS, physical tests will be performed within the actual ODD (on-road) or at any test facility that replicates the ODD conditions and shall be determined by the manufacturer and the type approval authority. The ADS shall be tested on-road in accordance with the applicable law of the Member States and provided that tests can be carried out safely and without any risk to other road users.

3. Environmental conditions

Tests shall be carried out under different environmental conditions, within the limits of the defined ODD for the ADS. For environmental conditions not tested that may occur within the defined ODD, the manufacturer shall demonstrate as part of the assessment to the satisfaction of the type-approval authority that the vehicle is safely controlled.

To test the requirements for failure of functions, self-testing of the ADS and initiation and implementation of a minimal risk manoeuvre, errors may be artificially induced and the vehicle may be artificially brought into situations where it reaches the limits of the defined operating range (e.g., environmental conditions).

4. System modifications for testing purposes

If ADS modifications are required in order to allow testing, e.g. road type assessment criteria or road type information (map data), it shall be ensured that these modifications do not affect the test results. These modifications shall in principle be documented and annexed to the test report. The description and the evidence of influence (if any) of these modifications shall be documented and annexed to the test report.

5. Vehicle conditions

• 5.1.Test mass

• The subject vehicle shall be tested with any permissible vehicle load. No load alteration shall be made once the test procedure has begun. The manufacturer shall demonstrate, through the use of documentation, that the ADS works at all load conditions.

• 5.2.The subject vehicle shall be tested at the tyre pressure recommended by the manufacturer.

• 5.3.It shall be verified, that the condition of the system is according to the intended testing purpose (e.g. in a fault-free condition or with the specific faults to be tested).

6. Test tools

In addition to real vehicles, state-of-the-art test tools may be used to carry out the tests, replacing real vehicles and other road users (e.g., soft targets, mobile platforms, etc.). The replacement test tools shall comply with the characteristics relevant for sensory performance assessment, real vehicles and other traffic participants. Tests shall not be carried out in a way that would endanger the personnel involved, and significant damage of the vehicle being tested must be avoided where other means of validation are available.

7. Test parameter variation

The manufacturer shall declare the system boundaries to the type-approval authority. The type-approval authority shall define different combinations of test parameters (e.g. present speed of the vehicle, type and offset of target, curvature of lane, etc.) in order to test the ADS. The selected test cases shall provide sufficient test coverage for all scenarios, test parameters and environmental influences. Adequate robustness of the perceptions systems for the ADS against input/sensor data malfunction and adverse environmental conditions shall be demonstrated.

Test parameters selected by the type-approval authority shall be recorded in a test report in a manner that allows traceability and repeatability of the test setup.

8. Tests scenarios to assess the performance of the ADS on a test track (points 8.1., 8.2., 8.5, 8.6, 8.7, 8.8, 8.9.) and on-road (8.3., 8.4., 8.10.)

The scenarios included in the following points have to be considered a minimum set of tests. At the request of the type-approval authority, additional scenarios that are part of the ODD can be executed. If a scenario described in point 8 of this annex does not belong to the ODD of the vehicle, it shall not be taken into consideration.

Depending on the ODD, test scenarios shall be selected as part of the type-approval test. The test scenarios shall be selected in accordance with Part 1 of this annex. Type-approval testing may be carried out on the basis of simulations, manoeuvres on the test track and driving tests on real road traffic. However, it may not be based solely on computer simulations and at the time of type-approval, the type-approval authority shall conduct or shall witness at least the following tests to assess the behaviour of the ADS.

8.1.   Lane keeping

The test shall demonstrate that the fully automated vehicle does not leave its lane and maintains a stable motion inside its lane across the speed range and different curvatures within its system boundaries.

• 8.1.1.The test shall be based on the ODD of the ADS and shall be executed at least:

• a) with a minimum test duration of 5 minutes;

• b) with a passenger car target as well as a power-two-wheeler (PTW) target as the other vehicle;

• c) with a lead vehicle swerving in the lane; and

• d) with another vehicle driving close beside in the adjacent lane.

8.2.   Lane changing manoeuvre (LCM)

The tests shall demonstrate that the fully automated vehicle does not cause an unreasonable risk to safety of the vehicle occupants and other road users during a lane change procedure, and that the ADS is able to assess the criticality of the situation before starting the lane change manoeuvre (LCM) throughout the entire operational speed range. These tests are only required if the fully automated vehicle is capable of performing lane changes either during a Minimal risk manoeuvre or during regular operation.

• 8.2.1.The following tests shall be executed:

• a) with the fully automated vehicle performing lane change to the adjacent (target) lane;

• b) merging at lane end;

• c) merging into an occupied lane.

• 8.2.2.The tests shall be executed at least:

• a) with different vehicles, including a power two-wheeler (PTW) approaching from the rear;

• b) in a scenario where it is possible to execute a lane changing manoeuvre in regular operation;

• c) in a scenario where a lane changing manoeuvre in regular operation is not possible due to a vehicle approaching from the rear;

• d) with an equally fast vehicle following behind in the adjacent lane, preventing a lane change;

• e) with a vehicle driving beside in the adjacent lane preventing a lane change;

• f) in a scenario where a LCM during a minimal risk manoeuvre is possible and executed;

• g) in a scenario where the fully automated vehicle reacts to another vehicle that starts changing into the same space within the target lane, to avoid a potential risk of collision.

8.3.   Response to different road geometries

These tests shall ensure, that the fully automated vehicle detects and adapts to a variation of different road geometries which can occur within the intended ODD across its whole speed range.

• 8.3.1.The test shall be executed with at least the list of scenarios below based on the ODD of the ADS:

• a) T-junctions (3-way intersections) with and without traffic lights, with different rights of way;

• b) crossroads (4 or more way intersections) with and without traffic lights, with different rights of way;

• c) roundabouts.

• 8.3.2.Each test shall be executed at least:

• a) without a lead vehicle;

• b) with a passenger car target as well as a PTW target as the lead vehicle/other vehicle;

• c) with and without approaching or passing vehicles.

8.4.   Response to national traffic rules and road infrastructure

These tests shall ensure that the fully automated vehicle complies with national traffic rules and that it adapts to a various permanent and temporary changes of the road infrastructure (e.g. road construction sites) in the entire speed range.

• 8.4.1.The tests shall be executed with at least with the list of scenarios below that are relevant for the ODD of the ADS:

• a) different speed limit signs, so that the ADS has to change its speed according to the indicated values;

• b) signal lights and/or stop instructed by a road safety officer/enforcement agents with situations of going straight, turning left and right;

• c) pedestrian and cyclist crossings with and without pedestrians/cyclist approaching/on the road.

• d) temporary modifications: e.g., road maintenance operations indicated by traffic signs, cones and other signalisation, access restrictions.

• e) motorway entry, exit and toll stations.

• 8.4.2.Each test shall be executed at least:

• a) without a lead vehicle;

• b) with a passenger car target as well as a PTW target as the lead vehicle/other vehicle.

8.5.   Collision avoidance: Avoid a collision with road users or objects blocking the lane

The test shall demonstrate that the fully automated vehicle avoids a collision with a stationary vehicle, road user or fully or partially blocked lane up to the maximum specified speed of the ADS.

• 8.5.1.This test shall be executed at least with the following scenarios, where relevant in the ODD:

• a) with a stationary passenger car target;

• b) with a stationary PTW target;

• c) with a stationary pedestrian target;

• d) with a pedestrian target crossing the lane with a speed of 5 km/h, also in the presence of other objects relevant in the ODD (e.g. a ball, a shopping bag, etc.);

• e) with a pedestrian target moving a speed of up to 5 km/h within and partially occupying the lane of the ADS and following the same or the opposite direction of the fully automated vehicle;

• f) with a pedestrian target swerving in the same lane of the fully automated vehicle;

• g) with a cyclist target crossing the lane with a speed of 15 km/h;

• h) with a cyclist target which is travelling in the same direction with a speed of 15 km/h;

• i) with the fully automated vehicle turning right and crossing the path of the cyclist travelling in the same direction with a speed of 15 km/h;

• j) with a target representing a blocked lane;

• k) with a target partially within the lane;

• l) with one or more different types of unpassable objects relevant in the ODD (e.g., a dustbin, a fallen bicycle or scooter, a fallen traffic sign, a stationary or moving ball, etc.);

• m) with multiple consecutive obstacles blocking the lane relevant in the ODD (e.g., in the following order: ego-vehicle -motorcycle – car);

• n) on a curved section of road.

8.6.   Avoid emergency braking before a passable object in the lane. A ‘passable object’ is such an object, that may be rolled over without causing an unreasonable risk to the vehicle occupants or other road users.

The test shall demonstrate that the fully automated vehicle is not initiating an Emergency Braking with a deceleration demand greater than 5 m/s² due to a passable object in the lane relevant for the ODD (e.g., a manhole lid or a small branch) up to the maximum specified speed of the ADS.

• 8.6.1.This test shall be executed at least with the following scenarios, where relevant in the ODD:

• a) without a lead vehicle;

• b) with a passenger car target as well as a PTW target as the lead vehicle/other vehicle.

8.7.   Following a lead vehicle

The test shall demonstrate that the fully automated vehicle is able to maintain and restore a stable motion and a safety distance to a vehicle in front and is able to avoid a collision with a lead vehicle which decelerates up to its maximum deceleration.

• 8.7.1.This test shall be executed at least with the following scenarios, where relevant in the ODD:

• a) across the entire speed range of the fully automated vehicle;

• b) using a passenger car target a PTW target as well as a bicycle target as lead vehicle, provided standardised PTW targets suitable to safely perform the test are available;

• c) for constant and varying lead vehicle velocities (realistic speed profile);

• d) for straight and curved sections of road;

• e) for different lateral positions of lead vehicle in the lane;

• f) with a deceleration of the lead vehicle of at least 6 m/s² mean fully developed deceleration until standstill.

8.8.   Lane change of another vehicle into lane (cut-in)

The test shall demonstrate that the fully automated vehicle is capable of avoiding a collision with a vehicle or other road user cutting into the lane of the fully automated vehicle up to a certain criticality of the cut-in manoeuvre.

• 8.8.1.The criticality of the cut-in manoeuvre shall be determined according to the provisions introduced Part 1 of this annex and depending on the distance between the rear-most point of the cutting-in vehicle and front-most point of the fully automated vehicle.

• 8.8.2.The test shall be executed at least with the following scenarios, where relevant for the ODD:

• a) with different TTC, distance and relative velocity values of the cut-in manoeuvre, covering types of cut-in scenarios in which a collision can be avoided and those in which a collision cannot be avoided;

• b) with cutting-in vehicles travelling at constant longitudinal speed, accelerating and decelerating;

• c) with different lateral velocities, lateral accelerations of the cut-in vehicle;

• d) with a passenger car, PTW as well as bicycle targets as the cutting-in vehicle, provided standardised PTW targets suitable to safely perform the test are available.

8.9.   Stationary obstacle after lane change of the lead vehicle (cut-out)

The test shall demonstrate that the fully automated vehicle is capable of avoiding a collision with a stationary vehicle, road user or blocked lane that becomes visible after a preceding vehicle avoided a collision by an evasive manoeuvre. The test shall be based on the requirements set out in Annex II and scenario parameters in Part 1 of this annex. For conditions not tested that may occur within the defined operating range of the vehicle, the manufacturer shall demonstrate as part of the assessment described in Annex III, Part 2 to the satisfaction of the relevant authorities that the vehicle is safely controlled.

• 8.9.1.The test shall be executed at least with the following scenarios, where relevant for the ODD:

• a) with a stationary passenger car target centred in lane;

• b) with a PTW target centred in lane;

• c) with a stationary pedestrian target centred in lane;

• d) with a target representing a blocked lane centred in lane;

• e) with multiple consecutive obstacles blocking the lane (e.g. in the following order: ego-vehicle – lane change vehicle – motorcycle – car).

8.10.   Parking

The test shall demonstrate that the ADS is able to park in different parking spaces and parking layouts under different conditions; and that during the parking manoeuvre it is not causing damage to the surrounding objects, road users and itself.

• 8.10.1.The test shall be executed at least with the following scenarios, where relevant for the ODD:

• (a) with parking spaces parallel and perpendicular to the road;

• (b) on even and slant surfaces;

• (c) with other vehicles in the surrounding parking spaces, including PTWs and bicycles;

• (d) parking to parking spaces with different geometrical dimensions;

• (e) on different road slope angles;

• (f) with another vehicle cutting in to the parking space during the parking manoeuvre.

8.11.   Navigating in a parking facility

The test shall demonstrate that the ADS is able to handle the low driving speed and the general lack of visibility that may occur in a parking.

• 8.11.1.The test shall be executed at least with the following scenarios, where relevant for the ODD:

• a) with an initially obstructed pedestrian target crossing the path of the fully automated vehicle with a speed of 5 km/h.

• b) with a vehicle driving out of a parking place in front of the fully automated vehicle.

• c) with a stationary obstacle in the path of the fully automated vehicle.

• d) with different paths, where the infrastructure is obstructing the field of view.

• e) with a small obstacle on the floor after a ramp obstructed by other objects in the path of the fully automated vehicle.

8.12.   Specific scenarios for motorway

• 8.12.1.Motorway entry

• The test shall demonstrate that the ADS is able to safely enter the motorway.

• 8.12.1.1.The test shall be executed at least with the following scenarios, where relevant for the ODD:

• a) with different vehicles, including a PTW approaching from the rear;

• b) with vehicles approaching with different speed from the rear;

• c) with a platoon of vehicles driving beside in the adjacent lane.

• 8.12.2.Motorway exit

• The test shall demonstrate that the ADS is able to safely exit the motorway.

• 8.12.2.1.The test shall be executed at least with the following scenarios, where relevant for the ODD:

• a) without a lead vehicle;

• b) with a passenger car target as well as a PTW target as the lead vehicle/other vehicle.

• c) with other vehicle(s) or obstacle(s) blocking the motorway exit.

• 8.12.3.Toll station

• The test shall demonstrate that the ADS is able to select the proper passing gate, and adapt its speed to that permitted within the toll area.

• 8.12.3.1.The test shall be executed at least with the following scenarios, where relevant for the ODD:

• a) with and without a lead vehicle;

• b) with other vehicles blocking the passing gate(s);

• c) with closed and opened passing gates.

• d) with different permitted speeds in the toll area.

8.13.   For dual mode vehicles, transition between the manual driving mode and the fully automated mode.

The test shall demonstrate that the ADS takes over the DDT in a safe manner and only when the vehicle is standstill.

• 8.13.1.The test shall be executed at least with the following scenarios, where relevant for the ODD:

• a) with and without human driver present in the vehicle;

• b) with open and closed vehicle doors;

• c) with and without obstacles around the vehicle,

• d) inside and outside of the dedicated parking area, if applicable

• 8.13.2.The test shall be executed at least with the following scenarios, where relevant for the ODD:

• a) in a situation where transition is possible and executed;

• b) in a situation where transition is not possible to be executed.

PART 4

PRINCIPLES FOR CREDIBILITY ASSESSMENT FOR USING VIRTUAL TOOLCHAIN IN ADS VALIDATION

1. General

• 1.1.The credibility can be achieved by investigating and assessing five properties of Modelling and Simulation (M&S):

• (a) capability – what can the M&S do, and what the risks are associated with it;

• (b) accuracy – how well does M&S reproduce the target data;

• (c) correctness – how sound & robust are M&S data and algorithms;

• (d) usability – what training and experience is needed.

• (e) fit for purpose – how suitable is the M&S for the ODD and ADS assessment.

• 1.2.At the same time, the credibility assessment framework shall be general enough to be used for different M&S types and applications. However, the goal is complicated by the broad differences between ADS features and the variety of M&S types and applications. These considerations require a (risk-based/informed) credibility assessment framework relevant and appropriate to all M&S applications.

• 1.3.The credibility assessment framework provides a general description of the main aspects considered for assessing the credibility of an M&S solution together with principles on the role of third parties assessors in the validation process with respect to credibility. Concerning the latter point, the type-approval authority shall investigate the produced documentation supporting credibility at the assessment phase, whereas the actual validation tests occur once the manufacturer has developed the integrated simulation systems.

• 1.4.Ultimately, the outcome of the current credibility assessment shall define the envelope in which the virtual tool can be used to support the ADS assessment.

• 1.5.The requirements of this part are therefore intended to demonstrate the credibility of any simulation model or virtual toolchain for its use in ADS validation.

2. Definitions

For the purpose of this annex

• 2.1.‘abstraction’ means the process of selecting the essential aspects of a source system or referent system to be represented in a model or simulation, while ignoring non-relevant aspects. Any modelling abstraction carries with it the assumption that shall not significantly affect the intended uses of the simulation tool.

• 2.2.‘closed loop testing’ means a virtual environment that takes the actions of the element-in-the loop into account. Simulated objects respond to the actions of the system (e.g. system interacting with a traffic model).

• 2.3.‘deterministic’ means a term describing a system whose evolution over time can be predicted exactly and a given set of input stimuli will always produce the same output.

• 2.4.‘driver-in-the-loop (DIL)’ is typically conducted in a driving simulator used for testing the human–automation interaction design. DIL has components for the driver to operate and communicate with the virtual environment.

• 2.5.‘Hardware-In-the-Loop (HIL)’ involves the final hardware of a specific vehicle sub-system running the final software with input and output connected to a simulation environment to perform virtual testing. HIL testing provides a way of replicating sensors, actuators and mechanical components in a way that connects all the I/O of the electronic control units (ECU) being tested, long before the final system is integrated.

• 2.6.‘model’ is a description or representation of a system, entity, phenomenon, or process.

• 2.7.‘model calibration’ is the process of adjusting numerical or modelling parameters in the model to improve agreement with a referent.

• 2.8.‘model Parameter’ are numerical values used to support characterizing a system functionality. A model parameter has a value that cannot be observed directly in the real world but that must be inferred from data collected in the real world (in the model calibration phase).

• 2.9.‘model-in-the-loop (MIL)’ is an approach which allows quick algorithmic development without involving dedicated hardware. This level of development usually involves high-level abstraction software frameworks running on general-purpose computing systems.

• 2.10.‘open loop testing’ means a virtual environment that does not take the actions of the element-in-the loop into account (e.g. system interacting with a recorded traffic situation).

• 2.11.‘probabilistic’ is a term pertaining to non-deterministic events, the outcomes of which are described by a measure of likelihood.

• 2.12.‘proving ground or test-track’ is a physical testing facility closed to the traffic where the performance of an ADS can be investigated on the real vehicle. Traffic agents can be introduced via sensor stimulation or via dummy devices positioned on the track.

• 2.13.‘sensor stimulation’ is a technique whereby artificially generated signals are provided to the element under testing in order to trigger it to produce the result required for verification of the real world, training, maintenance, or for research and development.

• 2.14.‘simulation’ is the imitation of the operation of a real-world process or system over time.

• 2.15.‘simulation model’ is a model whose input variables vary over time.

• 2.16.‘simulation toolchain’ is a combination of simulation tools that are used to support the validation of an ADS.

• 2.17.‘software-in-the-loop (SIL)’ is where the implementation of the developed model will be evaluated on general-purpose computing systems. This step can use a complete software implementation very close to the final one. SIL testing is used to describe a test methodology, where executable code such as algorithms (or even an entire controller strategy), is tested within a modelling environment that can help prove or test the software.

• 2.18.‘stochastic’ means a process involving or containing a random variable or variables. Pertaining to chance or probability.

• 2.19.‘validation of the simulation model’ is the process of determining the degree to which a simulation model is an accurate representation of the real world from the perspective of the intended uses of the tool.

• 2.20.‘vehicle-in-the-loop (VIL)’ is a fusion environment of a real testing vehicle in the real-world and a virtual environment. It can reflect vehicle dynamics at the same level as the real-world and it can be operated on a vehicle test bed or on a test track.

• 2.21.‘verification of the simulation model’ is the process of determining the extent to which a simulation model or a virtual testing tool is compliant with its requirements and specifications as detailed in its conceptual models, mathematical models, or other constructs.

• 2.22.‘virtual testing’ is the process of testing a system using one or more simulation models.

3. Components of the credibility assessment framework and related documentation requirements

• 3.1.The credibility assessment framework introduces a way to assess and report the credibility of M&S based on quality assurance criteria where the levels of confidence in the results can be indicated. In other words, the credibility is established by evaluating the following M&S influencing factors that are considered as main contributors for M&S properties and therefore for the overall M&S credibility: (a) M&S management; (b) team’s experience and expertise; (c) M&S analysis and description; (d) data/input pedigree and (e) verification; validation, uncertainty Characterisation. Each of these factors indicates the level of quality achieved by M&S, and the comparison between the obtained levels and the required levels shall determine whether the M&S is credible and fit to use for virtual testing. A graphical representation of the relationship between the components of the credibility assessment framework is shown below.

• Image omitted; see official source

• 3.2.Models and simulation management.

• 3.2.1.The M&S lifecycle is a dynamic process with frequent releases that shall be monitored and documented. Management activities shall be established to support the M&S in a work product management fashion. Relevant information on the following aspects shall be provided.

• 3.2.2.The M&S management process shall:

• (a) Describe the modifications within the releases;

• (b) Designate the corresponding software (e.g., specific software product and version) and hardware arrangement (e.g., XiL configuration);

• (c) Record the internal review processes that accepted the new releases;

• (d) Be supported throughout the full duration of the virtual model utilization.

• 3.2.3.Release management.

• 3.2.3.1.Any M&S toolchain’s version used to release data for certification purposes shall be stored. The virtual models constituting the testing toolchain shall be documented in terms of the corresponding validation methods and acceptance thresholds to support the overall credibility of the toolchain. The developer shall enforce a method to trace generated data to the corresponding M&S version.

• 3.2.3.2.Quality check of virtual data. Data completeness, accuracy, and consistency shall be ensured throughout the releases and lifetime of an M&S toolchain to support the verification and validation procedures.

• 3.2.4.Team’s experience and expertise.

• 3.2.4.1.Even though experience and expertise (E&E) are already covered in a general sense within the organization, it is important to establish the basis for confidence in the specific E&E for M&S activities.

• 3.2.4.2.The credibility of M&S depends not only on the quality of the simulation models but also on the E&E of the personnel involved in the validation and usage of the M&S. For instance, a proper understanding of the limitations and validation domain will prevent the possible misuse of M&S or misinterpretation of its results.

• 3.2.4.3.Therefore, it is important to establish the basis for the manufacturer’s confidence on the E&E of:

• (a) The Teams that will validate the simulation toolchain and,

• (b) The Teams that will use the validated simulation for the execution of virtual testing with the purpose of validating the ADS.

• 3.2.4.4.A proper management of the team’s E&E increases the level of confidence on the credibility of M&S and its outcomes by ensuring that the human factors behind the M&S are taken into consideration and any possible human component risk is controlled, as is expected in any suitable Management System

• 3.2.4.5.If the manufacturer’s tool chain incorporates or relies upon inputs from organisations or products outside of the manufacturer’s own team, the manufacturer will provide an explanation of measures it has taken to support its confidence in the quality and integrity of those inputs.

• 3.2.4.6.Team’s E&E consists of two levels.

• 3.2.4.6.1.Organizational level

• The credibility is established by setting up processes and procedures to identify and maintain skills, knowledge, and experience to perform M&S activities. The following processes shall be established, maintained and documented:

• (i) Process to identify and evaluate the individual’s competence and skills;

• (ii) Process for training competent personnel to perform M&S-related duties

• 3.2.4.6.2.Team level

• Once a M&S has been finalised, its credibility is mainly dictated by the skills and knowledge of the individual/team that will validate the M&S toolchain and use the M&S for the validation of the ADS. Credibility is established by documenting that these teams have received adequate training to fulfil their duties.

• The manufacturer shall then:

• (i) provide the basis for the manufacturer’s confidence in the E&E of the individual/team that validates the M&S Toolchain.

• (ii) provide the basis for the manufacturer’s confidence in the E&E of the individual/team that uses the simulation to carry out virtual testing in order to validate the ADS.

• The manufacturer’s demonstration of how it applies the principles of ISO 9001 or a similar best practice or standard to ensure the competence of its M&S organization and the individuals in that organisation will be the basis for this determination. The type-approval authority may not substitute its judgment on the E&E of the organisation or its members with that of the manufacturer.

• 3.2.5.Data/input pedigree

• 3.2.5.1.The data/input pedigree contains a record of traceability from the manufacturer’s data used in the validation of the M&S.

• 3.2.5.2.Description of the data used for the M&S

• (a) The manufacturer shall document the data used to validate the model and note important quality characteristics;

• (b) The manufacturer shall provide documentation showing that the data used to validate the models covers the intended functionalities that the toolchain aims at virtualising;

• (c) The manufacturer shall document the calibration procedures used to fit the virtual models’ parameters on the collected input data.

• 3.2.5.3.Effect of the data quality (e.g. data coverage, signal to noise ratio, and sensors’ uncertainty/bias/sampling rate) on model parameters uncertainty.

• The quality of the data used to develop the model will affect the estimation and calibration of the model parameters. Uncertainty in model parameters will be another important aspect in the final uncertainty analysis.

• 3.2.6.Data/output pedigree

• 3.2.6.1.The data/output pedigree contains a record of the M&S outputs used for the ADS validation.

• 3.2.6.2.Description of the data generated by the M&S

• (a) The manufacturer shall provide information on any data and scenarios used for virtual testing toolchain validation.

• (b) The manufacturer shall document the exported data and note important quality characteristics.

• (c) The manufacturer shall trace a M&S output to the corresponding simulation setup

• 3.2.6.3.Effect of data quality on M&S credibility

• (a) The M&S output data shall be sufficiently wide to ensure the correct execution of the validation computation. The data shall sufficiently reflect the ODD relevant to the virtual assessment of the ADS.

• (b) The output data shall allow for a consistency/sanity check of the virtual models via possibly exploiting redundant information

• 3.2.6.4.Managing stochastic models

• (a) Stochastic models shall be characterised in terms of their variance

• (b) Stochastic models shall be ensured the possibility of deterministic re-execution

• 3.3.M&S analysis and description

• 3.3.1.The M&S analysis and description aim to define the whole M&S and identify the parameter space that can be assessed via virtual testing. It defines the scope and limitations of the models and toolchain and the uncertainty sources that can affect its results.

• 3.3.2.General description

• 3.3.2.1.The manufacturer shall provide a description of the complete toolchain along with how the simulation data will be used to support the ADS validation strategy

• 3.3.2.2.The manufacturer shall provide a clear description of the test objective

• 3.3.3.Assumptions, known limitations and uncertainty sources

• 3.3.3.1.The manufacturer shall motivate the modelling assumptions that guided the design of the M&S toolchain

• 3.3.3.2.The manufacturer shall provide evidence on:

• (i) How the manufacturer-defined assumptions play a role in defining the limitations of the toolchain;

• (ii) The level of fidelity required for the simulation models

• 3.3.3.3.The manufacturer shall provide justification that the tolerance for sim-real correlation is acceptable for the test objective

• 3.3.3.4.Finally, this section shall include information on the sources of uncertainty in the model. This will represent an important input to final uncertainty analysis, which will define how the model outputs can be affected by the different sources of uncertainty of the model used.

• 3.3.4.Scope (how the M&S is used in the ADS validation)

• 3.3.4.1.The credibility of the virtual tool shall be enforced by a clearly-defined scope of utilization the developed models.

• 3.3.4.2.The matured M&S shall allow a virtualisation of the physical phenomena to a degree of accuracy which matches the fidelity level required for certification. Thus, the M&S will act as a ‘virtual proving ground’ for ADS testing.

• 3.3.4.3.Simulation models need dedicated scenarios and metrics for validation. The scenario selection used for validation shall be sufficient so that the toolchain will perform in the same manner in scenarios outside of the validation scope.

• 3.3.4.4.The Manufacturer shall provide a list of validation scenarios together with the limitations of the corresponding parameters.

• 3.3.4.5.The ODD analysis is a crucial input to derive requirements, scope and effects that the M&S must consider in order to support ADS validation.

• 3.3.4.6.Parameters generated for the scenarios will define extrinsic and intrinsic data for the toolchain and the simulation models.

• 3.3.5.Criticality assessment

• 3.3.5.1.The simulation models and the simulation tools used in the overall tool-chain shall be investigated in terms of their responsibility in case of a safety error in the final product. The proposed approach for criticality analysis is derived from ISO 26262, which requires qualification for some of the tools used in the development process.

• 3.3.5.2.In order to derive how critical the simulated data are, the criticality assessment shall consider the following parameters:

• (a) The consequences for human safety e.g. severity classes in ISO 26262.

• (b) The degree to which the simulated results influence’s the ADS

• 3.3.5.3.From the perspective of the criticality assessment, the three possible cases for assessment are:

• (a) Those models or tools that are clear candidates for following a full credibility assessment;

• (b) Those models or tools that may or may not be candidates for following the full credibility assessment at the discretion of the assessor;

• (c) Those models or tools that are not required to follow the credibility assessment.

• 3.4.Verification

• 3.4.1.The verification of an M&S involves analysing the correct implementation of the conceptual/mathematical models building up the M&S toolchain. The verification contributes to the M&S’s credibility by providing assurance that the M&S will not exhibit unrealistic behaviour for a set of inputs that cannot be tested. The procedure is based on a multi-step approach including code verification, calculation verification and sensitivity analysis.

• 3.4.2.Code verification

• 3.4.2.1.Code verification involves tests demonstrating that no numerical/logical flaws affect the virtual models

• 3.4.2.2.The manufacturer shall document the execution of proper code verification techniques, e.g. static/dynamic code verification, convergence analysis and comparison with exact solutions if applicable

• 3.4.2.3.The manufacturer shall provide documentation showing that the exploration in the domain of the input parameters was sufficiently wide to identify parameters’ combination for which the M&S shows unstable or unrealistic behaviour. Coverage metrics of combinations of parameters may be used to demonstrate the required exploration of the models behaviours.

• 3.4.2.4.The manufacturer shall adopt sanity/consistency checking procedures whenever data allows.

• 3.4.3.Calculation verification

• 3.4.3.1.Calculation verification deals with the estimation of numerical errors affecting the M&S

• 3.4.3.2.The manufacturer shall document numerical error estimates (e.g. discretization error, rounding error, iterative procedures convergence);

• 3.4.3.3.The numerical errors shall be kept sufficiently bounded to not affect validation.

• 3.4.4.Sensitivity analysis

• 3.4.4.1.Sensitivity analysis aims to quantify how model output values are affected by changes in the model input values and thus to identify the parameters having the greatest impact on the simulation model results. The sensitivity study also helps to determine the extent to which the simulation model satisfies the validation thresholds when it is subjected to small variations of the parameters. It is therefore fundamental to support the credibility of the simulation results.

• 3.4.4.2.The manufacturer shall provide supporting documentation demonstrating that the most critical parameters influencing the simulation output have been identified by means of sensitivity analysis techniques such as by applying a perturbation of the model’s parameters;

• 3.4.4.3.The manufacturer shall demonstrate that robust calibration procedures have been adopted when identifying and calibrating the most critical parameters in order to increase the credibility of the developed toolchain.

• 3.4.4.4.Ultimately, the sensitivity analysis results will also help to define the inputs and parameters whose uncertainty Characterisation needs particular attention in order to properly define the uncertainty of the simulation results.

• 3.4.5.Validation

• 3.4.5.1.The quantitative process of determining the degree to which a model or a simulation is an accurate representation of the real world from the perspective of the intended uses of the M&S requires the selection and definition of several elements.

• 3.4.5.2.Measures of performance (metrics)

• 3.4.5.2.1.The measures of performance are the metrics used to compare the simulation model with the real world. Measures of performance are defined during the M&S analysis.

• 3.4.5.2.2.Metrics for validation may include:

• (i) discrete value analysis e.g. detection rate, firing rate;

• (ii) time evolution e.g. positions, speeds, acceleration;

• (iii) flow of actions-based analysis e.g. distance/speed calculations, TTC calculation, brake initiation.

• 3.4.5.3.Goodness of fit measures

• 3.4.5.3.1.The analytical frameworks are used to compare real world and simulation metrics. They are generally key performance indicators (KPIs) indicating the statistical comparability between two sets of data.

• 3.4.5.3.2.The validation shall show that these KPIs are met.

• 3.4.5.4.Validation methodology

• 3.4.5.4.1.The manufacturer shall define the logical scenarios used for virtual testing toolchain validation. They shall be able to cover to the maximum possible extent the ODD of virtual testing for ADS validation

• 3.4.5.4.2.The exact methodology depends on the structure and purpose of the toolchain. The validation may consist of one or more of the following:

• (i) validate subsystem models e.g. environment model (road network, weather conditions, road user interaction), sensor models (radio detection and ranging (RADAR), light detection and ranging (LiDARs), camera), vehicle model (steering, braking, powertrain);

• (ii) validate vehicle system (vehicle dynamics model together with the environment model);

• (iii) validate sensor system (sensor model together with the environment model);

• (iv) validate integrated system (sensor model + environment model with influences from vehicle model).