What is the importance of validating the models on time?

Every project is a compromise between cost-efficiency, speed and quality. In IFRS9 at the moment, time dominates: the models must be validated by the 2018 deadline. So that implies a compromise in quality or an increase in resource and cost. Each bank strikes its own balance, of course, but the IFRS9 regulator has also shown a pragmatic approach to this problem – the models’ level of sophistication can be agreed with the Auditor to grow over time with experience.

This is fine, but it is a headache for the validator. What standard (of model and of validation) applies now, and what does it grow into over time? The model must be approved in the end, so where do we find (and put in policy) the dynamic time-dependent equilibrium between permission and challenge?

I have no easy answer here, but I think time-pressure will compel Auditors and model validators to allow more graduation in their procedures and policies than is often currently the case. That unfortunately needs more time and resources to manage, and so we return to where we started.

Could you elaborate on the best practices of validation and backtesting of IFRS 9 models?

The validation of IFRS9 models is quite different to the validation of IRB models that we’re all more used to. I think validation has to rethink
many of its principles and practices to remain useful and active in this new IFRS9 world.
We all recognise that IFRS9 uses different, unfamiliar kinds of models and methodologies, so we’ll find corresponding differences in validation and a resulting demand for new technical skills and experience among the validators.

Less well recognised is the fact that IFRS9 restricts some of the most basic and trusted model risk mitigants available to IRB model validators. Validation outcomes such as conservative model adjustments, or the restriction of model usage, are encouraged, even required by IRB regulation. In contrast, IFRS9 models are expected to be intricate and accurate enough to deliver exact long-range predictions algorithmically for all assets in a given portfolio.

The risks for IFRS9 modelling are clear: over fitting and complexity, over-confidence in central estimates, structural weaknesses caused by models-within-models; and in this world of amplified model risk, validators are prevented from using their most useful and flexible model risk mitigants. IFRS9 validation is not going to work well if it is simply a restricted version of IRB model validation, so it needs to respond to other peculiar features of IFRS9.

IRB and IFRS9 models are calibrated by data, but IFRS9 models are particularly strongly influenced by pre-defined structures and assumptions. These structures can be financial and accounting-based, or implied by standards (e.g. the need to build exogenous factors explicitly into the models); and the assumptions that shape the models come from a wide group of experts in Credit Risk, Finance, Economics. All these points constrain the model’s structure and parameter ranges, defining the model’s essential behaviours and performance, long before the data arrives to refine the final calibrations.

So validators need to turn their challenge onto these pre-set constraints as much as onto the traditional data-driven estimates. They should reveal and challenge the underlying assumptions systematically and quantitatively, and review the whole model system, not just the individual models. This may need new statistical skills, calling on the Bayesian tools of subjective modelling for example, or of correlated networks. Quant validators will need to get their hands dirty speaking to the experts and eliciting their opinions and confidence. These are all fascinating areas of statistics and research, and I am excited they may become mainstream.

I am also keen to explore new modes of communicating IFRS9 model risk to the model owners and users. For example, should model failure or error be a feature of some of the forward-looking scenarios? I think we all accept that some model error is a probable scenario. This subtle mechanism for conservatism is the one small opening allowed in IFRS9 for compensating model error.

Who should be responsible for backtesting these models?

Thinking of backtesting as a data-based statistical process, I see no reason why this should be arranged differently from current practice in Credit Risk or Collective Provisions models. Banks have settled on first line or
second line or even third line to carry out these tests and they usually have good reasons.

On the other hand, IFRS9 models draw strongly on assumptions and expert opinions. These also need “backtesting”, i.e. a challenge that looks how well these assumptions hold up under past conditions. For this kind of test, with such diverse expert sources, I do think a single team needs to coordinate the challenge, with sufficient skill and experience. That looks like an independent validation function, probably second line.

What would you like to achieve by attending the 5th Annual Credit Risk Modelling under IFRS 9 Conference?

I’m looking forward to the usual mixture of good talks, good conversations and networking, as well as a general appreciation of where Banks are meeting IFRS9 challenges and how they’re overcoming them. I’d consider it a success for me, if the delegates come away knowing more of the new challenges IFRS9 puts on the validators, and of the special model risks of the IFRS9 approach.

Request a Brochure

Learn, explore and unleash your inner chef.

For more information, contact:
Yiota Andreou



Ahead of the 5th Annual Credit Risk Modelling under IFRS 9 Conference, we spoke with Alan Forrest, Head of Independent Validation at Clydesdale Bank about the best practices of validation and backtesting of IFRS 9 models. 

Previous Attendees Include:
ABN Amro Bank
Banco Espirito Santo
Bank of England
Bank of Ireland
Barclays Bank
BNP Paribas
Ceska Sporitelina AS
Commonwealth Bank of Australia
Credit Agricole
Credit Suisse
Credito Trevigiano
Danske Bank
Deutsche Bank
DZ Bank
Erste Group Bank AG
Intesa Sanpaolo
JP Morgan


About the Conference:

This marcus evans event provides the opportunity to discuss and compare solutions to modelling challenges with other banks before implementation and to benchmark their approaches to validation. The conference will also cover the most up-to-date expectations of supervisors. The 5th Annual Credit Risk Modelling under IFRS 9 Conference will take place from the 29th until the 30th of June 2017 at Hilton Canary Wharf in London, United Kingdom.

Copyright © 2017 Marcus Evans. All rights reserved.

Practical Insights From:

Erste Group

Deutsche Postbank



Lloyds Banking


FBN Bank

BNP Paribas



Habib Bank AG Zurich


Clydesdale Bank


About the speaker:

Alan Forrest is a Credit Risk Modeller and Model Validator, with over 15 years experience in UK Financial Services. He recently joined Clydesdale and Yorkshire Bank as Head of the Independent Model Validation Unit. leading a team of 6 experienced quantitative analysts and model validators to support the Bank’s aims in modelling all areas of Risk, including IFRS9.

Before joining CYBG, he led for 2 years the Commercial Credit Risk Modelling team in the Williams and Glyn Bank, within Royal Bank of Scotland, setting up the new function within the planned challenger bank. Previous to that, he had over 8 years experience validating Banking Book Credit Risk models, in RBS and in HBoS, covering retail and corporate areas, IRB and operational functions, as well as Economic Capital and Stress Testing models.

He has contributed actively to industry debate and to the methodology of Low Default Portfolios, and to the quantification of Model Risk in Credit Risk models. He is a Professional Statistician and frequent presenter at Industry Conferences, with a Masters in Applied Statistics and a PhD in Pure Mathematics, as well as over 10 years international experience in Mathematical Research and Lecturing at University level.


The best practices of validation and backtesting of IFRS 9 models




An interview with the Head of Independent Validation from Clydesdale Bank

Alan Forrest Head of Independent Validation at Clydesdale Bank

Fix the following errors: