What is the importance of validating the models on time?

Every project is a compromise between cost-efficiency, speed and quality. In IFRS9 at the moment, time dominates: the models must be validated by the 2018 deadline. So that implies a compromise in quality or an increase in resource and cost. Each bank strikes its own balance, of course, but the IFRS9 regulator has also shown a pragmatic approach to this problem – the models’ level of sophistication can be agreed with the Auditor to grow over time with experience.

This is fine, but it is a headache for the validator. What standard (of model and of validation) applies now, and what does it grow into over time? The model must be approved in the end, so where do we find (and put in policy) the dynamic time-dependent equilibrium between permission and challenge?

I have no easy answer here, but I think time-pressure will compel Auditors and model validators to allow more graduation in their procedures and policies than is often currently the case. That unfortunately needs more time and resources to manage, and so we return to where we started.

Could you elaborate on the best practices of validation and backtesting of IFRS 9 models?

The validation of IFRS9 models is quite different to the validation of IRB models that we’re all more used to. I think validation has to rethink
many of its principles and practices to remain useful and active in this new IFRS9 world.
We all recognise that IFRS9 uses different, unfamiliar kinds of models and methodologies, so we’ll find corresponding differences in validation and a resulting demand for new technical skills and experience among the validators.

Less well recognised is the fact that IFRS9 restricts some of the most basic and trusted model risk mitigants available to IRB model validators. Validation outcomes such as conservative model adjustments, or the restriction of model usage, are encouraged, even required by IRB regulation. In contrast, IFRS9 models are expected to be intricate and accurate enough to deliver exact long-range predictions algorithmically for all assets in a given portfolio.

The risks for IFRS9 modelling are clear: over fitting and complexity, over-confidence in central estimates, structural weaknesses caused by models-within-models; and in this world of amplified model risk, validators are prevented from using their most useful and flexible model risk mitigants. IFRS9 validation is not going to work well if it is simply a restricted version of IRB model validation, so it needs to respond to other peculiar features of IFRS9.

IRB and IFRS9 models are calibrated by data, but IFRS9 models are particularly strongly influenced by pre-defined structures and assumptions. These structures can be financial and accounting-based, or implied by standards (e.g. the need to build exogenous factors explicitly into the models); and the assumptions that shape the models come from a wide group of experts in Credit Risk, Finance, Economics. All these points constrain the model’s structure and parameter ranges, defining the model’s essential behaviours and performance, long before the data arrives to refine the final calibrations.

So validators need to turn their challenge onto these pre-set constraints as much as onto the traditional data-driven estimates. They should reveal and challenge the underlying assumptions systematically and quantitatively, and review the whole model system, not just the individual models. This may need new statistical skills, calling on the Bayesian tools of subjective modelling for example, or of correlated networks. Quant validators will need to get their hands dirty speaking to the experts and eliciting their opinions and confidence. These are all fascinating areas of statistics and research, and I am excited they may become mainstream.

I am also keen to explore new modes of communicating IFRS9 model risk to the model owners and users. For example, should model failure or error be a feature of some of the forward-looking scenarios? I think we all accept that some model error is a probable scenario. This subtle mechanism for conservatism is the one small opening allowed in IFRS9 for compensating model error.

Who should be responsible for backtesting these models?

Thinking of backtesting as a data-based statistical process, I see no reason why this should be arranged differently from current practice in Credit Risk or Collective Provisions models. Banks have settled on first line or
second line or even third line to carry out these tests and they usually have good reasons.

On the other hand, IFRS9 models draw strongly on assumptions and expert opinions. These also need “backtesting”, i.e. a challenge that looks how well these assumptions hold up under past conditions. For this kind of test, with such diverse expert sources, I do think a single team needs to coordinate the challenge, with sufficient skill and experience. That looks like an independent validation function, probably second line.

What would you like to achieve by attending the 5th Annual Credit Risk Modelling under IFRS 9 Conference?

I’m looking forward to the usual mixture of good talks, good conversations and networking, as well as a general appreciation of where Banks are meeting IFRS9 challenges and how they’re overcoming them. I’d consider it a success for me, if the delegates come away knowing more of the new challenges IFRS9 puts on the validators, and of the special model risks of the IFRS9 approach.