All Rights Reserved. marcus evans ® 2021

18th Edition Model Risk

Strengthening model risk management by advancing automation strategies and optimizing model validation whilst remaining compliant with regulations

June 28-29, 2021
A Virtual Event

What our delegates think of us:

An Interview with Arthur Maghakian, Managing Director, Model Risk Management, at Goldman Sachs

Arthur Maghakian is a managing director in the Model Risk Management Group of Goldman Sachs, which is responsible for the independent validation of models used across the firm’s business activities. Arthur has over 20 years of experience in the financial industry, focusing on market, credit, liquidity and model risks throughout his career. He joined Goldman Sachs in 1997 as an analyst and rejoined in 2016 as a managing director.

What are the top priorities within model risk at the moment?

I would probably start with ML models as one of top priorities, which is not surprising, considering the exponential growth of ML inventory in the financial industry and the laser-sharp focus of regulators on ML. Regulators are especially concerned about the social and ethical component of ML as well as the explainability of ML models. I saw a chart published by the BIS Basel committee showing how frequently ML is mentioned in the speeches of the senior regulators. Not surprisingly this chart is exponential as well. 

Moving beyond ML models, and observing the entire universe of financial models, we can think about it as two subsets: mostly data-driven or mostly assumption-driven models. ML models obviously belong to the data-driven or empirical models, and questions of data relevance, quality, biases, dynamically updated data, etc. are important not only for ML models but for all empirical models - and this is definitely a priority for model risk. 

Finally, for assumption-driven models, we know that from time to time some of these assumptions fail, sometimes in a spectacular way. Such cases are frequently called black swans because these events never happened before and were considered impossible until they actually happened. We had a few black swans in 2020, e.g. negative oil prices and negative dividend yields. Obviously it’s very hard to predict a black swan and ensure that a model covers it. But both modelers and model risk managers can work together to ensure that infrastructure behind models is flexible enough, so when such an event occurs, models can be changed quickly to address the issue.

What are the biggest challenges with machine learning models and incorporating these in the model inventory?

I already mentioned data biases and model explainability. While the issue of data biases is very broad, regulators are especially concerned about data biases which might reflect social biases, e.g. coming from historical data. For some ML models, lack of model explainability or the “black box problem” is another major challenge. Similar to the data issue, the black box problem carries risk on its own, but it might be magnified even more if overlapped with social dimension. Imagine, for example, an application which makes drastic decisions in consumer finance without any explanations. 

In terms of ML model inventory, there are two edge cases that should  be considered separately.  One is the border line between ML and non-ML models. The spectrum of empirical models is continuous – it starts with simple linear regression and ends with deep learning. Where the boundary is drawn between ML and non-ML is somewhat subjective. More importantly, both ML and non-ML models are under the umbrella of MRM framework and border-line cases are relatively simple and explainable. The second edge case is more interesting. These are the ML applications which do not meet the standard model definition. We will see how the regulators will address this gap, but it is clear that “cliff effect” should not exist, i.e. some form of control should exist for non-model MLs.

What areas of technological investment are you focusing on? Where, for example, are you planning to invest within next 3 to 6 months?

Model validation is a slow and labor-intensive process which will always require humans, especially for assessing models conceptually during the initial validation. However many steps in this process can be automated. For example, refreshing the results during re-validation, continuous ongoing monitoring, generating parts of documentation, model change controls, etc. can be either fully or partially automated. Moreover, building a platform that would integrate model risk controls at the level of core architecture sounds like a really interesting idea. Integration of model lifecycle within such platform would help the modelers to incorporate best model risk management practices into their work and would allow them to work in parallel with model validators. Potentially, such a platform could drastically reduce the time required for documenting and validating models. This would be particularly important for ML models, where we expect that the number of models and the speed of their development will continue to increase, while many developers will be less technical.

In terms of our platform, (our conferences are informal and intimate peer led meetings where all speakers and delegates are senior executives from top financial institutions), how do you see it assisting you with overcoming the challenges you currently face? 

Dialog with peers is extremely important for model risk managers. It allows us to share the ideas, understand better the regulatory requirements, observe different approaches and solutions to the same problems. From that point of view, your conference provides an ideal forum that I always enjoy attending.

Assessing the best approaches to model inventory within ML models

• Understanding the definition of models and enhancing this within ML frameworks
• Determining models from non-models and when to classify ML techniques as models
• Knowing where to draw the line as to what can be included within inventory and validation processes dependent upon the level of model risk
  
  

For registration pricing and multiple attendee discounts, please contact:

Jeremy Wise
jeremywi@marcusevansch.com

Arthur will be Presenting on Day Two of the Virtual Event!