This is quite a broad question, so will try and answer to the best of context.
Basically, considerations of input data is the following:
Sufficiency:
-Do the data meet the requirements of the model specification?
-If the model will be used repeatedly, are the data in a consistent format every time?
Reliability:
-Reconciliation to other sources (preferably audited): e.g., does an asset file reconcile to the balance sheet, or do benefit/premium totals reconcile to other company records?
-Summarize and compare input data to prior periods, if applicable.
-Check and investigate outliers (e.g., age 115, zero benefit, zero premium).
-How are missing data handled—through assumptions or errors flagged?
-Review data assumptions periodically to ensure appropriateness.
-Confirm the size of the data file is consistent with prior periods
It is almost a common sense/reasonability check. If you are building/testing a model, but lets say, it is considerably different period to period. Is that good (no it is not)? If you are building/testing a model, but you only have 5 data points, does that seem okay (no)? If you are building/testing a model to see frequency, but half of the data is missing claim counts, is it still appropriate?
how about these 3 questions?
1. what considerations when updating a model's risk rating?
2. how to assess a model? (can I use the considerations in assessing the severity/likelihood of model failure?)
3. how to validate a model? are they asking how to validate when using a model?
Find answers below! Just wanted to ask, where are you pulling the screenshots/comments from? Thanks!
1. Risk-rating a model is to “assess how risky a model is so that the amount of work done to choose, validate, and document a model may be appropriate to the circumstances”. A model is assessed separately for severity and likelihood of failure, and the risk-rating is determined by balancing the two aspects. So to update for a model's risk rating, it would be the same - you would review these same factors to see if severity or likelihood has changed.
2. Yup, that's exactly it! There is the uni-dimensional and multi-dimensional approach and from there there is the severity/likelihood framework to assess further (how likely issues/failure is, how material impact is, etc)
3. Yup, it's asking how do you validate to see if a model is "good". I think the wiki/battlecards covers it pretty well (also need to consider if a new or existing model) but things like: Validation of data inputs, validation of assumptions, validation of results, etc
These validation steps also tie in with risk rating !
No concrete plan for this possible update but something we are still looking into - just a note, from the comments that CAS releases it is difficult to just have an answer/comment on their overall comment, thus taking a bit of time. Thanks!
Hello @exam6addoil , sorry for the late response, we were unable to access the forum for the past 2 days so was unavailable to respond.
As for your ask
1. What are considerations of determining a model risk rating
A - It is quite similar to the above question/answer that I provided to your earlier question. Risk-rating a model is to “assess how risky a model is so that the amount of work done to choose, validate, and document a model may be appropriate to the circumstances”. A model is assessed separately for severity and likelihood of failure, and the risk-rating is determined by balancing the two aspects.
What are considerations for when to update a model risk rating.
A - From the source, these are your considerations for when to update the risk rating
Reassess if a model fails;
Reassess on a regular cycle, e.g., every five years;
Reassess when model use changes; and
Reassess if the impact of results change greater than some tolerance level set
in advance
Just some extra details for you for the "when using a model outside expertise"
The extent to which the model has been reviewed by experts in the applicable
field; and
The risk rating associated with the model.
The actuary would make a reasonable attempt to understand the following:
• The basic workings of the model including its inputs, outputs, and general
approach
• The testing and validation work that was completed
• The model’s complexity and the control framework used
Comments
This is quite a broad question, so will try and answer to the best of context.
Basically, considerations of input data is the following:
Sufficiency:
-Do the data meet the requirements of the model specification?
-If the model will be used repeatedly, are the data in a consistent format every time?
Reliability:
-Reconciliation to other sources (preferably audited): e.g., does an asset file reconcile to the balance sheet, or do benefit/premium totals reconcile to other company records?
-Summarize and compare input data to prior periods, if applicable.
-Check and investigate outliers (e.g., age 115, zero benefit, zero premium).
-How are missing data handled—through assumptions or errors flagged?
-Review data assumptions periodically to ensure appropriateness.
-Confirm the size of the data file is consistent with prior periods
It is almost a common sense/reasonability check. If you are building/testing a model, but lets say, it is considerably different period to period. Is that good (no it is not)? If you are building/testing a model, but you only have 5 data points, does that seem okay (no)? If you are building/testing a model to see frequency, but half of the data is missing claim counts, is it still appropriate?
Stuff like this is considerations of input data.
Hope this helps, thanks!
how about these 3 questions?
1. what considerations when updating a model's risk rating?
2. how to assess a model? (can I use the considerations in assessing the severity/likelihood of model failure?)
3. how to validate a model? are they asking how to validate when using a model?
Find answers below! Just wanted to ask, where are you pulling the screenshots/comments from? Thanks!
1. Risk-rating a model is to “assess how risky a model is so that the amount of work done to choose, validate, and document a model may be appropriate to the circumstances”. A model is assessed separately for severity and likelihood of failure, and the risk-rating is determined by balancing the two aspects. So to update for a model's risk rating, it would be the same - you would review these same factors to see if severity or likelihood has changed.
2. Yup, that's exactly it! There is the uni-dimensional and multi-dimensional approach and from there there is the severity/likelihood framework to assess further (how likely issues/failure is, how material impact is, etc)
3. Yup, it's asking how do you validate to see if a model is "good". I think the wiki/battlecards covers it pretty well (also need to consider if a new or existing model) but things like: Validation of data inputs, validation of assumptions, validation of results, etc
it is from the exam summaries, is it possible to have comment/answer for each summary from BA?
https://www.casact.org/exams-admissions/exam-results/post-exam-summaries
Ah got it. Good point on that, we will see what we can do and let you know - Thanks for bringing this up!
any updates?
Hey,
No concrete plan for this possible update but something we are still looking into - just a note, from the comments that CAS releases it is difficult to just have an answer/comment on their overall comment, thus taking a bit of time. Thanks!
what are the answers for these 2 questions?
did BA stop answering question during exam period?
Hello @exam6addoil , sorry for the late response, we were unable to access the forum for the past 2 days so was unavailable to respond.
As for your ask
1. What are considerations of determining a model risk rating
A - It is quite similar to the above question/answer that I provided to your earlier question. Risk-rating a model is to “assess how risky a model is so that the amount of work done to choose, validate, and document a model may be appropriate to the circumstances”. A model is assessed separately for severity and likelihood of failure, and the risk-rating is determined by balancing the two aspects.
A - From the source, these are your considerations for when to update the risk rating
Reassess if a model fails;
Reassess on a regular cycle, e.g., every five years;
Reassess when model use changes; and
Reassess if the impact of results change greater than some tolerance level set
in advance
are my answers correct?
considerations when using a model outside expertise:
considerations when during sensitivity testing:
Yes, I think you have the right points.
Just some extra details for you for the "when using a model outside expertise"
The extent to which the model has been reviewed by experts in the applicable
field; and
The risk rating associated with the model.
The actuary would make a reasonable attempt to understand the following:
• The basic workings of the model including its inputs, outputs, and general
approach
• The testing and validation work that was completed
• The model’s complexity and the control framework used