Most important features at time of prediction


After training a model, is it possible to determine which features dictate the prediction made for a new data point that was not in the training set? I am aware of feature importance and how to extract that information from the model, but that is only for training the model. I would like to know if it’s possible to determine the most important features at prediction time.