Most important features at time of prediction

Hello,

After training a model, is it possible to determine which features dictate the prediction made for a new data point that was not in the training set? I am aware of feature importance and how to extract that information from the model, but that is only for training the model. I would like to know if it’s possible to determine the most important features at prediction time.

Thanks!