PartitionShap: channel-first partitioning and channel SHAP plots. #1944
Conversation
|
Hi Evan, Paper on Deep Learning Interpretability in Time Series Predictions, which seems to perform what is aimed here, but for PyTorch: https://arxiv.org/pdf/2010.13924v1.pdf it also has code here: https://github.com/ayaabdelsalam91/TS-Interpretability-Benchmark A similar attempt but for Lime: https://github.com/emanuel-metzenthin/Lime-For-Time Also this post on a similar aim but for LSTM and Pytorch: https://andrecnf.github.io/2019/07/31/InterpretingRecurrentNeuralNetworksOnMultivariateTimeSeries.html |
|
Thanks for the links! Good weekend reading :) |
|
What would be required to get this merged? I find it very useful and am working to extend it to time series, which I think would be beneficial to the community. |
|
@ekrell this PR looks interesting. Would you like to raise it over at https://github.com/dsgibbons/shap so we can discuss it further? |
|
This PR is now used in a publication: https://www.cambridge.org/core/journals/environmental-data-science/article/aggregation-strategies-to-improve-xai-for-geoscience-models-that-use-correlated-highdimensional-rasters/F6017A23BEF0BD48969225D68DF819A2 It would be very nice to have it included in the SHAP repo, instead of having to rely on my lab's own fork of a very old version of SHAP. |
There are a few of us interested in using SHAP to explore the channel-wise contributions. For example, for multispectral satellite imagery or 3D CNNs where the channels are time, temperatures, etc.
To help with initial explorations, I added support for
shap/plots/_image.pyto display requested channels. For example, if I want to see all RGB bands I can send in [0, 1, 2]. By default, it will sum the channels as before.I also added a second partition scheme where the channels are split before the the rows and columns. I suspect it will take some time to figure out the best partition schemes and masking methods for these various problems, but hopefully this will be useful for exploring. Again, its an optional parameter that will not affect existing code.
Here is a quick demo: https://github.com/ekrell/shap_multiband/blob/main/MultiChannel_PartitionSHAP_Demo.ipynb