Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GOES-R GLM L2 Gridded product reader and small ABI L1b changes #854

Merged
merged 18 commits into from
Dec 11, 2019

Conversation

deeplycloudy
Copy link
Contributor

@deeplycloudy deeplycloudy commented Jul 13, 2019

Add GOES Geostationary Lightning Mapper support to SatPy. The first type supported is the gridded GLM imagery. This is not the lower-level, L2 point data produced by GOES ground system, but is instead the files produced by glmtools.

  • Finish adding all GLM imagery types to the yaml.
  • Tests added and test suite added to parent suite
  • Tests passed
  • Passes flake8 satpy
  • Fully documented
  • Add your name to AUTHORS.md if not there already

Update: This pull request also includes small changes to the ABI L1b reader. Mainly it drops the legacy satellite_altitude/longitude/latitude attributes from being added to loaded data. Other code was refactored to be shared between the GLM, ABI L1b, and ABI L2 readers.

@djhoese
Copy link
Member

djhoese commented Jul 14, 2019

Is this based on the L2 data? Would that then technically make this L3 data? I guess not. Wondering if a more specific reader name is needed like glm_l2_imagery or something. @deeplycloudy @mraspaud thoughts? Main concern is if we add a reader for the point data then it can"t also be called glm_l2, but is probably a more "accurate" name for that reader versus this reader.

@deeplycloudy
Copy link
Contributor Author

deeplycloudy commented Jul 15, 2019 via email

@coveralls
Copy link

coveralls commented Aug 24, 2019

Coverage Status

Coverage increased (+0.05%) to 87.364% when pulling 2bdc33f on deeplycloudy:glm-imagery into f03fd62 on pytroll:master.

Copy link
Member

@djhoese djhoese left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I"ve been working on some ABI L2 fixes/updates now that I merged the PR for the initial version (updates are in #905). I"m wondering if some/most of this can be based on the ABI base reader?

satpy/etc/readers/glm_l2.yaml Outdated Show resolved Hide resolved
@deeplycloudy
Copy link
Contributor Author

I"m certainly supportive of reusing the ABI infrastructure since my goal has been to have the GLM imagery be just another ABI channel. If you want to point to the essential bits from which I could inherit, I have no objections to making that change.

I wrote my reader from scratch because the ABI reader has a lot of machinery for tasks specific to the radiance data (and surely other matters that I don"t understand), whereas the GLM data can use the default xarray reader (since it comes from the xarray writer in the first place). So it was easier to do that as a first effort than to try to understand everything the ABI reader was doing.

@djhoese
Copy link
Member

djhoese commented Sep 20, 2019

Completely understood and I agree, but @yufeizhu600 did some work to get the ABI L2 reader working which included separating out the "shareable" functionality to one abi_base.py module. I"d still wait on #905 before starting the work on anything. It may be simple enough that I can just do it and have you review it, but I"m not sure my time availability coming up.

That said, I"m trying to get some last minute proof-of-concept things done for my talk at the Joint Satellite Conference in Boston in a week and a GLM/ABI composite from Satpy would be pretty sweet. But I also need to actually have slides to show so I should probably work on that first.

Edit: The abi_base.py work was done recently I should have said. So it only recently made easier to do this.

@codecov
Copy link

codecov bot commented Sep 27, 2019

Codecov Report

Merging #854 into master will increase coverage by 0.04%.
The diff coverage is 92.81%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #854      +/-   ##
==========================================
+ Coverage   87.31%   87.36%   +0.04%     
==========================================
  Files         181      183       +2     
  Lines       28042    28158     +116     
==========================================
+ Hits        24486    24601     +115     
- Misses       3556     3557       +1
Impacted Files Coverage Δ
satpy/readers/abi_l2_nc.py 69.23% <ø> (+9.65%) ⬆️
satpy/readers/abi_l1b.py 98.3% <ø> (ø) ⬆️
satpy/tests/test_readers.py 98.01% <100%> (+0.04%) ⬆️
satpy/readers/__init__.py 94.56% <100%> (+0.28%) ⬆️
satpy/tests/reader_tests/__init__.py 98.36% <100%> (+0.02%) ⬆️
satpy/tests/reader_tests/test_abi_l1b.py 97.22% <100%> (+0.85%) ⬆️
satpy/tests/reader_tests/test_abi_l2_nc.py 93.84% <100%> (-1.1%) ⬇️
satpy/readers/abi_base.py 88.54% <66.66%> (-2.36%) ⬇️
satpy/readers/glm_l2.py 93.18% <93.18%> (ø)
satpy/tests/reader_tests/test_glm_l2.py 94.87% <94.87%> (ø)
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f03fd62...2bdc33f. Read the comment docs.

@djhoese
Copy link
Member

djhoese commented Sep 27, 2019

@deeplycloudy I made some important changes that look big but really aren"t big to how the data is loaded.

  1. I changed the file handler to be based on the ABI L1b/L2 base file handler. This let"s the GLM file handler benefit from some work I"ve done on getting better more consistent precision with the GOES AreaDefinition.
  2. I kept the scene_abbr in the file name pattern to be consistent with ABI L1b and changed the area_code in the ABI L2 (FYI @yufeizhu600). This way all three readers use the same names for these fields.
  3. Fixed a couple trailing spaces that our linters didn"t like and added docstrings.
  4. I added extra metadata to the reader YAML based on some work I did relatively recently to make readers more usable for GUI applications and other tools.
  5. I added an available_datasets method to dynamically add resolution information to the datasets when the files are loaded. This method also checks if the variable actually exists in the file. I did this after noticing that some of the L2 files on Unidata"s THREDDS server that you pointed me to didn"t have any data variables (I assume this is because there was no lightning data for the time I picked).

@djhoese djhoese mentioned this pull request Oct 14, 2019
6 tasks
@djhoese djhoese changed the title Basic, working GOES GLM reader based on ABI Add GOES-R GLM L2 Gridded product reader Oct 14, 2019
@mraspaud mraspaud added component:readers enhancement code enhancements, features, improvements labels Oct 24, 2019
@djhoese djhoese marked this pull request as ready for review November 7, 2019 20:57
@djhoese djhoese requested a review from mraspaud as a code owner November 7, 2019 20:57
# Conflicts:
#	satpy/readers/abi_l1b.py
#	satpy/tests/reader_tests/__init__.py
#	satpy/tests/reader_tests/test_abi_l1b.py
@djhoese djhoese changed the title Add GOES-R GLM L2 Gridded product reader Add GOES-R GLM L2 Gridded product reader and small ABI changes Nov 7, 2019
@djhoese djhoese changed the title Add GOES-R GLM L2 Gridded product reader and small ABI changes Add GOES-R GLM L2 Gridded product reader and small ABI L1b changes Nov 7, 2019
@mraspaud mraspaud added this to the v0.19.0 milestone Nov 8, 2019
@djhoese
Copy link
Member

djhoese commented Dec 11, 2019

@mraspaud This is ready for re-review.

Copy link
Member

@mraspaud mraspaud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mraspaud mraspaud merged commit 7fb6ecf into pytroll:master Dec 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component:readers enhancement code enhancements, features, improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants