Implement the AIASynopticClient #7842

GillySpace27 · 2024-10-18T20:35:11Z

PR Description

This PR addresses issue #7828 , implementing the new Client as requested. In its current state the client returns found files, but it does not handle a.Wavelenth or a.Sample yet, which I'd like to ask for assistance in implementing.

GillySpace27 · 2024-10-18T20:47:37Z

I need help troubleshooting Wavelength and Sample attrs which aren't doing anything right now.

Example usage

time_range = a.Time("2023-10-11 00:00:00", "2023-10-11 01:00:00")
instrument = a.Instrument("AIASynoptic")
wavelength = a.Wavelength(193 * u.angstrom)
results = Fido.search(time_range, instrument, wavelength)
print(results)

Matchdict

The content of the matchdict from the pre_search_hook has this:
"""

special variables
function variables
'Instrument' =['aiasynoptic']
'Physobs' =['intensity']
'Source' =['SDO']
'Provider' =['JSOC']
'Level' =['1.5']
'Resolution' =['Synoptic']
'Start Time'=Time object: scale='utc' format='isot' value=2023-10-11T00:00:00.000
'End Time'=Time object: scale='utc' format='isot' value=2023-10-11T01:00:00.000
'Wavelength' =<sunpy.net.attrs.Wavelength(193.0, 193.0, 'Angstrom')>
"""

Console Printout:

Results from 1 Provider:

279 Results from the AIASynopticClient:
Source: https://jsoc1.stanford.edu/data/aia/synoptic/

   Start Time               End Time         Instrument  Physobs  Source Provider Level Resolution Wavelength

2023-10-11 00:00:00.000 2023-10-11 00:00:59.999 AIASYNOPTIC intensity SDO JSOC 1.5 SYNOPTIC 94
2023-10-11 00:00:00.000 2023-10-11 00:00:59.999 AIASYNOPTIC intensity SDO JSOC 1.5 SYNOPTIC 131
2023-10-11 00:00:00.000 2023-10-11 00:00:59.999 AIASYNOPTIC intensity SDO JSOC 1.5 SYNOPTIC 171
2023-10-11 00:00:00.000 2023-10-11 00:00:59.999 AIASYNOPTIC intensity SDO JSOC 1.5 SYNOPTIC 193
2023-10-11 00:00:00.000 2023-10-11 00:00:59.999 AIASYNOPTIC intensity SDO JSOC 1.5 SYNOPTIC 211
2023-10-11 00:00:00.000 2023-10-11 00:00:59.999 AIASYNOPTIC intensity SDO JSOC 1.5 SYNOPTIC 304
2023-10-11 00:00:00.000 2023-10-11 00:00:59.999 AIASYNOPTIC intensity SDO JSOC 1.5 SYNOPTIC 335
2023-10-11 00:00:00.000 2023-10-11 00:00:59.999 AIASYNOPTIC intensity SDO JSOC 1.5 SYNOPTIC 1600
2023-10-11 00:00:00.000 2023-10-11 00:00:59.999 AIASYNOPTIC intensity SDO JSOC 1.5 SYNOPTIC 4500
2023-10-11 00:02:00.000 2023-10-11 00:02:59.999 AIASYNOPTIC intensity SDO JSOC 1.5 SYNOPTIC 94

sunpy/net/dataretriever/sources/__init__.py

sunpy/net/dataretriever/sources/tests/example_aia_synoptic.ipynb

sunpy/net/dataretriever/sources/tests/synoptic_manual_test.py

nabobalis · 2024-10-18T22:48:01Z

sunpy/net/attr.py

@@ -448,7  448,8 @@ def __contains__(self, other):
        if isinstance(other, Range):
            return self.min <= other.min and self.max >= other.max
        else:
-            return self.min <= other <= self.max
+            # return self.min <= other <= self.max
+            return self.min.value <= int(other) <= self.max.value


What query should I run to trigger the bug here?

sunpy/net/dataretriever/sources/aia_synoptic.py

nabobalis · 2024-10-23T18:58:42Z

sunpy/net/dataretriever/sources/aia_synoptic.py

+    required = {a.Time, a.Instrument}
+    optional = {a.Sample, a.Level, a.Wavelength, a.ExtentType}


Suggested change

required = {a.Time, a.Instrument}

optional = {a.Sample, a.Level, a.Wavelength, a.ExtentType}

required = {a.Time, a.Instrument, a.ExtentType, a.Provider}

optional = {a.Sample, a.Wavelength}

Maybe this would work, it's a lot to add for a query but it should make sure we only get the results from this client.

nabobalis · 2024-10-23T18:58:50Z

sunpy/net/dataretriever/sources/aia_synoptic.py

+            a.Source: [("SDO", "The Solar Dynamics Observatory.")],
+            a.Wavelength: [(f"{wv:04d}", f"{wv} Å") for wv in cls.known_wavelengths],
+            a.Provider: [("JSOC", "Joint Science Operations Center at Stanford.")],
+            a.Level: [("synoptic", "Level 1.5 data processed for quicker analysis.")],


Suggested change

a.Level: [("synoptic", "Level 1.5 data processed for quicker analysis.")],

a.Level: [("1.5", "Level 1.5 data processed for quicker analysis.")],

I guess I was hoping to maybe just use the Level attribute. It seems appropriate: it labels a homogenious data set from JSOC with standardized processing steps and observational parameters. By referencing the synoptic level of AIA data you are specifying this in a way users probably already know to do. But this is just my preferred implementation and if you'd like it more standardized I understand.

Hmm, fair. I will have to let the others chime in on this matter before we make a change here.

nabobalis · 2024-10-23T19:12:51Z

sunpy/net/dataretriever/sources/aia_synoptic.py

+        required_attrs = {a.Instrument, a.Level}
+        supported_attrs = {a.Time, a.Instrument, a.Sample, a.Level, a.Wavelength, a.ExtentType}


Should be able to be this from the client itself.

Actually does the main class's version of this not handle this already?

I love the idea that only those two could be required but all the others are accepted too

I'm not sure what you mean about the main class, are you just saying that these values match the defaults?

Sorry I wasn't clear.

This client inherits from a base class which should have a def _can_handle_query(self, *query): already which means that I think this isn't needed for this client. But it would need to be checked out to confirm or deny it.

I just went through and tried troubleshooting the inherited method and I'm missing something because the only way the client returns results is if I use my class' version of the function. It's probably just a syntax misunderstanding since the two versions are very similar.

INteresting, I have suspected that there is a bug somewhere in that function for a while now. I think this confirms it. For now, we can leave this here and worry about that in the future.

sunpy/net/dataretriever/sources/tests/example_aia_synoptic.ipynb

sunpy/net/dataretriever/sources/tests/synoptic_manual_test.py

sunpy/net/dataretriever/sources/aia_synoptic.py

sunpy/net/dataretriever/sources/tests/test_aia_synoptic.py

Co-authored-by: Nabil Freij <[email protected]>

GillySpace27 · 2024-11-28T19:35:29Z

I'm sorry I haven't tied this up yet, though it turns out that major flooding at the JSOC datacenter makes this less important for the time being. That said, what are the steps I need to take to keep/get the ball rolling on this PR?

Thanks again for your unending patience!

nabobalis · 2025-01-02T02:15:02Z

I'm sorry I haven't tied this up yet, though it turns out that major flooding at the JSOC datacenter makes this less important for the time being. That said, what are the steps I need to take to keep/get the ball rolling on this PR?

Thanks again for your unending patience!

Sorry I somehow missed this message.

Unfortunately I think until the JSOC server is back, there isn't much we can do directly.
Having a quick glance over the PR, I see no outstanding issues, so hopefully when the server is back, we can handle the last remaining bits.

nabobalis · 2025-01-08T20:25:56Z

Seems like the server is back with some data. So we can go back to finishing this PR up.

GillySpace27 added 8 commits October 18, 2024 13:21

hotfix the quantity comparison for wavelengths

ef8ed3f

update the __init__ files in dataretriever and sources

4fddb3c

init AIASynopticClient in Sources

43088b4

add standard test file

2504767

temporarily include bespoke test files

9095859

tweak examples

d74cb27

clean up synoptic client

578b0f0

move import to top of file

2548761

nabobalis mentioned this pull request Oct 18, 2024

Add a client to Fido which can discover the synoptic AIA series #7841

Closed

nabobalis reviewed Oct 18, 2024

View reviewed changes

sunpy/net/dataretriever/sources/__init__.py Outdated Show resolved Hide resolved

nabobalis reviewed Oct 18, 2024

View reviewed changes

sunpy/net/dataretriever/sources/tests/example_aia_synoptic.ipynb Outdated Show resolved Hide resolved

nabobalis reviewed Oct 18, 2024

View reviewed changes

sunpy/net/dataretriever/sources/tests/synoptic_manual_test.py Outdated Show resolved Hide resolved

nabobalis reviewed Oct 18, 2024

View reviewed changes

sunpy/net/dataretriever/sources/aia_synoptic.py Outdated Show resolved Hide resolved

GillySpace27 added 6 commits October 20, 2024 19:53

iteration savepoint

c801575

fix wavelength selection

3398c47

first fix of sample

54ba715

simplify the manual test case

247517a

extensively refactor the main client

8441345

clean notebook example

2bf7ade

nabobalis reviewed Oct 23, 2024

View reviewed changes

ayshih changed the title ~~Implement the AIASynoptiClient, second try due to autoformatter~~ Implement the AIASynopticClient, second try due to autoformatter Oct 23, 2024

nabobalis reviewed Oct 23, 2024

View reviewed changes

sunpy/net/dataretriever/sources/tests/example_aia_synoptic.ipynb Outdated Show resolved Hide resolved

nabobalis reviewed Oct 23, 2024

View reviewed changes

sunpy/net/dataretriever/sources/tests/synoptic_manual_test.py Outdated Show resolved Hide resolved

GillySpace27 added 3 commits October 23, 2024 13:42

Delete manual testing scripts

20c5216

Dump sources/__init__.py content

740e5d9

update required vs supported attrs

5c8c925

nabobalis changed the title ~~Implement the AIASynopticClient, second try due to autoformatter~~ Implement the AIASynopticClient Oct 23, 2024

nabobalis reviewed Oct 23, 2024

View reviewed changes

sunpy/net/dataretriever/sources/aia_synoptic.py Show resolved Hide resolved

nabobalis reviewed Oct 23, 2024

View reviewed changes

sunpy/net/dataretriever/sources/aia_synoptic.py Show resolved Hide resolved

nabobalis reviewed Oct 23, 2024

View reviewed changes

sunpy/net/dataretriever/sources/tests/test_aia_synoptic.py Outdated Show resolved Hide resolved

nabobalis reviewed Oct 23, 2024

View reviewed changes

sunpy/net/dataretriever/sources/tests/test_aia_synoptic.py Outdated Show resolved Hide resolved

GillySpace27 and others added 3 commits October 23, 2024 19:14

Update sunpy/net/dataretriever/sources/tests/test_aia_synoptic.py

dae0c15

Co-authored-by: Nabil Freij <[email protected]>

Update sunpy/net/dataretriever/sources/tests/test_aia_synoptic.py

a83a5a3

Co-authored-by: Nabil Freij <[email protected]>

Merge branch 'main' into AIASynopticSecondTry

06f0873

nabobalis added net Affects the net submodule Whats New? Needs a section added to the current Whats New? page. labels Jan 2, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement the AIASynopticClient #7842

Implement the AIASynopticClient #7842

GillySpace27 commented Oct 18, 2024

GillySpace27 commented Oct 18, 2024

nabobalis Oct 18, 2024

nabobalis Oct 23, 2024

nabobalis Oct 23, 2024

GillySpace27 Oct 23, 2024

nabobalis Oct 23, 2024

nabobalis Oct 23, 2024

nabobalis Oct 23, 2024

GillySpace27 Oct 23, 2024

nabobalis Oct 23, 2024 •

edited

Loading

GillySpace27 Oct 24, 2024

nabobalis Oct 24, 2024

GillySpace27 commented Nov 28, 2024

nabobalis commented Jan 2, 2025

nabobalis commented Jan 8, 2025

		required = {a.Time, a.Instrument}
		optional = {a.Sample, a.Level, a.Wavelength, a.ExtentType}

	a.Level: [("synoptic", "Level 1.5 data processed for quicker analysis.")],
	a.Level: [("1.5", "Level 1.5 data processed for quicker analysis.")],

		required_attrs = {a.Instrument, a.Level}
		supported_attrs = {a.Time, a.Instrument, a.Sample, a.Level, a.Wavelength, a.ExtentType}

Implement the AIASynopticClient #7842

Are you sure you want to change the base?

Implement the AIASynopticClient #7842

Conversation

GillySpace27 commented Oct 18, 2024

PR Description

GillySpace27 commented Oct 18, 2024

Example usage

Matchdict

Console Printout:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nabobalis Oct 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GillySpace27 commented Nov 28, 2024

nabobalis commented Jan 2, 2025

nabobalis commented Jan 8, 2025

nabobalis Oct 23, 2024 •

edited

Loading