Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

General Topic: Developing metadata objects to support transfer of large external files #3

Open
dponti opened this issue Jan 18, 2023 · 1 comment
Open

General Topic: Developing metadata objects to support transfer of large external files #3

dponti opened this issue Jan 18, 2023 · 1 comment
Labels
Data acquisition Topic specific to reporting of "raw" data acquired directiy in the field question Further information is requested

Comments

@dponti
Copy link
Collaborator

dponti commented Jan 18, 2023

It may not be efficient to transfer data acquisition results for some techniques as ASCII data within the xml, especially where data acquisition software stores its data in binary format. One of DIGGS" primary goal is to allow an end user to be able to have enough information about the measurement to assess the efficacy of the final result and/or be able to reprocess the raw data if need be.

Given that some approaches typically generate large data files during acquisition, how do we best allow users to access this data?

Are there standard formats we should reference or support?

Can Seg-Y be used for universal binary transfer as opposed to proprietary formats?
@dponti dponti added question Further information is requested Data acquisition Topic specific to reporting of "raw" data acquired directiy in the field labels Jan 18, 2023
@nickmachairas
Copy link
Member

AVRO and Parquet are popular file formats for large datasets. SEG-Y seems to be geared towards seismic data only, I don"t know much about it.

How large is a large data file that warrants looking into alternatives? A 500MB XML file would be a problem some time ago but nowadays most laptops can easily parse files of that size.

Are there issues other than compute and transfer with large files?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Data acquisition Topic specific to reporting of "raw" data acquired directiy in the field question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nickmachairas @dponti