Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dynamic metadata block generation and DVUploader integration #16

Merged
merged 111 commits into from
Apr 4, 2024
Merged
Changes from 1 commit
Commits
Show all changes
111 commits
Select commit Hold shift click to select a range
b9ce205
Added connect method
JR-1991 Mar 6, 2023
6cb6da6
Added info method
JR-1991 Mar 6, 2023
0526202
Token and URL are now drawn from instance
JR-1991 Mar 6, 2023
9b086b3
New module
JR-1991 Mar 6, 2023
2358408
Added dep
JR-1991 Mar 6, 2023
ce6da05
Added missing dep
JR-1991 Mar 6, 2023
fc47900
Added versions
JR-1991 Mar 6, 2023
1786d6d
Added maxsize to match 3.7 implementation
JR-1991 Mar 6, 2023
038b9e4
Fixed attribute name passed wrong for add funs
JR-1991 Mar 8, 2023
df072cf
Added default for ad method
JR-1991 Mar 20, 2023
1ddc72e
Left LRU to default size
JR-1991 Apr 27, 2023
a92dc87
Fix test
JR-1991 Apr 27, 2023
5bd5aa7
Merge branch 'flexible-connect' of https://github.com/gdcc/easyDatave…
JR-1991 Apr 27, 2023
f56ce6d
Fix 3.7 lur_cache requirements
JR-1991 Apr 27, 2023
e00a410
Fix 3.7 lru_cache
JR-1991 Apr 27, 2023
faddb4b
Changed caching type
JR-1991 Apr 27, 2023
32b2f3b
Reverted and changed action to not use buggy 3.7
JR-1991 Apr 27, 2023
192381d
Fixed tests ...
JR-1991 Apr 27, 2023
02ee315
Moved and removed modules not used anymore
JR-1991 Sep 1, 2023
44ccb37
added dataverse interface
JR-1991 Sep 1, 2023
97c21c7
Updated readme and example
JR-1991 Sep 1, 2023
b665b44
added dep
JR-1991 Sep 1, 2023
0c6c41b
Update Readme.md
JR-1991 Sep 1, 2023
7150a28
Update Readme.md
JR-1991 Sep 1, 2023
bad4a06
bump version
JR-1991 Nov 27, 2023
60e4214
pop if field is given
JR-1991 Nov 27, 2023
d82b8b7
fix version check for >6
JR-1991 Nov 29, 2023
94f2421
fix `date` object not serialized correctly
JR-1991 Jan 2, 2024
79df3e0
migrate to `pydantic` 2.x
JR-1991 Jan 2, 2024
4ff2427
validate required fields upon upload/update
JR-1991 Jan 2, 2024
559da73
use latest `pydantic` version
JR-1991 Jan 2, 2024
94e3cc7
remove redundant tests
JR-1991 Jan 2, 2024
6a4680d
fix python version
JR-1991 Jan 2, 2024
426ce73
add `rich` dep
JR-1991 Jan 2, 2024
d1a5661
remove unused packages
JR-1991 Jan 2, 2024
fcc53ad
restructure and remove repo
JR-1991 Jan 2, 2024
3749e8b
bump version
JR-1991 Jan 2, 2024
6976f1a
fix imports
JR-1991 Jan 2, 2024
697558d
move tests
JR-1991 Jan 2, 2024
ba2a60f
adjust test imports
JR-1991 Jan 2, 2024
c21c9d5
Remove old test framework
JR-1991 Jan 3, 2024
b5e2782
add new unit and integration tests
JR-1991 Jan 3, 2024
39f7be9
patch `asyncio` for nested loops
JR-1991 Jan 3, 2024
eaf51d2
convert `Config` class to `ConfigDict`
JR-1991 Jan 3, 2024
c29958b
re-factor class generation to file
JR-1991 Jan 3, 2024
ee08500
re-factor connectivity to file
JR-1991 Jan 3, 2024
9104084
add async block fetching
JR-1991 Jan 3, 2024
7bd1a6f
remove unused exceptions
JR-1991 Jan 3, 2024
a7a28e2
update deps
JR-1991 Jan 3, 2024
3c88d92
remove unused workflows
JR-1991 Jan 3, 2024
b9ed6d7
split test workflows to `unit`and `integration`
JR-1991 Jan 3, 2024
eccef0f
drop `3.7` support
JR-1991 Jan 3, 2024
820eb99
use `List` instead of `list` for 3.8 support
JR-1991 Jan 3, 2024
b45a8b8
use current action versions
JR-1991 Jan 3, 2024
0df7e1e
remove test
JR-1991 Jan 3, 2024
a235472
add workflow badges
JR-1991 Jan 3, 2024
0e151b5
add python badge
JR-1991 Jan 3, 2024
325fcbc
remove CLI
JR-1991 Jan 16, 2024
356348c
integrate dvuploader
JR-1991 Jan 16, 2024
f3939d4
minor refactor
JR-1991 Jan 16, 2024
701b427
remove unused exceptions
JR-1991 Jan 16, 2024
b9d33df
add dvuploader dep
JR-1991 Jan 16, 2024
4dbb75a
remove link tag
JR-1991 Jan 16, 2024
2a1e04c
add missing dep
JR-1991 Jan 16, 2024
ab188c4
Convert URL object to string
JR-1991 Jan 29, 2024
c3dadbf
fix values not converted to strings
JR-1991 Jan 31, 2024
8654001
add `from_json` import to instance object
JR-1991 Jan 31, 2024
c624748
use handler instead of path
JR-1991 Jan 31, 2024
9eb50b5
populate by `typeName` upon deserialization
JR-1991 Jan 31, 2024
4ba9952
remove unused imports
JR-1991 Jan 31, 2024
69b3d24
remove unused whitespaces
JR-1991 Jan 31, 2024
8693c9d
use `str` as default JSON export fun
JR-1991 Feb 1, 2024
f512b57
adapt to alias
JR-1991 Feb 1, 2024
2bedbfd
cast url to string
JR-1991 Feb 1, 2024
9b2d5bd
remove unused imports and better printing
JR-1991 Feb 8, 2024
19f268d
ensure `str` is passed
JR-1991 Feb 8, 2024
63db52b
remove breaking typo
JR-1991 Feb 8, 2024
8924dca
move extra kwargs to `json_schema_extra`
JR-1991 Feb 8, 2024
17254e9
ensure arguments are passed as `str`
JR-1991 Feb 8, 2024
aed94e1
adapt dataset update to dvuploader
JR-1991 Feb 8, 2024
db789f1
add all metadata to `File` object
JR-1991 Feb 8, 2024
fe6d71d
add update and download tests
JR-1991 Feb 8, 2024
bf7e16e
remove slash to fix tests
JR-1991 Feb 8, 2024
dbed410
remove unused module
JR-1991 Feb 9, 2024
8fede31
re-factor file downloader
JR-1991 Feb 9, 2024
5926be8
remove unused arguments
JR-1991 Feb 9, 2024
bd27787
change types and add kwargs
JR-1991 Feb 9, 2024
488cb61
remove unused args
JR-1991 Feb 9, 2024
35a0c13
filter files if `filenames` are provided
JR-1991 Feb 9, 2024
1292477
extend `info` by add funs
JR-1991 Feb 12, 2024
0066519
allow detailed block list
JR-1991 Feb 12, 2024
9c4b185
add metadatablock listing
JR-1991 Feb 12, 2024
b63a9e6
add rich printing and async download
JR-1991 Feb 12, 2024
a778987
async file downloads
JR-1991 Feb 12, 2024
148f3cf
fix older py versions failing
JR-1991 Feb 12, 2024
87c1f7e
minor mods
JR-1991 Feb 12, 2024
4c7d337
update tests
JR-1991 Feb 12, 2024
1691dfd
allow parallel downloads
JR-1991 Feb 13, 2024
91a532f
remove unused import
JR-1991 Feb 13, 2024
81fdbb3
move downloader to separate file
JR-1991 Feb 13, 2024
7edaa3e
remove unused dep
JR-1991 Feb 13, 2024
d2d5352
allow regex patterns
JR-1991 Feb 13, 2024
e1f5b9a
additional kwargs for load from url
JR-1991 Feb 14, 2024
e00da3b
fix `add_directory` overriding previous files
JR-1991 Mar 6, 2024
92329b7
set version to `latest` if not available
JR-1991 Mar 11, 2024
dfff3fb
fix empty content added
JR-1991 Mar 18, 2024
233a5a0
typo
JR-1991 Mar 19, 2024
914f3d6
convert `Url` object to string
JR-1991 Mar 19, 2024
39132c6
map `str` by default
JR-1991 Mar 19, 2024
33b325e
add `categories` kwargs - fixes #15
JR-1991 Mar 20, 2024
cca73a5
Merge pull request #17 from gdcc/15-feature-request-add-file-tags-cat…
JR-1991 Apr 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Token and URL are now drawn from instance
  • Loading branch information
JR-1991 committed Mar 6, 2023
commit 05262025376a6c7758dcc68cde887a9cf4004b5e
26 changes: 13 additions & 13 deletions easyDataverse/tools/uploader/uploader.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 6,7 @@
import tqdm
import zipfile

from urllib.parse import urljoin
from typing import List, Optional

from pyDataverse.api import NativeApi, DataAccessApi
Expand Down Expand Up @@ -46,6 47,7 @@ def upload_to_dataverse(
"""

api, _ = _initialize_pydataverse(DATAVERSE_URL, API_TOKEN)

ds = Dataset()
ds.from_json(json_data)

Expand Down Expand Up @@ -78,22 80,15 @@ def upload_to_dataverse(
raise Exception("Could not upload")


def _initialize_pydataverse(DATAVERSE_URL: Optional[str], API_TOKEN: Optional[str]):
def _initialize_pydataverse(DATAVERSE_URL: str, API_TOKEN: str):
"""Sets up a pyDataverse API for upload."""

# Get environment variables
if DATAVERSE_URL is None:
try:
DATAVERSE_URL = os.environ["DATAVERSE_URL"]

except KeyError:
raise MissingURLException
raise MissingURLException

if API_TOKEN is None:
try:
API_TOKEN = os.environ["DATAVERSE_API_TOKEN"]
except KeyError:
raise MissingCredentialsException
raise MissingCredentialsException

return NativeApi(DATAVERSE_URL, API_TOKEN), DataAccessApi(DATAVERSE_URL, API_TOKEN)

Expand Down Expand Up @@ -194,7 189,11 @@ def update_dataset(
content_loc (Optional[str], optional): If specified, the ZIP that is used to upload will be stored at the destination provided. Defaults to None.
"""

url = f"{os.environ['DATAVERSE_URL']}/api/datasets/:persistentId/versions/:draft?persistentId={p_id}"
url = urljoin(
DATAVERSE_URL,
f"/api/datasets/:persistentId/versions/:draft?persistentId={p_id}",
)

response = requests.put(
url,
json=json_data,
Expand Down Expand Up @@ -229,7 228,8 @@ def update_dataset(
is_filepid=False,
)

# Upload files that havent been added yet
__uploadFiles(new_files, p_id, api, content_loc)
if new_files:
# Upload files that havent been added yet
__uploadFiles(new_files, p_id, api, content_loc)

return True