Fixed download.py (Empty file downloads, encoding errors) #65
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixed the following errors - 1. Python defaulting to ASCII encoding instead of Unicode. Set global encoding to "utf-8" 2. Sript downloading 0kb empty FILE objects. Some files being generated contained ":", which is a restricted character in filenames on Windows and MacOS and was truncating filenames before the extension, added to the replacements list in clean_text. 3. Download failures. Used the requests library to improve success rates of downloads, which seemed to work a bit better.
|
To run the script in Python 3, one has to remove the line |
|
Does Python 3 automatically encode in Unicode then? I"ll make a few changes to make sure it works in Python 3, thanks |
|
Yes. BTW thanks for this wonderful repo! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixed the following errors -
Set global encoding to "utf-8"
Some filenames being generated contained ":", which is a restricted
character in filenames on Windows and MacOS and was truncating filenames
before the extension, added to the replacements list in clean_text.
Used the requests library to improve success rates of downloads, which
seemed to work a bit better.