Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

S3 Copy/Move performance optimizations #64

Open
araddon opened this issue Jul 11, 2018 · 1 comment
Open

S3 Copy/Move performance optimizations #64

araddon opened this issue Jul 11, 2018 · 1 comment

Comments

@araddon
Copy link
Contributor

araddon commented Jul 11, 2018 •
edited
Loading

There are Copy/Move that are not currently implemented on S3. These are performance optimizations that when copy/move from s3 -> to s3 don't do network transfer, instead allow s3 to do the copy/move through native s3 api.

cloudstorage/store.go

Lines 73 to 85 in 2731960

// StoreCopy Optional interface to fast path copy. Many of the cloud providers
// don't actually copy bytes. Rather they allow a "pointer" that is a fast copy.
StoreCopy interface {
// Copy from object, to object
Copy(ctx context.Context, src, dst Object) error
}
// StoreMove Optional interface to fast path move. Many of the cloud providers
// don't actually copy bytes.
StoreMove interface {
// Move from object location, to object location.
Move(ctx context.Context, src, dst Object) error
}

Google implementation

cloudstorage/google/store.go

Line 174 in 2731960

// Copy from src to destination

@araddon araddon mentioned this issue Jul 11, 2018
Closed
@araddon
Copy link
Contributor Author

araddon commented Jul 15, 2018

Not entirely sure this can be pushed down, the copy partially can:
https://stackoverflow.com/questions/47334556/aws-sdk-go-s3-move-directory-between-bucket
https://stackoverflow.com/questions/28830839/aws-s3-move-an-object-to-a-different-folder

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@araddon