The engineering team has decided on a plan for this project. Some assumptions first:

• Passwords will never be exported. Users will have to re-confirm their email on the new platform.
• The first solution will take the shape of simple import/export scripts meant to be used by engineers only
• We want to use standard, human-readable, GDPR-compliant formats (JSON, JSONL)
• We won't assume users will be starting with a fresh instance (avoid collisions of IDs)
• In the future, we may want to be aware of other instances (i.e. need to identify a transaction in a network of instances)

Based on that, 3 approaches were discussed:

1. GraphQL approach
   • Format: defined by the queries
   • Pros:
     • Re-use permission logic to filter out data (e.g. a payment method)
     • Simple export implementation
     • Format is consistent with our public API
   • Cons:
     • Business cost is moved to import (need to map entities from GraphQL to DB)
     • GraphQL will need adaptations to work properly (e.g. surface data, hack limits)
     • Not everything is exposed today on GraphQL, and we may not want to expose everything (histories, legal docs, paypal plans, paypal products, ...etc). Yet the best guarantee that the platform will continue to work is to have everything.
     • Limited control over performances
     • Will eventually lead to enormous exports (lot of duplication)
     • The queries to get everything would actually be quite complex to write
2. Query the data directly
   • Format: JSONL, one file per table, 1 row per entry
   • Risk: Not implementing permissions correctly could lead to leaked data
   • Pros:
     • Simplified import
     • Guarantee that all data is there
     • We can build upon @kewitz's existing work, which already covers the base MVP for this project
3. Use pg_dump/pg_restore
   • Risks: Not filtering enough => exporting too much (security)
   • Pros:
     • Simple export & import
   • Cons:
     • Non-standard format
     • Auto-increment ID conflict issues

Based on engineering discussions, we have decided to build upon @kewitz's existing work which was already implemented based on the 2nd approach. However, we reckon that the comments made on security risks are valid concerns and special attention will be given to make sure we don't export more than we should. Whenever possible, we'll try to re-use and put in common the permission logic used by GraphQL.

This week

• Start looking at Collective Export/Import: Permissions layer #7487 (@Betree)
• Improve typings and add tests to the existing import/export scripts #7494 (@kewitz)
• Start looking at Collective Export/Import: Improve the export format (JSONL archive) #7489 (@hdiniz)

This week

• Collective Export/Import: Permissions layer #7487 (@Betree)
• Collective Export/Import: Improve the export format (JSONL archive) #7489 (@hdiniz)

Last week

• @hdiniz did zip file implementation

This week

• Add tests and put for review Collective Export/Import: Permissions layer #7487 (@Betree)
• Review Collective Export/Import: Improve the export format (JSONL archive) #7489 (@hdiniz)
• Start looking at Collective Export/Import: Implement a strategy for dealing with IDs & de-duplicating on import #7488 (@kewitz)
• Figure a recipe to use a a test with prod data

This week

• Start looking at Collective Export/Import: Implement a strategy for dealing with IDs & de-duplicating on import #7488 (@kewitz)
• Figure a recipe to use a test with prod data (@kewitz)
• Collective Export/Import: Paginate results in traverse function #7493 (@Betree)