-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Zero restart migration #33
Comments
@DmitryDrobotov would you have time to start with this? Thanks! |
@ribose-jeffreylau @ronaldtse I've spent yesterday in my thoughts how to implement so magic behavior and here is some point I realized.
|
Thus, any running server instances would use the latest migration version (version A background rake task could then run migration version
class TranscryptUserSSN < Transcryption
def old
{ algo, salt, etc. }
end
def new
{ algo, salt, etc. }
end
end and no more Hence, the configuration is implicitly the "head" of a chain of migration files. The "head" can be different for each row, indicated by a separate column (for each encrypted column) via the transcryption order ID (e.g. |
I was actually thinking of a less drastic solution like this, so that the latest configuration is still located in the model, but the old one that we want to get rid of (in time) will be stored in the data migration, and still available if any server instances are still running on old code.
class TranscryptUserSSN < Transcryption
def old
{ algo, salt, etc. }
end
def new
UserModel.new_ssn_configuration
end
UserModel.register_old_configuration(:ssn, &old)
end |
@ribose-jeffreylau @ronaldtse thanks guys! Now I see the picture how to implement that. We will need:
Sounds great to me! What about you? |
@DmitryDrobotov this is actually a great solution! This is actually something more generic than just for encrypted attributes (which may or may not be in a separate gem). This suggests that we have a versioned schema per column cell (i.e., each column cell will have a schema version). For each schema-ed attribute column we will have an extra column to store the attribute schema version. The flow works like this:
What do you think? |
@ronaldtse Sounds good, that is the same what I meant except generic way like |
My concern was purely about scope -- if the test suite should test only against generic attribute schema it probably should be in/from a separate library. Do you think these gems are useful for this functionality? |
Hi @DmitryDrobotov . What else do you think is needed before we can continue? |
@ronaldtse I don't think that these gems can be useful. Because we need zero-downtime migrations of dynamicaly calculated value. For example, if row with id=1 has already been migrated it should use new configuration of |
Hey there! The only difference between zero downtime approach suggested in master branch and this one is that we have those few extra lines of code in the migrated model, which we want to get rid of now? Do I understand it right? And what is the difference between this issue and that one: #24 ? |
The major difference with #24 :
|
Hi @nattfodd ! The mechanism detailed in this issue focuses on data migration. It tries not to touch #24 concerns the bootstrapping of non-encrypted columns into encrypted columns, and vice versa, which will most probably change the DB schema. Hope it makes things clearer :) |
@ribose-jeffreylau sounds clearer, indeed. But I believe it should be extra temporary column anyway, as @DmitryDrobotov suggested right above. It may be done in a migration script though, something like:
It will affect actual SQL DB schema, but it wont affect |
@nattfodd actually the solution discussed with @DmitryDrobotov is not a temporary column, but a permanent column. The problem with a temporary column is that the database schema will be in flux -- for example, if multiple servers are reading values but suddenly the column is dropped (especially in many databases a column drop is implemented as a table adjust), it will cause havoc. A straightforward solution is to have a separate, permanent column to manage "data cell schema", the data of which indicates the "data schema version" of the original cell. This is similar to putting the Rails schema version number in a more granular way, instead of a single version for the entire database, we have versions per value. The benefits of a permanent column (or columns) are great:
The cost of using this is the implementation of a data layer that can switch (read/write) between data schema versions:
|
Goal
Implies
attr_encrypted
columns.
when set-up code is deployed.
Model definitions.
Perhaps in a config file, or even another DB table.
Potential tools
Instead of running rake tasks to run transcryptions,
rails-data-migrations
seems like a good alternative. It stores its migration history in the table
data_migrations
.DB
Currently,
db/schema.rb
db/migration/xxx_.rb
app/models/xxx.rb
Column
Some ideas on encryption schema:
db/encryption_schema.yaml
migration:
db/transcryption/2017125_transcrypt_user_ssn.rb
config:
Transcryptor.schema(:user, :ssn)
app/models/user.rb
How to make this work? What else needs to be defined? What's the closest we can acheive?
cc: @DmitryDrobotov
The text was updated successfully, but these errors were encountered: