Thank you! Way back I was trying to write my own declarative db schema generator for go (it’d generate the sql as well as the go models), so I’ve been looking for something like this!
This is a minor complaint from a newbie to the cloud database environment. As I watched this video, I assumed that Atlas was the no SQL database from mongoDB. It took a while before it dawned on me that atlas is the name of a standalone database management tool. I did a Google search on atlas schema management, and found entries for the standalone tool, and for the schema management capabilities of the mongo, DB Atlas database. I think you can see why people not totally immersed in this field could be confused. Also, I get the impression that the atlas schema management tool works for SQL schemas, and not for no SQL schemas. Again, it would’ve been nice to simply establish the context for this discussion. On the other hand, I have noticed that many technical videos on RUclips tend to assume that their viewers are familiar with their context and do not take the trouble to provide an overview fot their discussion. The information from this RUclips channel is excellent. Keep up the good work!
Atlas is a terrible name for that project. It's hard to Google it, it's used in too many other projects, etc. Most of schema management tools are focused on SQL DBs, mostly because there is not such a big need to manage schemas on nosql DBs. As for the context... It's hard to balance it. If there's too much of it, some will go somewhere else when they know it. On the other hand, if there's not enough of it, others will go somewhere else because it's confusing. I myself am never sure whether I should do more or less introduction and background on any given topic :(
Well, basically, a declarative approach to database schema could be a step forward, but I think something very important is missing here, when we talk about Kubernetes nature of agile deployments: One of the basic ideas of cloud-native (or container-native) is the separation of logic and state, and the ability to upgrade versions with zero downtime. This zero-downtime service update is done by switching to the new version gradually ("rolling update", "canary deployment") or at once ("blue-green deployment"). Either way, Continuous Deployment and Agile concepts allow you to decide that this version upgrade was a mistake, and to roll back to the previous version, keeping the service running with minimal disruption to the clients. Kubernetes made it very easy to roll a STATELESS application version forwards and backwards, but dealing with invalid PERSISTENT state that might have been caused during the version upgrade / downgrade, is a totally a different story. There are some basic rules that you have to follow (or mistakes that you have to avoid) in order to keep that STATEFULL version upgrade / downgrade process easy and frictionless. The basic rule is "don't make breaking changes during adjacent version upgrades!". Simple as that. Breaking changes are, for example, deleting a column, shortening a column length, changing column type and alike. Applying these changes will make it impossible to rollback a deployment, and for sure will cause data loss which requires downtime and restore from a snapshot (if you're lucky enough to have one). Imagine what happens if you have for example a column called ID which is numeric at application version 1.0. Then you decide on app version 1.1 to CHANGE ITS TYPE to string. There are 2 issues here: 1. Due to Kubernetes nature (actually cloud-native nature), you have to support, for a relatively short period of time, the existence of 2 live versions on the air: version 1.0 and 1.1. If the database schema cannot support them running concurrently, you cannot make such a database change with zero downtime. One of the version will have problems reading from or writing to unexpected schema, from its point of view. 2. So let's think in a safer way. Let's say you didn't CHANGED COLUMN TYPE, but instead created a new column called "ID_STR" which is used by application version 1.1. In order for both versions to read from the same database, this schema-change should populate the newly created "ID_STR" with values from the old numeric column "ID". This way both versions will be able to simultaneously READ from the same database without issues. However, new WRITES to the database, will be written to the newly created column "ID_STR", leaving the old column "ID" null or zero. This is okay for the long period, assuming the deployment succeeded, and application version 1.0 which uses the column "ID" will not live any more. However, temporarily, during the deployment time, when application version 1.0 is partially live and get some traffic, it will not know how to deal with newly created data that used the new "ID_STR" column. It simply sees zero or null in the "ID" column. And worse, what if the deployment failed for some reason and you need to roll back to version 1.0? You are now stuck with some newly created data that version 1.0 cannot read and may fail to process (null or zero IDs). This is an example of a non backwards-and-forwards compatible database schema change. And it happens so many times, making Kubernetes's promise to "upgrade with zero downtime" be a nice dream, not a reality. So what I would expect from a REAL cloud-native database schema management tool is to tell me explicitly "WARNING! this change is not backwards / forwards compatible and may cause issues during version rollout and rollback, are you sure you want to apply it?" Actually, I am not sure that looking at the schema-change, you can tell if you are taking a risk of not being able to rollback to previous version with zero downtime. Maybe you need knowledge of both database schema and application logic. In the example above, let's say that version 1.1 is configured to temporarily keep generating only numeric IDs, while storing them in a string-typed column. Then it can also populate the numeric old "ID" column, keeping a zero-downtime rollback possible. Only when deployment completed and there are no more live pods running version 1.0, you can safely configure version 1.1 to start generating string-based IDs and stop populating the numeric old ID column. Then, on the upgrade to version 1.2, you may safely delete column ID altogether, because you are sure that no live pod is using it at the moment or will be using it in the future. This seems like somewhat complex logic to follow, but this is where we fall with downtimes, even if we use Kubernetes. So let's start thinking how to improve this DB-upgrade process in a safer and frictionless way, not just ask ourselves if technically a tool uses CRDs and operators or not. This is, in my opinion, the way to get to a "Kubernetes native database schema management tool".
Continuing my vision, I think a "deprecated flag" should be applied to columns that are going to be deleted. This is now a 2-step process: you upgrade the application version, while tagging some column as deprecated. Then, on next upgrade, you can safely delete deprecated columns. And add 2 more things: 1. On application runtime, give a clear warning on application startup that "this version is using deprecated database columns". This will occur in the example above, for version 1.0, during a long deployment (say canary deployment), when pod has restarted. 2. Make the database schema tool give a clear warning if you are going to delete a non-deprecated column. Anyway, don't allow (or clearly warn before) changing column type or shorten column length. Work in an immutable way: create a new column, deprecate the old way, then, on next version upgrade, delete it. This may make application database schema upgrade somewhat safer, I hope.
@@DevOpsToolkit This is a good question. In my opinion, you should have a temporary background process (writable-view or external script), running until deployment completes, populating values between the old and the new columns for each newly created record. I am curious why you started your question with "If". Do you see another possible way to make a continuous deployment (e.g. canary) without having 2 service versions running concurrently, having to deal (both reading and writing) with a slightly different DB schema? (assuming zero downtime of course) You can of course limit the writes temporarily during the deployment, or to route writes only to the new version, if possible (e.g. REST POST, PUT and PATCH verbs), and the new version will make sure the old version can still read the new data. But this is not fully high-availability solution. It's probably a question of balance between solution complexity and service availability.
@@DavidPeleg-kv9to DB Schemas are similar to APIs. The idea scenario is to make new versions compatible with previous ones (equivalent to releasing a minor instead of a major release). I said "if", mostly because incompatible versions are the last resort. Sometimes there is no other way around it. Still, it's only the last resort when all other options are discarded.
Liquibase ecosystem has improved quite a bit. Liquibase has their own package manager, container image, and supports yaml files! That said, do you have any suggestions for managing entire dbms. Create Roles/users, database, database permissions, schema, schema permissions?
I tend to add crossplane to the mix. It does not do all you mentioned but it does manage everything available though a provider (e.g. gcp, AWS, etc.). plus there is the SQL provider that helps with databases and users.
@DevOps Toolkit i added crossplane and was super impressed. Thanks you! However I am still looking for a scalable solution for databaseSchema management. I don't see anything in the Atlas docs for creating a schema or modifying schema permissions. Any suggestions there?
Does Atlas support complex changes, for example create temp tables to back up data for data migrations?.. You mentioned Atlas has a terraform provider, does this mean we can use it with Crossplane and Argo? Also, can it support multi-tenancy( Ie, make changes to X number of schemas that are the same on the same DB server, where X is a list of tenant_ids ? )
"Complex changes" should work in the versioned mode. Terraform cannot be used with Argo CD, at least not directly. If you wrap it into a Kubernetes resource (e.g. crossplane terraform provider, tf operator, etc.) than it would work.
I think I'd like the declarative format a lot more if I could `DROP TABLE comments`, and then Atlas knows how to put all the data I'm deleting into an S3 bucket or something so that if I `atlas rollback`, the data can be restored without doing something that would cause other data loss like a snapshot restore. That said, you probably don't need to do any of that kind of stuff if you're following good practices around non-breaking change management and data warehousing/archival and so on.
You can roll back with Atlas. However, I do NOT recommend doing that in production (only in pre production). The moment the first transaction enters the DB with the new schema, there is no going back without loosing thst transaction or, if you're lucky, without a serious roll back plan that will transition data from the new to the old format and assuming that is even possible (it's often not). Try to roll forward...
Yeah. I did a video about it as soon as the operator was released. Unfortinately, RUclips does not low updates so I cannot reminder that con from the first video.
What do you use to manage database schemas?
TypeScript End-to-End: Prisma, EdgeDB. For events: -> Materialize
knex
Flyway. Till DevOps and SREs are 100% responsible for the schemas.... Id rather write XML // SQL instead of HCL... (sr Java dev here)
@@_ooooak I haven't used that one. Adding it to my TODO list...
liquibase
Wow you got me really interested in Atlas !
Great video, Viktor! Thank you!
FYI: v0.11.0 introduced the Atlas Kubernetes Operator. -1 con :)
True. That was one of my complained, they listened and added the operator. I've been using it ever since.
Thank you! Way back I was trying to write my own declarative db schema generator for go (it’d generate the sql as well as the go models), so I’ve been looking for something like this!
I used flyway in the past for java projects, it works well with maven builds.
This is a minor complaint from a newbie to the cloud database environment. As I watched this video, I assumed that Atlas was the no SQL database from mongoDB. It took a while before it dawned on me that atlas is the name of a standalone database management tool. I did a Google search on atlas schema management, and found entries for the standalone tool, and for the schema management capabilities of the mongo, DB Atlas database. I think you can see why people not totally immersed in this field could be confused. Also, I get the impression that the atlas schema management tool works for SQL schemas, and not for no SQL schemas. Again, it would’ve been nice to simply establish the context for this discussion. On the other hand, I have noticed that many technical videos on RUclips tend to assume that their viewers are familiar with their context and do not take the trouble to provide an overview fot their discussion. The information from this RUclips channel is excellent. Keep up the good work!
Atlas is a terrible name for that project. It's hard to Google it, it's used in too many other projects, etc.
Most of schema management tools are focused on SQL DBs, mostly because there is not such a big need to manage schemas on nosql DBs.
As for the context... It's hard to balance it. If there's too much of it, some will go somewhere else when they know it. On the other hand, if there's not enough of it, others will go somewhere else because it's confusing. I myself am never sure whether I should do more or less introduction and background on any given topic :(
I totally understand. I will continue to watch your videos to gain more understanding of cloud technologies. Thanks so much.
Well, basically, a declarative approach to database schema could be a step forward, but I think something very important is missing here, when we talk about Kubernetes nature of agile deployments:
One of the basic ideas of cloud-native (or container-native) is the separation of logic and state, and the ability to upgrade versions with zero downtime.
This zero-downtime service update is done by switching to the new version gradually ("rolling update", "canary deployment") or at once ("blue-green deployment"). Either way, Continuous Deployment and Agile concepts allow you to decide that this version upgrade was a mistake, and to roll back to the previous version, keeping the service running with minimal disruption to the clients.
Kubernetes made it very easy to roll a STATELESS application version forwards and backwards, but dealing with invalid PERSISTENT state that might have been caused during the version upgrade / downgrade, is a totally a different story.
There are some basic rules that you have to follow (or mistakes that you have to avoid) in order to keep that STATEFULL version upgrade / downgrade process easy and frictionless.
The basic rule is "don't make breaking changes during adjacent version upgrades!". Simple as that.
Breaking changes are, for example, deleting a column, shortening a column length, changing column type and alike. Applying these changes will make it impossible to rollback a deployment, and for sure will cause data loss which requires downtime and restore from a snapshot (if you're lucky enough to have one).
Imagine what happens if you have for example a column called ID which is numeric at application version 1.0. Then you decide on app version 1.1 to CHANGE ITS TYPE to string. There are 2 issues here:
1. Due to Kubernetes nature (actually cloud-native nature), you have to support, for a relatively short period of time, the existence of 2 live versions on the air: version 1.0 and 1.1. If the database schema cannot support them running concurrently, you cannot make such a database change with zero downtime. One of the version will have problems reading from or writing to unexpected schema, from its point of view.
2. So let's think in a safer way. Let's say you didn't CHANGED COLUMN TYPE, but instead created a new column called "ID_STR" which is used by application version 1.1. In order for both versions to read from the same database, this schema-change should populate the newly created "ID_STR" with values from the old numeric column "ID". This way both versions will be able to simultaneously READ from the same database without issues.
However, new WRITES to the database, will be written to the newly created column "ID_STR", leaving the old column "ID" null or zero. This is okay for the long period, assuming the deployment succeeded, and application version 1.0 which uses the column "ID" will not live any more. However, temporarily, during the deployment time, when application version 1.0 is partially live and get some traffic, it will not know how to deal with newly created data that used the new "ID_STR" column. It simply sees zero or null in the "ID" column.
And worse, what if the deployment failed for some reason and you need to roll back to version 1.0? You are now stuck with some newly created data that version 1.0 cannot read and may fail to process (null or zero IDs).
This is an example of a non backwards-and-forwards compatible database schema change.
And it happens so many times, making Kubernetes's promise to "upgrade with zero downtime" be a nice dream, not a reality.
So what I would expect from a REAL cloud-native database schema management tool is to tell me explicitly "WARNING! this change is not backwards / forwards compatible and may cause issues during version rollout and rollback, are you sure you want to apply it?"
Actually, I am not sure that looking at the schema-change, you can tell if you are taking a risk of not being able to rollback to previous version with zero downtime. Maybe you need knowledge of both database schema and application logic.
In the example above, let's say that version 1.1 is configured to temporarily keep generating only numeric IDs, while storing them in a string-typed column. Then it can also populate the numeric old "ID" column, keeping a zero-downtime rollback possible. Only when deployment completed and there are no more live pods running version 1.0, you can safely configure version 1.1 to start generating string-based IDs and stop populating the numeric old ID column. Then, on the upgrade to version 1.2, you may safely delete column ID altogether, because you are sure that no live pod is using it at the moment or will be using it in the future.
This seems like somewhat complex logic to follow, but this is where we fall with downtimes, even if we use Kubernetes.
So let's start thinking how to improve this DB-upgrade process in a safer and frictionless way, not just ask ourselves if technically a tool uses CRDs and operators or not.
This is, in my opinion, the way to get to a "Kubernetes native database schema management tool".
Continuing my vision, I think a "deprecated flag" should be applied to columns that are going to be deleted. This is now a 2-step process: you upgrade the application version, while tagging some column as deprecated. Then, on next upgrade, you can safely delete deprecated columns.
And add 2 more things:
1. On application runtime, give a clear warning on application startup that "this version is using deprecated database columns". This will occur in the example above, for version 1.0, during a long deployment (say canary deployment), when pod has restarted.
2. Make the database schema tool give a clear warning if you are going to delete a non-deprecated column. Anyway, don't allow (or clearly warn before) changing column type or shorten column length. Work in an immutable way: create a new column, deprecate the old way, then, on next version upgrade, delete it.
This may make application database schema upgrade somewhat safer, I hope.
If the old table is deprecated, it means that people are still using it in parallel with the new one. How would you do data migration?
@@DevOpsToolkit
This is a good question. In my opinion, you should have a temporary background process (writable-view or external script), running until deployment completes, populating values between the old and the new columns for each newly created record.
I am curious why you started your question with "If". Do you see another possible way to make a continuous deployment (e.g. canary) without having 2 service versions running concurrently, having to deal (both reading and writing) with a slightly different DB schema? (assuming zero downtime of course)
You can of course limit the writes temporarily during the deployment, or to route writes only to the new version, if possible (e.g. REST POST, PUT and PATCH verbs), and the new version will make sure the old version can still read the new data. But this is not fully high-availability solution. It's probably a question of balance between solution complexity and service availability.
@@DavidPeleg-kv9to DB Schemas are similar to APIs. The idea scenario is to make new versions compatible with previous ones (equivalent to releasing a minor instead of a major release). I said "if", mostly because incompatible versions are the last resort. Sometimes there is no other way around it. Still, it's only the last resort when all other options are discarded.
nice
Liquibase ecosystem has improved quite a bit. Liquibase has their own package manager, container image, and supports yaml files!
That said, do you have any suggestions for managing entire dbms. Create Roles/users, database, database permissions, schema, schema permissions?
I tend to add crossplane to the mix. It does not do all you mentioned but it does manage everything available though a provider (e.g. gcp, AWS, etc.). plus there is the SQL provider that helps with databases and users.
@DevOps Toolkit i added crossplane and was super impressed. Thanks you!
However I am still looking for a scalable solution for databaseSchema management. I don't see anything in the Atlas docs for creating a schema or modifying schema permissions. Any suggestions there?
@@acelinkio doc.crds.dev/github.com/crossplane-contrib/provider-sql might help.
Does Atlas support complex changes, for example create temp tables to back up data for data migrations?.. You mentioned Atlas has a terraform provider, does this mean we can use it with Crossplane and Argo? Also, can it support multi-tenancy( Ie, make changes to X number of schemas that are the same on the same DB server, where X is a list of tenant_ids ? )
"Complex changes" should work in the versioned mode.
Terraform cannot be used with Argo CD, at least not directly. If you wrap it into a Kubernetes resource (e.g. crossplane terraform provider, tf operator, etc.) than it would work.
👏👏👏
Could you use Atlas with the Terraform provider with Atlantis for CD?
I'm guessing that you can but I haven't tried it with Atlantis myself 🙁
Sadly it does not support postgresql extension such as Timescale. Any alternatives in this case?
Schemahero does not support it either :(
I think I'd like the declarative format a lot more if I could `DROP TABLE comments`, and then Atlas knows how to put all the data I'm deleting into an S3 bucket or something so that if I `atlas rollback`, the data can be restored without doing something that would cause other data loss like a snapshot restore.
That said, you probably don't need to do any of that kind of stuff if you're following good practices around non-breaking change management and data warehousing/archival and so on.
How you manage the rollback then ?
You can roll back with Atlas. However, I do NOT recommend doing that in production (only in pre production). The moment the first transaction enters the DB with the new schema, there is no going back without loosing thst transaction or, if you're lucky, without a serious roll back plan that will transition data from the new to the old format and assuming that is even possible (it's often not). Try to roll forward...
Sounds interesting!
Can it treat renames properly?
I haven't had the case of renaming a table or a column just yet so I haven't tried that out :(
Atlas now has a Kubernetes Operator and documents usage with ArgoCD. That Con should be gone.
Yeah. I did a video about it as soon as the operator was released. Unfortinately, RUclips does not low updates so I cannot reminder that con from the first video.