Upgrading a core production database is usually the stuff of dev nightmares. It often involves maintenance windows, scheduled downtime, and the looming fear of a botched migration. But what if you could execute a major version jump with almost zero downtime and absolute confidence?
Recently, we upgraded our main production database from MySQL 8.0 to MySQL 8.4. By utilizing AWS Blue/Green Deployments and scheduling the operation during a low-traffic period (Japan’s Golden Week), we pulled it off smoothly.
Here is a look at our exact playbook, the errors we caught before they became disasters, and how you can replicate this success.
Phase 1: The Pre-Flight Checks
You cannot blindly upgrade a database, especially with MySQL 8.4 introducing major breaking changes (like the removal of the mysql_native_password plugin).
The Magic Command mysqlsh
Before touching anything in AWS, we ran the official MySQL upgrade checker against our 8.0 database. This utility scans your schema and flags potential fatal errors that would cause an upgrade to fail.
mysqlsh -- util check-for-server-upgrade root@your-db-host:3306 --target-version=8.4.0
Takeaway: Never skip this step. Treat every "Error" in the output as a hard blocker that must be resolved before proceeding.
The DEFINER Problem
During our checks, we ran into a classic database migration issue: rogue DEFINER clauses.
When you create database objects like Views, Routines (Functions/Procedures), Triggers, or Events, MySQL attaches a DEFINER to them (e.g., DEFINER='some_user'@'%'). If your database was ever restored from a test or staging environment, it might contain objects owned by a user that does not exist in your production environment. If you upgrade the database with missing definers, those objects can break or prevent the upgrade entirely.
The Fix:
You cannot simply UPDATE the information schema to fix this. The best way to resolve it is to dump the objects, find/replace the old definer with your standard root user (or CURRENT_USER), and re-import them.
For Routines: Use mysqldump --routines --no-create-info --no-data to extract them, replace the definers in your code editor, and re-run the script.
For Views, Triggers, and Events: You must similarly DROP and recreate them with the correct DEFINER clause before starting the upgrade process.
Phase 2: AWS Configuration Prep
With the database clean, we had to prepare our AWS RDS environment.
Parameter Groups
AWS RDS relies on Parameter Groups to manage database configurations (like memory limits, connection timeouts, etc.). You cannot use an 8.0 Parameter Group on an 8.4 database. Before upgrading, you must create a brand new Parameter Group specifically for the MySQL 8.4 family and map your custom variables over.
Enabling Binlogs (Binary Logging)
This is the secret sauce for a seamless upgrade. We ensured that binary logging binlog) was enabled on our primary 8.0 database.
Why? Because when AWS creates the new 8.4 environment, it uses these binlogs to keep track of every single INSERT, UPDATE, and DELETE happening on the old database, streaming those changes to the new one in real-time.
Phase 3: The Trial Run
We operate by the principle of Learning from failure—but we prefer to fail in a safe sandbox.
Before touching production, we executed this entire Blue/Green deployment strategy on our Test DB instance. This dry run gave us the confidence we needed. It allowed us to accurately time the switchover, verify that the application could connect to the new 8.4 instance, and confirm that our automated scripts worked as expected.
Phase 4: The Blue/Green Deployment
So, what exactly is a Blue/Green deployment?
In a traditional upgrade, your database goes offline, updates, and comes back online. In a Blue/Green deployment:
Blue is your current live 8.0 database.
AWS clones Blue and upgrades the clone to 8.4. This is your Green environment.
AWS sets up replication so Green is constantly catching up to Blue.
When you are ready, you click "Switchover." AWS swaps the DNS endpoints behind the scenes. Green becomes the new live database, and Blue is safely pushed aside.
The Benefits
Safe Rollbacks: If the switchover fails, the old Blue environment is still sitting right there, perfectly intact.
Pre-Testing: You can connect to the Green database before the switchover to test your application against it.
Minimal Downtime: The actual cutover is incredibly fast.
⚠️ The Cost Factor
Keep in mind that while the Green environment is syncing, you are running two identical database clusters. Your AWS RDS bill will effectively double for those specific hours or days. Do not leave a Blue/Green deployment running indefinitely!
Phase 5: The Switchover Execution
Even with Blue/Green deployments, it is wise to minimize active database transactions during the cutover.
Timing is Everything: We scheduled the production switchover during Japan's May Golden Week—a period of historically low traffic for our platform.
The Switchover Playbook:
Pause Background Jobs: We gracefully turned off all Laravel queue workers and cron schedulers across our application instances to ensure no background tasks failed mid-flight.
Hit the Button: We initiated the switchover in the AWS Console.
Wait 60 Seconds: The DNS switchover was instantaneous, taking only 1 to 2 minutes to fully resolve.
Resume: We turned the queue workers and cron jobs back on.
The Best Part: Zero Application Changes
Because AWS handles the endpoint swapping at the DNS level, we did not have to change a single line of application code or update any .env database URLs. The application simply blinked, reconnected, and was suddenly speaking to MySQL 8.4.
Final Thoughts
Database upgrades don't have to be terrifying. By combining the strict pre-flight checks of mysqlsh, resolving definer hygiene, practicing on a test environment, and leveraging AWS Blue/Green deployments, you can scale your infrastructure at full speed with minimal risk to your users.
Have questions about MySQL upgrades or Dev practices? Let's discuss in the comments below!
