12-factor app and once-off scripts?
According to 12-factor app, once off scripts should be run in an environment identical to production.
So if you have a script like python backfill_missing_records.py then you should run it in an environment that matches production, i.e., same docker image, same environment variables, etc.
But this is pretty hard to do?
Our production system is docker containers running on ECS fargate. We can have jenkins issue a task that pulls the latest docker container, runs the desired script and then exits, however --
1 - this is slow
2 - feedback cycles are slow (you don't find out if the script had a problem for several minutes)
3 - compared to SSHing into a production machine to run the script, you can't immediately take corrective action if something does go wrong
Up till now we've been maintaining machines in production environments that expose a SSH port and we've been manually going into those machines and running once of scripts as needed. However, these machines aren't guaranteed to stay in parity with production and generally just have a copy of the repository and some of the correct environment variables.
These machines are different from prod (EC2 vs ECS, no docker image, parameter drift, etc). But on the other hand, running these kinds of scripts in a prod-like environment seems pretty inconvenient as well.
How do other teams resolve this?
