I have a generic cleanup script that I'd like to run after every migration. Is there a good way to have this script run after each migration (short of including the script itself as a change every time I do a migration?)
I see that this question has been asked before here Pre- and Post-migration scripts for Flyway and the answer at that time was no, not really.
Has the answer changed at all in the past 1.5 years?
With flyway 3.0 the situation has changed and there are now callback scripts possible. In this situation an afterMigration.sql file could be used to do the cleanups.
See https://flywaydb.org/documentation/concepts/callbacks and https://flywaydb.org/documentation/tutorials/callbacks for more information.
This has not changed. Use any of the suggested workarounds for now.
I've looked at the suggestions here and Pre- and Post-migration scripts for Flyway and would like to point out a use case that I can't see which workaround (if any) would be most applicable. The use case is to have a dba create a restore point before running developer created migrations.
Right now, with our manual (non-flyway) migration process, a dba creates a restore point before running a set of migrations. The migrations would run fine without the restore point. But if they don't have the correct code (say missing creating a column), it's often preferable to roll back to the oracle restore point, to avoid downtime and give the developer time to work on a fix.
I don't think requiring the developer to include a migration that does that restore point makes sense, because: 1. They might forget (it should automatically happen, without developer intervention) 2. depending on the state of the schema, there may different starting migrations, so if the one that includes the restore point is not run, it may be old, and data may have changed in the interim.
Having a separate migration that does the restore point has similar drawbacks: 1. They would have to manually create a new migration that is essentially a copy of an old migration with a different version number to do the restore point.
For development schemas, with large existing data, it's not practical to clean out the schema while developing a migration, because it predates flyway and may take significant time to recreate from scratch.
For development, ideally the workflow is something like this: 1. create restore point 2. develop migration(s), run using flyway 3. roll back to restore point if migration doesn't work as required.
If there is a way to automate step #1 across the board, it would allow us to use flyway and eliminate the need for a dba, except in the cases where something went wrong and there would be a roll back necessary. There may be a more 'flyway' way to approach the problem, but the workarounds I found don't seem to fit into our existing workflow.
We had the same problem. I.e., call a bunch of scripts always before and after every migration. E.g., deleting and creating materialized view, granting permissions to tables. These scripts do not change from migration to migration, but they need to be executed.
So I took the org.flywaydb.core.internal.callback.SqlScriptFlywayCallback callback class and adapted it for multiple files.
I tried to stay in the philosophy of flyway and use the following pattern.
Files starting with am__ or AM__ are an after migration script, those with bi__ are for before info, and so on.
I sort the scripts, so that they are executed in the correct order.
public class MultipleScriptPerCallback extends BaseFlywayCallback {
private static final Log LOG = LogFactory.getLog(SqlScriptFlywayCallback.class);
private static final String DELIMITER = "__";
private static final String BEFORE_CLEAN = "bc";
private static final String AFTER_CLEAN = "ac";
private static final String BEFORE_MIGRATE = "bm";
private static final String AFTER_MIGRATE = "am";
private static final String BEFORE_EACH_MIGRATE = "bem";
private static final String AFTER_EACH_MIGRATE = "aem";
private static final String BEFORE_VALIDATE = "bv";
private static final String AFTER_VALIDATE = "av";
private static final String BEFORE_BASELINE = "bb";
private static final String AFTER_BASELINE = "ab";
private static final String BEFORE_REPAIR = "br";
private static final String AFTER_REPAIR = "ar";
private static final String BEFORE_INFO = "bi";
private static final String AFTER_INFO = "ai";
private static final List<String> ALL_CALLBACKS = Arrays.asList(BEFORE_CLEAN, AFTER_CLEAN, BEFORE_MIGRATE, BEFORE_EACH_MIGRATE,
AFTER_EACH_MIGRATE, AFTER_MIGRATE, BEFORE_VALIDATE, AFTER_VALIDATE, BEFORE_BASELINE, AFTER_BASELINE, BEFORE_REPAIR,
AFTER_REPAIR, BEFORE_INFO, AFTER_INFO);
private Map<String, List<SqlScript>> scripts;
@Override
public void setFlywayConfiguration(FlywayConfiguration flywayConfiguration) {
super.setFlywayConfiguration(flywayConfiguration);
if (scripts == null) {
scripts = registerScripts(flywayConfiguration);
}
}
private Map<String, List<SqlScript>> registerScripts(FlywayConfiguration flywayConfiguration) {
Map<String, List<SqlScript>> scripts = new HashMap<>();
for (String callback : ALL_CALLBACKS) {
scripts.put(callback, new ArrayList<SqlScript>());
}
LOG.debug(String.format("%s - Scanning for Multiple SQL callbacks ...", getClass().getSimpleName()));
Locations locations = new Locations(flywayConfiguration.getLocations());
Scanner scanner = new Scanner(flywayConfiguration.getClassLoader());
String sqlMigrationSuffix = flywayConfiguration.getSqlMigrationSuffix();
DbSupport dbSupport = dbSupport(flywayConfiguration);
PlaceholderReplacer placeholderReplacer = createPlaceholderReplacer();
String encoding = flywayConfiguration.getEncoding();
for (Location location : locations.getLocations()) {
Resource[] resources;
try {
resources = scanner.scanForResources(location, "", sqlMigrationSuffix);
} catch (FlywayException e) {
// Ignore missing locations
continue;
}
for (Resource resource : resources) {
String key = extractKeyFromFileName(resource);
if (scripts.keySet().contains(key)) {
LOG.debug(getClass().getSimpleName() + " - found script " + resource.getFilename() + " from location: " + location);
List<SqlScript> sqlScripts = scripts.get(key);
sqlScripts.add(new SqlScript(dbSupport, resource, placeholderReplacer, encoding));
}
}
}
LOG.info(getClass().getSimpleName() + " - scripts registered: " + prettyPrint(scripts));
return scripts;
}
private String prettyPrint(Map<String, List<SqlScript>> scripts) {
StringBuilder prettyPrint = new StringBuilder();
boolean isFirst = true;
for (String key : scripts.keySet()) {
if (!isFirst) {
prettyPrint.append("; ");
}
prettyPrint.append(key).append("=").append("[").append(prettyPrint(scripts.get(key))).append("]");
isFirst = false;
}
return prettyPrint.toString();
}
private String prettyPrint(List<SqlScript> scripts) {
StringBuilder prettyPrint = new StringBuilder();
boolean isFirst = true;
for (SqlScript script : scripts) {
if (!isFirst) {
prettyPrint.append(", ");
}
prettyPrint.append(script.getResource().getFilename());
isFirst = false;
}
return prettyPrint.toString();
}
private String extractKeyFromFileName(Resource resource) {
String filename = resource.getFilename();
eturn filename.substring(0, (!filename.contains(DELIMITER)) ? 0 : filename.indexOf(DELIMITER)).toLowerCase();
}
private DbSupport dbSupport(FlywayConfiguration flywayConfiguration) {
Connection connectionMetaDataTable = JdbcUtils.openConnection(flywayConfiguration.getDataSource());
return DbSupportFactory.createDbSupport(connectionMetaDataTable, true);
}
/**
* @return A new, fully configured, PlaceholderReplacer.
*/
private PlaceholderReplacer createPlaceholderReplacer() {
if (flywayConfiguration.isPlaceholderReplacement()) {
return
new PlaceholderReplacer(flywayConfiguration.getPlaceholders(), flywayConfiguration.getPlaceholderPrefix(),
flywayConfiguration.getPlaceholderSuffix());
}
return PlaceholderReplacer.NO_PLACEHOLDERS;
}
@Override
public void beforeClean(Connection connection) {
execute(BEFORE_CLEAN, connection);
}
@Override
public void afterClean(Connection connection) {
execute(AFTER_CLEAN, connection);
}
@Override
public void beforeMigrate(Connection connection) {
execute(BEFORE_MIGRATE, connection);
}
@Override
public void afterMigrate(Connection connection) {
execute(AFTER_MIGRATE, connection);
}
@Override
public void beforeEachMigrate(Connection connection, MigrationInfo info) {
execute(BEFORE_EACH_MIGRATE, connection);
}
@Override
public void afterEachMigrate(Connection connection, MigrationInfo info) {
execute(AFTER_EACH_MIGRATE, connection);
}
@Override
public void beforeValidate(Connection connection) {
execute(BEFORE_VALIDATE, connection);
}
@Override
public void afterValidate(Connection connection) {
execute(AFTER_VALIDATE, connection);
}
@Override
public void beforeBaseline(Connection connection) {
execute(BEFORE_BASELINE, connection);
}
@Override
public void afterBaseline(Connection connection) {
execute(AFTER_BASELINE, connection);
}
@Override
public void beforeRepair(Connection connection) {
execute(BEFORE_REPAIR, connection);
}
@Override
public void afterRepair(Connection connection) {
execute(AFTER_REPAIR, connection);
}
@Override
public void beforeInfo(Connection connection) {
execute(BEFORE_INFO, connection);
}
@Override
public void afterInfo(Connection connection) {
execute(AFTER_INFO, connection);
}
private void execute(String key, Connection connection) {
List<SqlScript> sqlScripts = scripts.get(key);
LOG.debug(String.format("%s - sqlscript: %s for key: %s", getClass().getSimpleName(), sqlScripts, key));
Collections.sort(sqlScripts, new SqlScriptLexicalComparator());
for (SqlScript script : sqlScripts) {
executeScript(key, connection, script);
}
}
//Not private for testing
void executeScript(String key, Connection connection, SqlScript script) {
LOG.info(String.format("%s - Executing SQL callback: %s : %s", getClass().getSimpleName(), key,
script.getResource().getFilename()));
script.execute(new JdbcTemplate(connection, 0));
}
//Not private for testing
static final class SqlScriptLexicalComparator implements Comparator<SqlScript> {
@Override
public int compare(SqlScript o1, SqlScript o2) {
return Collator.getInstance().compare(o1.getResource().getFilename(), o2.getResource().getFilename());
}
}
}
