How can I roll-back a dataset transaction in Foundry?

Question

I have an incremental dataset on Foundry and a file with incorrect data was uploaded. How do I reverse this transaction so I can update the dataset with the correct data?

Answer 1

You can use foundry's Catalog API. You will first need to find the resource id (rid) of the transaction you want to revert to, this can be found under the history tab when you select the dataset in Monocle. You will also need the rid of your dataset, and a bearer token you've generated on your foundry instance. Run the following in a unix command-line or alternatively using the python requests library. (the requests library may be useful if you're on a Windows machine)

curl -X POST \
    "https://<CATALOG_URL>/api/catalog/datasets/<DATASET_RID>/branchesUpdate2/master" \
    -H "Authorization: Bearer <TOKEN>" \
    -H "Content-type: application/json" \
    -d '"'<TRANSACTION_RID>'"' \
    -k \
    -w "\n"

Replace what is between <> with your relevant variables, I've shown some samples below so you'll recognise the variables when you see them. Be sure and keep your bearer token a secret.

<CATALOG_URL>     <- your.url.com/foundry-catalog
<DATASET_RID>     <- ri.foundry.main.dataset.00000000-bbbb-cccc-dddd-000000000000
<TOKEN>           <- ey00000000...00000000
<TRANSACTION_RID> <- "ri.foundry.main.transaction.00000000-bbbb-cccc-dddd-000000000000"

Answer 2

Other option that I’ve done is:

1. Use contour to grab the transaction of choice and export as CSV followed by upload as static dataset.
2. Update schema to ensure it’s the desired types.
3. Go to incremental code and update set(‘modify’) to set(‘replace’) for the output dataframe if using @incremental declarator.
4. Add or replace Input reference with the static data reference. Update logic to reference new input for the output.
5. Build data set (this should do a snapshot using your static data)
6. change code back to modify and input back to original reference and any other code modes back to original. Any new build from there will increment off the last snapshot run you did.

It may not be very elegant but has been done numerous times.

Answer 3

Here is a python function to achieve the same as FJ_OC's curl example:

from urllib.parse import quote_plus
import requests

def update_branch(dataset_rid: str,
                  branch: str,
                  parent_branch_or_transaction_rid: str = None,
                  api_base,
                  token) -> dict:
    """
    Updates the latest transaction of branch 'branch' to the latest transaction
    of branch 'parent_branch' OR to the transaction on the same branch given as parameter
    Args:
        dataset_rid: Unique identifier of the dataset
        branch: The branch to update (e.g. master)
        parent_branch_or_transaction_rid: the name of the branch to copy the last transaction
        from or the transaction on the same branch to reset the dataset to

    Returns: branch response, e.g.:
    {'id': '..',
     'rid': 'ri.foundry.main.branch...',
     'ancestorBranchIds': [],
     'creationTime': '',
     'transactionRid': 'ri.foundry.main.transaction....'
     }

    """
    response = requests.post(f"{api_base}/foundry-catalog/api/catalog/datasets/{dataset_rid}/"
                             f"branchesUpdate2/{quote_plus(branch)}",
                             headers={'Content-Type': 'application/json', 'Authorization': f'Bearer {token}'},
                             data=f'"{parent_branch_or_transaction_rid}"'
                            )
    response.raise_for_status()
    return response.json()

Answer 4

Another solution-

To roll back on master: (Split in variable declaration and curl for better readability)

API_URL=‘https://.......com/foundry-catalog’
BRANCH_ID=master
DATASET_RID=ri.foundry.main.dataset.(...your dataset...)
TOKEN=(...your token...)

curl -X POST "${API_URL}/api/catalog/datasets/${DATASET_RID}/branchesUpdate2/${BRANCH_ID}" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d "\"ri.foundry.main.transaction.(...the transaction....)\""

(Please note that the transaction is in quotes)