RhinoETL merge table with webservice results

Question

I have a table in SQL Server that is populated from a Web Service. I want it to be refreshed on a scheduled basis. I would like to have something similar to an SQL Merge operation.

That is i define my source (Web Service) and my target (SQL Table) and i define how to handle missing from source missing from target and matches.

Lets consider a scenario where i have only two fields in the table Description and Deleted and the Web Service provides only the Description.

If a description is present in both the table and the web service then i just updated (or not).
If a description is present in the web service but not in the table i want it to be inserted
If a description is no longer present in the web server i want it marked as Deleted = true

What i currently have is:

public class WebServiceResults: AbstractOperation
{
    public WebServiceResults()
    {
        var WebService = new WebService();
        WSResults = WebService.GetResults();
    }
    IEnumerable<WSResult> WSResults  { get; set; }
    public override IEnumerable<Row> Execute(IEnumerable<Row> rows)
    {
        foreach(var obj in WSResults)
            yield return Row.FromObject(obj);
    }

}

class SQLTableResults : AbstractOperation
{
    public SQLTableResults()
    {
        SQLResults = data.MyTable.Select(x=>new {x.Description,x.Deletet});
    }
    Data data = new Data();
    IEnumerable<SQLResult> SQLResults  { get; set; }
    public override IEnumerable<Row> Execute(IEnumerable<Row> rows)
    {
        foreach (var obj in SQLResults)
            yield return Row.FromObject(obj);
        }
    }
    public override void Dispose()
    {
        data.Dispose();
        base.Dispose();
    }
}

class JoinTables : JoinOperation
{

    protected override Row MergeRows(Row leftRow, Row rightRow)
    {
        Row row = leftRow.Clone();
        row["Description2"] = rightRow["Description"];
        return row;
    }

    protected override void SetupJoinConditions()
    {
        FullOuterJoin
            .Left("Description")
            .Right("Description");
    }
}

class MergeTables : AbstractOperation
{
    Data data = new Data();
    public MergeTables()
    { }

    public override IEnumerable<Row> Execute(IEnumerable<Row> rows)
    {
        foreach (var obj in rows)
        {
            if (String.IsNullOrEmpty((string)obj["Description2"]))
            {
                //Code for not matched at target
                yield return Row.FromObject(obj);
            }
            if (String.IsNullOrEmpty((string)obj["Description"]))
            {
                //Code for not matched at source
                yield return Row.FromObject(obj);
            }

            {
                //Code for matched
                yield return Row.FromObject(obj);
            }
        }
    }
    public override void Dispose()
    {
        data.Dispose();
        base.Dispose();
    }
}

protected override void Initialize()
{
    Register(
           new JoinTables()
           .Left(new SQLTableResults())
           .Right(new WebServiceResults())
       );
    Register(new MergeTables());
    foreach (var error in GetAllErrors())
        Console.Write(error.Message);
}

Is this the way to go? I would imagine something more of a stepped process, like

Register(new NotMatchedAtSourceOperation());
Register(new NotMatchedAtTargetOperation());
Register(new MatchedOperation());

but as i understand it, each register returns its rows to the next, so if i filter for the not matched, then the other two will do nothing.

Should i create a new process for each case?

By the way, i am looking for documentation on RhinoEtl. Do you know of any links? Any tutorials?

score 1 · Accepted Answer · answered May 30 '14 at 16:06

Determine the merge action in a single operation using a full outer join. You can see an example here. It's not exactly what you need, so I'll try to adapt it to your situation below:

protected override Row MergeRows(Row wsRow, Row dbRow) {

    Row row;

    // if the db row doesn't exist, then the ws row is new, and it should be inserted
    if (dbRow["id"] == null) {
        row = wsRow.Clone();
        row["action"] = "Insert";
        row["deleted"] = false;
        return row;
    }

    // if the ws row doesn't exist, it should be marked as deleted in the database (if not already)
    if (wsRow["id"] == null) {
        row = dbRow.Clone();
        row["deleted"] = true;
        row["action"] = dbRow["deleted"].Equals(true) ? "None" : "Update";
        return row;
    }

    // ws and db descriptions match, but check and make sure it's not marked as deleted in database
    row = wsRow.Clone();
    row["deleted"] = false;
    row["action"] = dbRow["deleted"].Equals(true) ? "Update" : "None";
    return row;

}

protected override void SetupJoinConditions() {
    FullOuterJoin.Left("description").Right("description");
}

After you run this this operation, every row will have an action of "Insert", "Update", or "None." Based on this action, you can compose insert and update statements for a SqlBatchOperation to execute.

Sorry for taking so long to respond. This is what i was looking for. The only drawback i can see is that we cannot utilize IEnumerable so we end up "parsing" the whole datasets first. On the other hand i fail to find a way of doing this with IEnumerable without having to parse them prior.... Thanks anyway. — Giannis Paraskevopoulos, Jun 15 '14 at 09:52

RhinoETL merge table with webservice results

1 Answers1