2

We have some large Excel files (often in excess of 100MB each) that need to be regularly brought in to a SQL Server database and I am looking for options to automate this process.

It sounds like Microsoft is no longer supporting its Businesss Intelligence Development Studio (BIDS) and has replaced it with something called SQL Server Data Tools - Business Intelligence (SSDT-BI).

Because of this, it looks like my options are SSDT-BI or an SSIS package, but I am not sure which one to use. Can someone give a little bit of information about these two solutions? Any advice for finding a solution to this problem would be appreciated.

user7050575
  • 51
  • 1
  • 8
  • You develop ssis packages in ssdt. You have other options like Powershell. – SQLChao Dec 07 '16 at 20:47
  • We do a lot of transforms to our data and I have heard that SSIS packages only offer the ability to do very limited transforms. Is this correct? If so, do you think this should rule out SSDT and/or SSDT-BI for us? – user7050575 Dec 07 '16 at 20:49
  • Have you looked into PowerPivot? A few years ago, it was a very good solution for working with large Excel files, millions of rows, transforms, and aggregations. – digital.aaron Dec 07 '16 at 20:52
  • It depends. If you've never worked in SSIS I would suggest spinning up a test box with it installed so you can see what all is available. There's also some considerations from the DBA standpoint as there is maintenance involved such as SSIS catalog. – SQLChao Dec 07 '16 at 20:53
  • What is the difference between SSDT and SSDT-BI? I can't seem to find much information about the latter. – user7050575 Dec 07 '16 at 21:13
  • For versions prior to 2015, SSDT == database projects and SSDT-BI == SSRS, SSIS, and SSAS. VS 2015 and SSDT is now all-in-one and supports SSRS/SSAS/SSIS 2012, 2014, and 2016. – Peter Schott Dec 07 '16 at 23:35
  • SSIS with Excel sources is a one-way ticket to a land of pain and suffering, with many happy hours spent searching StackOverflow along the way. Might be an idea to look at C#/Powershell with (EPPlus or NPoi), or even Java/other JVM language with Poi. – Gavin Campbell Dec 08 '16 at 15:14

2 Answers2

0

I know this is an old post and it has nothing to do with SSIS nor SSDT but the OP was seemingly inquiring about other options that maybe available. So I wanted to add this for anyone who is looking for a easy way to import extract files without having to use either of those tools. I have created literally hundreds of processes where I'm required to import data from an extract file (CSV, Access, Excel, FoxPro, etc) the following is a Powershell snippet that will load up all sheets in an Excel Spreadsheet and then simply display the contents in a Data Grid but you should be able to easily add in your logic to import the data into a table for example. Constructive criticism is always welcome!

Clear-Host;
    ## You May Need to Download and Install the Microsoft Access Database Engine 2010 Redistributable: https://www.microsoft.com/en-us/download/details.aspx?id=13255

[String]$ExcelPath = "C:\Temp\TestSheet.xlsx";
[String]$TargetServer = "(local)";
[String]$TargetDatabase = "TestDB";

[String]$SourceConnectionString = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source={0};Extended Properties='Excel 12.0 Xml;HDR=YES';" -f $ExcelPath;
[String]$TargetConnectionString = "Data Source={0};Initial Catalog={1};Trusted_Connection=True;" -f $TargetServer, $TargetDatabase;

$SourceFactory = [System.Data.Common.DbProviderFactories]::GetFactory("System.Data.OleDb");
$TargetFactory = [System.Data.Common.DbProviderFactories]::GetFactory("System.Data.SqlClient");

    $SourceConnection = $SourceFactory.CreateConnection();
        $SourceConnection.ConnectionString = $SourceConnectionString;
        $SourceConnection.Open();

    $SourceCommand = New-Object $SourceFactory.CreateCommand();
        $SourceCommand.Connection = $SourceConnection;
        $SourceCommand.CommandTimeout = [Int32]::MaxValue;

    $TargetConnection = $TargetFactory.CreateConnection();
        $TargetConnection.ConnectionString = $TargetConnectionString;
        $TargetConnection.Open();

    $TargetCommand = New-Object $TargetFactory.CreateCommand();
        $TargetCommand.Connection = $TargetConnection;
        $TargetCommand.CommandTimeout = [Int32]::MaxValue;

foreach($table in $SourceConnection.GetSchema("Tables").Rows){
    try{
            ## Source
                [String]$TabName = $table["TABLE_NAME"];
                [String]$sqlString = "SELECT * FROM [{0}];" -f $TabName;

                $SourceCommand.CommandText = $sqlString;

                [System.Data.Common.DbDataReader]$SourceDataReader = $SourceCommand.ExecuteReader();
                $dtData = New-Object System.Data.DataTable;
                    $dtData.Load($SourceDataReader);

            ## Target -- Bulk Insert data
                if($dtData.Rows.Count -gt 0){
                    $TabName = "[{0}]" -f $TabName;
                    $sqlBulkCopy = New-Object System.Data.SqlClient.SqlBulkCopy($TargetConnection);
                    $sqlBulkCopy.DestinationTableName = $TabName;

                    foreach ($Column in $dtData.Columns){
                        $sqlBulkCopy.ColumnMappings.Add($column, $column)
                    };

                    $sqlBulkCopy.WriteToServer($dtData);
                }
       }catch{
            $table["TABLE_NAME"];
            $_.Exception.Message;
            $_.Exception.ItemName;
        };
};

#Housekeeping
    $sqlBulkCopy.Close();
    $sqlBulkCopy.Dispose();

    $SourceCommand.Dispose();
    $SourceDataReader.Dispose();
    $SourceConnection.Close();
    $SourceConnection.Dispose();

    $TargetCommand.Dispose();
    $TargetDataReader.Dispose();
    $TargetConnection.Close();
    $TargetConnection.Dispose();

    $TargetConnection.Close();
    $TargetConnection.Dispose();
    [System.GC]::Collect();
Mark Kram
  • 5,672
  • 7
  • 51
  • 70
-1

The d/c b/n SSIS And SSDT not quit but SSIS is one of The Components of SSDT, Means that SSDT contains SSIS, SSRS, and SSAS.