0

I'm trying to dump data from Firebird SQL 2.5 database with little or no success, because I'm trying to make a join between a table and stored procedure. The code representing the problematic query is following:

select
    U.SYMBOL
    , '0' AS FIRM_ID
    , U.DATA_OD AS DATA
    , IIF(EXTRACT(DAY FROM U.DATA_OD) = 1 AND EXTRACT(MONTH FROM U.DATA_OD) = 1, 'NR', 'ZW') AS PRZYCZYNA_ZMIANY_WYM
    , PS.WYMIAR AS WYMIAR
    , PS.ZALEGLY AS WYM_ZAL
    , ' ' AS WYMIAR_INW
    , ' ' AS WYMIAR_INW_ZAL
    , ' ' AS NAZWA_ABSENCJI
from 
    P_PRA_WYM_URL U
    left JOIN PS_WYK_URLOPOW(U.SYMBOL, U.DATA_OD, U.DATA_OD) ps ON 1=1
where
    U.SYMBOL IN (SELECT SYMBOL FROM P_PRACOWNIK)

The PS_WYK_URLOPOW stored procedure accepts three parameters: employee number, and two dates. All parameters are members of the P_PRA_WYM_URL table, so the stored procedure needs to be executed once for every record in that table. The are more than 40 000 records in that table.

I'm executing this query using following C# code:

if (AreSettingsPopulated())
            {
                string sFilepath = dSettings[FILEPATHKEY];
                string sUsername = dSettings[USERNAMEKEY];
                string sPassword = Encoding.UTF8.GetString(System.Convert.FromBase64String(dSettings[DBPASSWORDKEY]));

                string connectionString =
                                    "User={0};" +
                                    "Password={1};" +
                                    "Database={2};" +
                                    "DataSource=localhost;" +
                                    "Port=3050;" +
                                    "Dialect=1;" +
                                    "Charset=NONE;" +
                                    "Role=;" +
                                    "Connection lifetime=15;" +
                                    "Pooling=true;" +
                                    "MinPoolSize=0;" +
                                    "MaxPoolSize=50;" +
                                    "Packet Size=8192;" +
                                    "ServerType=0";

                FbConnection fbConn = null;

                try
                {
                    string sSqlQuery = "";

                    using (FileStream fs = new FileStream(string.Format("{0}.sql", dSettings[TABLENAMEKEY]), FileMode.Open))
                    {
                        using (StreamReader sr = new StreamReader(fs, Encoding.UTF8))
                        {
                            if (!sr.EndOfStream)
                                sSqlQuery = sr.ReadToEnd();
                            sr.Close();
                        }
                        fs.Close();
                    }

                    if (sSqlQuery != "")
                    {
                        DataTable dtDumpResults = new DataTable(dSettings[TABLENAMEKEY]);

                        fbConn = new FbConnection(string.Format(connectionString, sUsername, sPassword, sFilepath));
                        FbCommand fbcDumpCommand = new FbCommand(sSqlQuery, fbConn);
                        fbcDumpCommand.CommandTimeout = 90000000;

                        if (fbConn.State != System.Data.ConnectionState.Open)
                            fbConn.Open();

                        using (FbDataAdapter fbdaDumpCommand = new FbDataAdapter(sSqlQuery, fbConn))
                        {
                            this.engine.WriteLine("Dumping the contents from database...");
                            fbdaDumpCommand.Fill(dtDumpResults);
                            this.engine.WriteLine("Dump complete!");
                        }

                        if (dtDumpResults.Rows.Count > 0)
                        {
                            GenerateExportXlsFile(dtDumpResults, dSettings[TABLENAMEKEY]);
                        }
                        else
                            throw new ArgumentNullException("The result set returned null!");
                    }
                    else
                        throw new ArgumentNullException("The SQL query is empty!");
                }
                catch (Exception ex)
                {
                    return string.Format("Error while invoking plugin: {0}", ex.ToString());
                }
                finally
                {
                    if (fbConn != null && fbConn.State == System.Data.ConnectionState.Open)
                        fbConn.Close();
                }
            }
            else
                return "Execution failed! The settings are not populated!";

            return string.Format("Export of table {0} complete!", dSettings[TABLENAMEKEY]);
        }

If I try to dump the data with query shown above with Flamerobin, the pre-fetching mechanism works and loading the data every 500 records I'm able to get the complete dump, while with C# application the execution stops on fbdaDumpCommand.Fill(dtDumpResults); step.

The question is: how to optimize the query to make it work inside C# application?

Arioch 'The
  • 15,799
  • 35
  • 62
  • is `PS_WYK_URLOPOW` a *pure function* ? I mean, is it return value totally determined by input parameters and nothing else, so it is 100% repeatable? If so it might make sense to create `COMPUTED BY` column on the `P_PRA_WYM_URL` table and get rid of the JOIN altogether. Also - show the SP source and show the query *plan*, FR should show it. Also, is your SP `selectable` or not ? – Arioch 'The Sep 16 '19 at 09:32
  • If you would replace `fbdaDumpCommand.Fill(dtDumpResults);` with manual loop over the rows, just dumping column text representations into console log and skipping to `next` row - will THAT work? I think the issue is not in Firebird or the query, but either in `dtDumpResults` or in `.Net Provider` – Arioch 'The Sep 16 '19 at 09:45
  • @Arioch'The Exactly: if I try to run a loop in C# over all records and dump data for each employee one by one I get the results pretty quickly. – Krzysztof Jakóbczyk Sep 16 '19 at 10:33
  • @Arioch'The I did not mention that I cannot alter the database structure, just read from it. The SP is `selectable` and it is a *pure function*. – Krzysztof Jakóbczyk Sep 16 '19 at 10:36
  • then perhaps you can escalate the issue to those who can. Is this SP used with different tables or only with this one? You can either totally get rid of it or encapsulate its call into `COMPUTED BY` column declaration, thus getting rid of this ugly `left join`. – Arioch 'The Sep 16 '19 at 10:41
  • As a very stupid (and random) hack, try to wrap your query into `select * from ( your-query-here )` trying to flatten it all and to hide the very fact that you did a join ever. You wrote your program freezes because you are doing the join - so try to pretend you do not. I don't really think it should be triggered by join and thus this SQL wrapping should not make much difference, but who knows, why not just trying it at random? – Arioch 'The Sep 16 '19 at 11:18
  • Since you can loop in C# easily but can not `.Fill`, the issue is not about Firebird server or the query, it is somewhere inside C#, where my knowledge approaches zero. So, one avenue would be to dig into `.Fill` requirements - quite maybe that there are gotchas about the target container, maybe you have to prepare it (pro-actively allocate space, tune into block-insert mode from read-only mode or anything implementation-specific). That, if you missed something C#-generic. Or maybe you need to consult Jiri, .Net Provider author. He rarely visits SO though, dwelling in some Firebird maillists – Arioch 'The Sep 16 '19 at 11:28
  • https://groups.google.com/forum/#!forum/firebird-net-provider – Arioch 'The Sep 16 '19 at 12:00

0 Answers0