How to implement a more efficient search feature?

Question

In my database there are 3 column which is Name, Age, Gender. In the program, I only want to use 1 search button. When the button is clicked, the program determine which 3 of the textbox has input and search for the right data.

How do you work with the query? For example if Name and Gender has text, the query :

"Select * from table Where (Name = @name) AND (Gender = @gender)"

And when only name is entered, I only query for the name. Must I check textbox by textbox whether there is user input and then write multiple query for each of them? Or is there a better way to do this?

Edit (29/5/16) : I tried doing this another way like this

myCommand = New MySqlCommand("Select * from project_record Where
                       (FloatNo = @floatNo OR FloatNo = 'None') AND 
                       (DevCompanyName = @devCompanyName OR DevCompanyName = 'None') AND 
                       (DevType = @devType OR DevType = 'None') AND 
                       (LotPt = @lotPt OR LotPt = 'None') AND
                       (Mukim = @mukim OR Mukim = 'None') AND
                       (Daerah = @daerah OR Daerah = 'None') AND
                       (Negeri = @negeri OR Negeri = 'None') AND
                       (TempReference = @tempRef OR TempReference = 'None')", sqlConn)

But as you can guess already it will not work efficiently as well because if I only enter input for DevType and leave other textboxes blank, the query will not pull up all the records for DevType only. It will just display as no records.

Answer 1

Select * from table 
Where (Name = @name OR @name is Null) 
  AND (Gender = @gender OR @gender is Null)
 ...

it should be one query

Answer 2

Other answers have explained how to simplify the query. It is especially important to get rid of the ORs, since they inhibit any use of indexes.

Once you have the query build cleanly, you need to think about the dataset and decide which columns are usually used for filtering. Then make a few INDEXes for them. You won't be able to provide 'all' possible indexes, hence my admonition that you think about the dataset.

When building indexes, you can have single-column or multiple-column indexes. For your type of data, I would suggest starting with several 2-column indexes. Make sure each index starts with a different column.

For Where (Name = @name) AND (Gender = @gender), here are some notes:

INDEX(gender) is useless because of low 'cardinality';
INDEX(gender, name) might be useful, but the following would be better:
INDEX(name)

Things like name and DevCompanyName are virtually unique, so a 1-column index is probably good.

If you had gender and age, then INDEX(age, gender) might be useful.

MySQL will almost never use two indexes for a single SELECT.

By the way, the construction of the WHERE could be done in a Stored Procedure. You would need CONCAT, PREPARE, etc.

Answer 3

Original answer

(scroll down to see update)

Can you try the following:

• build a list only including values of the textboxes that have an input
• set a string of the join the items of that list together with the " AND " string
• append that string to your standard SELECT statement

The code looks like this:

Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click

    Dim Predicate1 As String = Me.TextBox1.Text
    Dim Predicate2 As String = Me.TextBox2.Text
    Dim Predicate3 As String = Me.TextBox3.Text
    Dim PredicateList As New List(Of String)
    Dim WhereClause As String
    Dim Query As String

    If Predicate1 <> String.Empty Then
        PredicateList.Add("Name=""" & Predicate1 & """")
    End If
    If Predicate2 <> String.Empty Then
        PredicateList.Add("Age=""" & Predicate2 & """")
    End If
    If Predicate3 <> String.Empty Then
        PredicateList.Add("Gender=""" & Predicate3 & """")
    End If

    WhereClause = String.Join(" AND ", PredicateList.ToArray)
    Query = "SELECT * FROM TABLE WHERE " & WhereClause
    MessageBox.Show(Query)

End Sub

Update

Further to the comments re SQL injection, here is an updated sample.

Dim Command As SqlClient.SqlCommand
Dim Predicate1 As String = Me.TextBox1.Text
Dim Predicate2 As String = Me.TextBox2.Text
Dim Predicate3 As String = Me.TextBox2.Text
Dim ParameterList As New List(Of SqlClient.SqlParameter)
Dim PredicateList As New List(Of String)
Dim BaseQuery As String = "SELECT * FROM TABLE WHERE "

If Predicate1 <> String.Empty Then
    PredicateList.Add("name = @name")
    ParameterList.Add(New SqlClient.SqlParameter("@name", Predicate1))
End If
If Predicate2 <> String.Empty Then
    PredicateList.Add("age = @age")
    ParameterList.Add(New SqlClient.SqlParameter("@age", Predicate2))
End If
If Predicate3 <> String.Empty Then
    PredicateList.Add("gender = @gender")
    ParameterList.Add(New SqlClient.SqlParameter("@gender", Predicate3))
End If

Command = New SqlClient.SqlCommand(BaseQuery & String.Join(" AND ", PredicateList.ToArray))
Command.Parameters.AddRange(ParameterList.ToArray)

Answer 4

COALESCE is your friend here. You can use it to make the where clause ignore comparisons where the parameter is NULL.

Select * from table Where (Name = COALESCE(@name,table.Name))
AND (Gender = COALESCE(@gender,table.Gender))

So, if the @name parameter is NULL, COALESCE(@name,table.Name) will return the value of the 'Name' column of the current row and (Name = COALESCE(@name,table.Name)) will always be true.

This assumes that if no value is entered in a textbox the corresponding parameter will be NULL. If instead it is a value such as 'None', you can use the NULLIF function to map 'None' to NULL

Select * from table Where 
(Name = COALESCE( NULLIF( @name, 'None'), table.Name)) 
AND (Gender = COALESCE( NULLIF( @gender, 'None'), table.Gender))

Answer 5

How to implement a more efficient search?

The answer partly depends on what your definition of efficient is. I suspect you mean less code and fewer if blocks etc. But fundamentally, running a new SELECT * query to apply a filter is inefficient because your base data set can be all the rows and you just fiddle with the users View of it.

I have a DB with random data in columns for Fish, Color (string), Bird, Group (int) and Active which should be similar enough for Name, Age and Gender in the question - or that other long thing at the bottom.

DataTable

Fill a datatable and bind it to a DGV:

' form level object
Private dtSample As DataTable
...
' elsewhere
Dim sql = "SELECT  Id, Name, Descr, `Group`, Fish, Bird, Color, Active FROM Sample"
Using dbcon As MySqlConnection = New MySqlConnection(MySQLConnStr)
    ' create SELECT command with the Query and conn
    Dim cmd As New MySqlCommand(sql, dbcon)
    ...
    daSample.Fill(dtSample)
    daSample.FillSchema(dtSimple, SchemaType.Source)
End Using
dgv2.DataSource = dtSample

Going forward, we can filter the user's view of that table without issuing a new query.

Filter Controls

If some of the fields are limited to certain selections, for instance Gender, you can use a ComboBox instead of a TextBox. This is to help the user succeed and avoid typos (Make or Mael instead of Male; or here, correctly spelling Baracuda I mean Baraccuda, er Barracuda correctly.

For illustration purposes, Fish is something where the user can type in anything at all, but Bird is constrained to a set of choices. If there is a Bird table, cboBird can be bound or populated from it. But you may also be able to populate it from the master/base table:

Dim birds = dtSample.AsEnumerable.Where(Function(d) d.IsNull(5) = False).
            Select(Function(d) d.Field(Of String)("Bird")).
            Distinct.
            ToArray()
cboBird.Items.AddRange(birds)

If "Finch" is a legal choice but there are none in the database, it wont show in the list. Depending on the app, this can be a Good Thing:

• If the user filters on Finch and there a no resulting records, you won't need a MessageBox or StatusBar message explaining the empty result set.
• If something is not in the list, you are signalling up front that there are none of those. It then becomes a matter of training why a known element isnt in the list.
• On the other hand, you'd have to repopulate those filter controls each time before they are used in case new records were added recently. If the controls are on a Dialog or different TabPage, this is easy to do as needed.
• It isnt always applicable, but it can help the user avoid typos.

It depends on the app whether either method is of value.

DBNull / 'none'

I am not sure why you are adding 'none' to each clause. If someone want to see all the 'John` or all the 'Cod' records, it doesn't seem like they would also be interested in 'none'. Personally, Null/DBNull seems a better way to handle this, but it is easy to add or not add either form.

It would seem more valuable to filter to just those with DBNull/None. The code above for the Bird List filters out DBNull and I would do so for none as well. Then, before the result is added to the ComboBox, add a `None' item first so it is at the top.

Again it depends on what the app does; Or = 'None', may make perfect sense in this case.

Filter

Using a TextBox for Fish and Group, a ComboBox for Bird and Color and a CheckBox for Active, the code can form the filter thusly:

Dim filterTerms As New List(Of String)
Dim filterFmt = "{0} = '{1}' "
'   OR:
' Dim filterFmt = "{0} = '{1}' OR {0} Is Null"
'   OR:
' Dim filterFmt = "{0} = '{1}' OR {0} = 'none'"
    
If String.IsNullOrEmpty(tbSearchFish.Text) = False Then
    Dim txt = tbSearchFish.Text.Replace("'", "''")
    filterTerms.Add(String.Format(filterFmt, "Fish", txt))
End If

If cboBird.SelectedIndex > -1 Then
    filterTerms.Add(String.Format(filterFmt, "Bird", cboBird.SelectedItem.ToString))
End If

If String.IsNullOrEmpty(tbGroup.Text) = False Then
    Dim n As Int32
    If Int32.TryParse(tbGroup.Text, n) Then
        filterTerms.Add(String.Format(filterFmt, "[Group]", n))
    End If
End If

If cboColor.SelectedIndex > -1 Then
    filterTerms.Add(String.Format(filterFmt, "Color", cboColor.SelectedItem.ToString))
End If

If chkActive.Checked Then
    ' NOTE: I do not have TreatTinyAsBoolean turned on
    '  for some reason
    filterTerms.Add(String.Format(filterFmt, "Active", "1"))
End If

If filterTerms.Count > 0 Then
    Dim filter = String.Join(" AND ", filterTerms)
    dtSample.DefaultView.RowFilter = filter

    Dim rows = dtSample.DefaultView.Count
End If

• Use whichever filterFmt is appropriate for what the app needs to do
• A filter term is only added to the list if the related control has a value (as per above, this could include a 'None').
• For the TextBox, it escapes any embedded ticks such as might be found in names like O'Malley or D'Artgnan. It replaces one tick with two.
• Since Group is a numeric, a valid Int32 input is tested
• If there are elements in the filterTerms list, a filter string is created
• The filter is applied to the DefaultView.Filter (you can use also use a DataView or a BindingSource) so that the code need not query the database to provide filter capabilities.
• Rows will tell you how many rows are in the current View.

The only halfway tricky one is a Boolean like Gender or Active because those actually resolve to three choices: {Any/Either, A, B}. For that, I would use a ComboBox and ignore it for SelectedIndex 0 as well. I didn't bother with this because the Combo concept is amply covered. Result:

Is it More "Efficient"?

It still depends.
It doesn't re-query the database to get rows the app can already have.
No new DBConnection, DBCommand or other DBProvider objects are created, just a list.
No need to dynamically create a SQL statement with N parameters in a loop to avoid SQL injection/special words and chars.
It doesn't even query the database for the items for the filter terms. If there is a static list of them in the DB, they could be loaded once, the first time they use the filters.
It is easy to remove the filter, no need to query yet again without WHERE clauses.
A ComboBox where applicable helps the user find what they want and avoid typos.
Is the SQL "cleaner". more "efficient? The code doesn't really mess with new SQL, just some WHERE clauses.

Is there less code? I have no idea since we just see the result. It doesnt string me as a lot of code to do what it does.

Answer 6

In my database there are 3 column which is Name, Age, Gender. In the program, I only want to use 1 search button. When the button is clicked, the program determine which 3 of the textbox has input and search for the right data.

And when only name is entered, I only query for the name. Must I check textbox by textbox whether there is user input and then write multiple query for each of them? Or is there a better way to do this?

SELECT * FROM `table`
WHERE (`name` = @name AND `name` IS NOT NULL)
OR (`age` = @age AND (`age`>0 OR `age` IS NOT NULL))
OR (`gender` = @gender AND `gender` IS NOT NULL);

With the above query if all text boxes have value, the result will not be one record (as if you where using logical AND between fields). If you want only that record you will filter it server-side with php from the rest of the results.

You can check the results on your own in this Fiddle

EDIT

In order to solve the above inconvenience (not bringing easily single results when needed) i got a little help from this answer and re-wrote the above query as:

SELECT *, IF(`name`=@name, 10, 0) + IF(`age`=@age, 10, 0) + IF(`gender`=@gender, 10, 0) AS `weight` 
FROM `table` 
WHERE (`name` = @name AND `name` IS NOT NULL) 
OR (`age` = @age AND (`age`>0 OR `age` IS NOT NULL)) 
OR (`gender` = @gender AND `gender` IS NOT NULL) 
HAVING `weight`=30;

OR to still get all records with a weight on result

SELECT *, IF(`name`=@name, 10, 0) + IF(`age`=@age, 10, 0) + IF(`gender`=@gender, 10, 0) AS `weight` 
FROM `table` WHERE (`name` = @name AND `name` IS NOT NULL) 
OR (`age` = @age AND (`age`>0 OR `age` IS NOT NULL)) 
OR (`gender` = @gender AND `gender` IS NOT NULL) 
ORDER BY `weight` DESC;

Answer 7

You were pretty close. Let's look at

(FloatNo = @floatNo OR FloatNo = 'None')

So you want the field either to be the given input or 'None'? But there are (supposedly) no records in your table with FloatNo 'None'. What you really want to do is find out whether the input is none (i.e. empty):

(FloatNo = @floatNo OR @floatNo = '')

And for the case the user types in a blank by mistake, you can ignore this, too:

(FloatNo = @floatNo OR TRIM(@floatNo) = '')

The whole thing:

myCommand = New MySqlCommand(
  "Select * from project_record Where
         (FloatNo = @floatNo OR TRIM(@floatNo) = '') AND 
         (DevCompanyName = @devCompanyName OR TRIM(@devCompanyName) = '') AND 
         (DevType = @devType OR TRIM(@devType) = '') AND 
         (LotPt = @lotPt OR TRIM(@lotPt) = '') AND
         (Mukim = @mukim OR TRIM(@mukim) = '') AND
         (Daerah = @daerah OR TRIM(@daerah) = '') AND
         (Negeri = @negeri OR TRIM(@negeri) = '') AND
         (TempReference = @tempRef OR TRIM(@tempRef) = '')", sqlConn)

Answer 8

What is wrong with your approach?

Just change (FloatNo = @floatNo OR FloatNo = 'None')
to
(FloatNo = @floatNo OR FloatNo = '' or FloatNo IS NULL)

And do that for every criteria.
Your query will respect empty values and NULL values after that.