1

I have a php script that saves in a sqlite database some schedules. It always worked until I noticed nearly two weeks ago that sqlite had a weird behavior. I save the dates in a text field and the records can't be ordered by date.

For example, I have some recent added dates for February and if I run the query to return the dates after August 22nd:

SELECT `date`, hex(`date`), substr(`date`, 1, 1), substr(`date`, 1) 
FROM Schedules WHERE `date` > '2019-08-22' order by `date`

I get the records:

`date`              hex(`date`)                substr(`date`, 1, 1) substr(`date`, 1)
2019-08-22 13:00    323031392D30382D32322031333A3030    2           2019-08-22 13:00
2019-02-04 12:00    323031392D30322D30342031323A3030    2           2019-02-04 12:00
2019-02-18 12:00    323031392D30322D31382031323A3030    2           2019-02-18 12:00
2019-03-04 12:00    323031392D30332D30342031323A3030    2           2019-03-04 12:00

The first record is correct, I included it to have a comparison with the wrong ones. I thought about having some chars before the actual number but from the hex column it doesn't seem so. I even tried to request dates greater than '3000' to exclude the dash (maybe they had some unicode char) and it keep returning the last three records of the previous results.

The table is defined as:

CREATE TABLE `Schedules` (
  `id` INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
  `date` TEXT
);

My php code to add the record is:

class Schedule {
    public static function add_schedule($programmed_date, $programmed_time, $options) {
        $result = 0;
        date_default_timezone_set('Europe/Rome');
        $date = DateTime::createFromFormat('d/m/Y H:i', "$programmed_date $programmed_time");
        $now = new DateTime();
        if ($date < $now)
            return -1;
        $dbh = self::connect_to_database();
        $stmt = $dbh->prepare('INSERT INTO Schedules(`date`) VALUES (:date)');
        $stmt->bindValue(':date', $date->format('Y-m-d H:i'), SQLITE3_TEXT);
        try {
            $execution = $stmt->execute();
            if ($execution !== FALSE)
                $result = 1;
        }
        catch (Exception $ex) {}
        finally {
            $stmt->closeCursor();
        }
        $dbh = null;
        return $result;
    }

    private static function connect_to_database() {
        $db_path = Credentials::read_info('schedule')['db_path'];
        try {
            $dbh = new PDO('sqlite:'.$db_path);
            $dbh->setAttribute(PDO::ATTR_EMULATE_PREPARES, true);
            $dbh->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
        }
        catch(PDOException $e) {
            echo 'Connection failed: '.$e->getMessage();
            exit;
        }
        return $dbh;
    }
}

At this point I can't guess what's wrong with the code. The sql statement should return only the first record instead of four. Any thoughts? Thank you.

SQLite DateTime comparison led me to https://www.sqlite.org/lang_datefunc.html and I found a workaround, if I run the query:

SELECT `date`, hex(`date`), substr(`date`, 1, 1), substr(`date`, 1) 
FROM Schedules WHERE datetime(`date`) > datetime('2019-08-22') order by `date`

It works correctly returning only the record for August 22nd. However I keep wondering why the plain text comparison and order screws up the results when here https://www.db-fiddle.com/f/ar9JxpaCo2wcQB1MobyaVv/0 there is a dump and you clearly see that the first query I wrote here returns a different (and correct) result. If you want to try directly to my database you can download it from here

Ripper346
  • 662
  • 7
  • 22
  • 1
    Possible duplicate of [SQLite DateTime comparison](https://stackoverflow.com/questions/1975737/sqlite-datetime-comparison) – Mihai Matei Jan 23 '19 at 13:48
  • According to a comment in that question you should try something like `WHERE date > '2019-08-22 00:00' order by date` – Mihai Matei Jan 23 '19 at 13:51
  • Please provide a [mcve], i.e. a tailored toydatabase, which allows to reproduce the problem. Use sqlites `.dump` to get it into a shape you can post here. – Yunnosch Jan 23 '19 at 13:53
  • @Yunnosch https://www.db-fiddle.com/f/ar9JxpaCo2wcQB1MobyaVv/0 here the database, but it doesn't reproduce the error because the records aren't added with php – Ripper346 Jan 23 '19 at 15:00
  • @MateiMihai not really a duplicate. I have found from that question a workaround, I changed the where clause to `datetime(date) > datetime('2019-08-22')`. Although the dates as text should be comparable and sortable, so why in this case it doesn't work?! – Ripper346 Jan 23 '19 at 15:01
  • @Ripper346 According to http://www.sqlitetutorial.net/sqlite-date/ If you use the TEXT storage class to store date and time value, you need to use the ISO8601 string format as follows: `YYYY-MM-DD HH:MM:SS.SSS` – Mihai Matei Jan 23 '19 at 15:15
  • @MateiMihai ISO8601 standard define as datetimes `YYYY-MM-DD HH:MM` and `YYYY-MM-DD`, and also sqlite supports them according to official doc https://www.sqlite.org/lang_datefunc.html. Anyway it isn't a matter of support of dates, it is of support of order/compare a string because a string `'3'` doesn't come before `'2'`. It's like that they had different encoding but I checked and there is only one encoding, utf-8 – Ripper346 Jan 23 '19 at 16:01

2 Answers2

2

Well, that was fun. I downloaded the database and played in DB browser for sqlite (read only) until I found something interesting. It started with a fat-finger date > '21' (It found the 3 earlier dates). Finally this sql select *,typeof(date) from Schedules revealed something

349|2019-08-22 13:00|text
374|2019-02-04 12:00|blob
375|2019-02-18 12:00|blob
376|2019-03-04 12:00|blob

I created a repro and found it will insert the date as text if SQLITE3_TEXT is removed from the bindValue.

DinoCoderSaurus
  • 6,110
  • 2
  • 10
  • 15
  • Thank you. So this is probably a php bug. In the doc http://php.net/manual/en/sqlite3stmt.bindvalue.php they explain that `SQLITE3_TEXT: The value is a text string, stored using the database encoding (UTF-8, UTF-16BE or UTF-16-LE). [...] Formerly, if type has been omitted, it has defaulted to SQLITE3_TEXT.` So if we explicit add `SQLITE3_TEXT` it does something that by default it doesn't. – Ripper346 Jan 24 '19 at 09:18
  • 1
    From a briefly check of the source code of php the type is by default `SQLITE3_TEXT`. Also, there are two switch block that uses the same code for both `SQLITE_BLOB` and `SQLITE_TEXT`: `ZVAL_STRINGL(&zargs[i + is_agg], (char*)sqlite3_value_text(argv[i]), sqlite3_value_bytes(argv[i]));`. In a different spot however there's that line for blob and `ZVAL_STRING(data, (char*)sqlite3_column_text(stmt, column));` for text. I'm using php 7.0.33, I will try with the latest build, but I think that if that's the problem it will be also in the latest. If that's the case I will report the bug to php. – Ripper346 Jan 24 '19 at 09:19
1

From @DinoCoderSaurus answer you have the reason why your data is not behaving like it should.
The workaround I found is:

WHERE date || ''  > '2019-08-22'

this way you will get the expected result.
The concatenation with an empty string converts date back to TEXT.

forpas
  • 160,666
  • 10
  • 38
  • 76