1

I am trying to repeatedly update some values in C++ based on telemetry coming from a lightstreamer server. This telemetry is real-time data from the International Space Station (ISSLive), that I will be using to articulate a VR simulation model of the same. I have the VR simulation side of this covered; I just need help with the data stream. Here's a public gist of the code: https://gist.github.com/connorjak/9f853bb83559976e12c9e38811c3c0af

Working Starting Point

The starting point of this project is the ISS-Mimic project. Here is their (working) architecture:

(Running on Raspberry Pi)

  1. Python database_initialize.py removes then reinitializes a SQLite3 database in /dev/shm/iss_telemetry.db
  2. Python GUI.py establishes a SQLite3 connection to the same directory, applies isolation_level = None to the connection object
  3. At some point in GUI.py, a Python subprocess is opened for ISS_Telemetry.js
  4. NodeJS ISS_Telemetry.js establishes a SQLite3 connection to the same directory, with this line: var db = new sqlite3.Database("/dev/shm/iss_telemetry.db", sqlite3.OPEN_CREATE | sqlite3.OPEN_READWRITE);
  5. ISS_Telemetry.js establishes a LightstreamerClient connection to the lightstreamer server, with setSlowingEnabled(false)
  6. ISS_Telemetry.js listens on the Lightstreamer connection and updates the SQLite database with new data.
  7. GUI.py goes on to repeatedly execute select queries and fetchall() to get the entirety of the telemetry every tick.

Non-Working Current Architecture

Now, here is my architecture, with key differences in bold.

(Running on Windows 10 1903, WSL with Ubuntu 18.04 LTS)

  1. C++ ISSLIVE.cpp calls system(...) to start and wait for Python database_initialize.py
  2. Python database_initialize.py removes then reinitializes a SQLite3 database in /dev/shm/iss_telemetry.db
  3. C++ ISSLIVE.cpp establishes a SQLite3 connection to the same directory, does not apply isolation_level = None to the connection object (can't find equivalent functionality in C API)
  4. ISSLIVE.cppcalls system(...&) to start and not wait for NodeJS ISS_Telemetry.js
  5. NodeJS ISS_Telemetry.js establishes a SQLite3 connection to the same directory, with this line: var db = new sqlite3.Database("/dev/shm/iss_telemetry.db", sqlite3.OPEN_CREATE | sqlite3.OPEN_READWRITE);
  6. ISS_Telemetry.js establishes a LightstreamerClient connection to the lightstreamer server, with setSlowingEnabled(false)
  7. ISS_Telemetry.js listens on the Lightstreamer connection and updates the SQLite database with new data.
  8. ISSLIVE.cpp goes on to repeatedly execute select queries (sqlite3_exec(...)) to get the entirety of the telemetry every tick. The queries are passed with a callback to print all received table data to cout.

The Problem

Instead of going to the callback function and printing any data at all, sqlite3_exec(...) returns error code 15: PROTOCOL (Locking Protocol error). This happens after exactly ten seconds.

The Question

What should I do to debug this? I tried valgrind callgrind to pick out computation offenders, but it appears that whatever is happening is not compute-heavy (very low compute time spent in this code).

I could also try to replace this architecture with another. Should I? What would I use? I can't change the source server of this data. I had a false-start earlier with cpp-lsclient, a C++ implementation of Lightstreamer. That repo appears to only have a partial implementation of lightstreamer.

More Info

  • isolation_level = None should relate to the autocommit mode in SQLite. autocommit mode is on currently in the C++ code; I checked.
  • database_initialize.py finishes (and closes its connection) before anything else pertaining to the database runs
  • ISS_Telemetry.js closes when the C++ application closes
  • ISS_Telemetry.js appears to be working correctly; data is flowing in properly. It's possible there's still an issue in this code.
  • Python 3.6.8
  • NodeJS v12.13.1
  • npm 6.12.1
  • Javascript node-sqlite3 4.1.1 (actual sqlite library version ~3.30)
  • C++ sqlite3 3.22.0
  • lightstreamer-client 7.3.1
  • C++14 std
  • C++ code operating inside a pthread as a dynamically linked library, alongside a highly-multithreaded simulation platform
CourageousPotato
  • 199
  • 7
  • Try to execute `PRAGMA read_uncommitted = boolean;` ([docs](https://www.sqlite.org/isolation.html)) on connection as analog of `isolation_level = None`. P.S. It looks an weird architecture: the periodic run of query to get changes is no good. – Aikon Mogwai Dec 05 '19 at 18:22
  • @AikonMogwai Thanks! Looks like that at least changed my problem; I'll work on it some more and report back. – CourageousPotato Dec 06 '19 at 00:51
  • I added a github gist with my code now (https://gist.github.com/connorjak/9f853bb83559976e12c9e38811c3c0af) – CourageousPotato Dec 06 '19 at 05:45
  • I don't understand what is a reason to use C++ with sqlite? If you need to get a new data, just put in to console in Node.js and read stdout stream in C++ code. – Aikon Mogwai Dec 06 '19 at 06:15
  • I’m using SQLite at the moment with C++ because the NodeJS code is known-good. Your idea with other methods of passing from NodeJS to C++ is interesting though; I’ll try some things. – CourageousPotato Dec 06 '19 at 06:18
  • I ended up just replacing SQLite with a custom socket connection. Details are in the gist. – CourageousPotato Dec 09 '19 at 23:48
  • And is the problem solved? – Aikon Mogwai Dec 11 '19 at 17:04
  • @AikonMogwai Yes, the JavaScript code works fine, it was the C++ interface with SQLite that was the troublemaker. The new solution works. – CourageousPotato Dec 11 '19 at 20:26

0 Answers0