1

I plan to have a application which uses Sqlite databases as data files. Because different files can be opended more often, I want to cache the connections.

I'm very new to Rust; this is my first project... My problem is: Somewhen I run out of file handles, and I cannot create new database files.

What I tried so far:

  • test1(), will only work, if I implement Drop for MyPool. Drop will close the connection-pool. By doing this, I'm sure the file handles gets free again.
  • test2(), is the async version which I would need for my project (it will be a Rocket app). Here I'm not successful at all.

If you run the code, you would have to delete all db.* files afterwards.

// Cargo.toml
// tokio = { version = "1", features = ["rt-multi-thread", "macros" ] }
// futures = "0.3"
// sqlx = { version = "0.5", features = [ "runtime-tokio-native-tls", "sqlite", "migrate" ] }

use sqlx::{migrate::MigrateDatabase, sqlite::SqlitePoolOptions, Pool, Sqlite};
use futures::executor::block_on;
use std::sync::{Arc, Mutex};

#[derive(Clone)]
struct MyPool(Pool<Sqlite>);

impl Drop for MyPool {
    fn drop(&mut self) {
        println!("**** drop");
        block_on(
            self.0.close()
        );
    }
}

#[tokio::main]
async fn main() {
    test1().await;
    //test2().await;
}

async fn test1() {
    let mut pool: Vec<MyPool> = Vec::new();

    for i in 1..1000 {

        let db_name = format!("./db.{}.db", i);

        Sqlite::create_database(&db_name)
            .await.expect(format!("create {} failed", i).as_str());

        let conn = SqlitePoolOptions::new()
            .max_connections(5)
            .connect(&db_name).await.expect(format!("connect {} failed", i).as_str());

        if pool.len() == 10 {
            println!("Clenup");
            pool.clear();
        }

        println!("{}", i);
        pool.push(MyPool(conn));
    }
}

async fn test2() {
    let pool: Arc<Mutex<Vec<MyPool>>> = Arc::new(Mutex::new(Vec::new()));

    let tasks: Vec<_> = (0..1000)
    .map(|i| {
        let my_pool = pool.clone();

        tokio::spawn(async move {

            let db_name = format!("./db.{}.db", i);

            Sqlite::create_database(&db_name)
                .await.expect(format!("create {} failed", i).as_str());

            let conn = SqlitePoolOptions::new()
                .max_connections(5)
                .connect(&db_name).await.expect(format!("connect {} failed", i).as_str());

            {
                let mut locked_pool = my_pool.lock().expect("locked");
                if locked_pool.len() == 10 {
                    println!("Clenup");
                    locked_pool.clear();
                }

                println!("{}", i);
                locked_pool.push(MyPool(conn));
            }

        })
    }).collect();

    // Wait for all tasks to complete.
    futures::future::join_all(tasks).await;
}
macalloy
  • 43
  • 7
  • 1
    In the async version, all 1000 of your tasks start running *immediately* *, so there's the opportunity for all 1000 to be created before the pool has a chance to drop them. – kmdreko Apr 08 '21 at 20:37
  • I've implemented this already in a Rocket app. When I create 1000 curl requests I see the same problem. In this case there should be some delay and not all of them should run at the same time. But I'm sure it's a multithreading issue... – macalloy Apr 09 '21 at 05:41
  • When I lock the Mutex, isn’t this the signal for the other tasks, to wait? This means the 11th task will clean the cache and will be the first in the cache afterwards. So theoretically the should have no problem. No, wait... before the mutex is locked, the connection is already open. The same is true for all other threads. In the next step the Cache would be freed, but this is already too late. This means I would have to lock the mutex already earlier? – macalloy Apr 09 '21 at 20:29
  • Tried now to move the `let conn = SqlitePoolOptions::new()` part inside the block, where the mutex is locked. But this won't work because `std::marker::Send` is not implemented for mutex. How to solve this? – macalloy Apr 11 '21 at 07:08

0 Answers0