crateDB read 10k rows without iteration in Npgsql C# client

Question

I have 10k row data in crate database. How to read the data without iteration. I am using crateDB C# Npgsql client for my service.

var connString = "Host=myserver;Username=mylogin;Password=mypass;Database=mydatabase";

await using var conn = new NpgsqlConnection(connString);
await conn.OpenAsync();
// Retrieve all rows
var cmd = new NpgsqlCommand("select * from sensordata where timestamp >= extract(epoch from (now() - interval '1 week'))", conn);
            var result = new List<SensorDataViewModel>();

           using (var reader = cmd.ExecuteReader())
            {
                    while(reader.HasRows && reader.Read())
                    {
                       SensorDataViewModel item  = new SensorDataViewModel { 
                            sensorid = reader["sensorid"].ToString(),
                            deviceid = reader["deviceid"].ToString(),
                            reading = Convert.ToInt32(reader["reading"]),
                            timestamp = (double)reader["timestamp"]
                        };
                        result.Add(item);
                    }
            }

here im reading each row at a time in while loop. that take lot of time in processing ?

Use a DataAdapter (NpgsqlDataAdapter) and put data into a datatable. Code would look similar to this : https://learn.microsoft.com/en-us/dotnet/api/system.data.sqlclient.sqldataadapter?view=dotnet-plat-ext-5.0 — jdweng, Nov 13 '20 at 15:22

score 1 · Accepted Answer · answered Feb 05 '21 at 20:01

Maybe you need to consider using EntityFrameworkCore. For more details please refer to https://www.npgsql.org/efcore/index.html

Below is my sample code which I have tried. In the CrateDb contain 2 tables. One is customers and another is todo. These table has a relationship where customer.id=todo.customer_id.

The first and second query inside the sample is just select every records from the 2 tables respectively.

The third query is using join to retrieve related records according to the relationship and filtered by customers.id.

    class Program
    {
        static async Task Main(string[] args)
        {
            MemoContext memoContext = new MemoContext();

            //select * from customers
            List<Customer> customers = memoContext.customers.Select(x => new Customer
            {
                id = x.id,
                name = x.name,
                contactno = x.contactno,
                email = x.email
            }).ToList();

            //select * from todo
            List<Todo> todos = memoContext.todo.Select(x => new Todo
            {
                complete = x.complete,
                customer_id = x.customer_id,
                id = x.id,
                title = x.title
            }).ToList();

            //SELECT c.name, c.email, c.contactno, t.title, t.complete 
            //FROM customers AS c 
            //JOIN todo AS t ON t.customer_id = c.id 
            //WHERE c.id=1
            var memo = memoContext.customers.Join(
                    memoContext.todo,
                    c => c.id,
                    t => t.customer_id,
                    (c, t) => new
                    {
                        id = c.id,
                        name = c.name,
                        email = c.email,
                        contactno = c.contactno,
                        todotitle = t.title,
                        complete = t.complete
                    }).Where(n => n.id == 1).ToList();
        }
    }


    class MemoContext : DbContext
    {
        public DbSet<Customer> customers { get; set; }
        public DbSet<Todo> todo { get; set; }

        protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
            => optionsBuilder.UseNpgsql("Host=localhost;Port=5432;Username=crate;SSL Mode=Prefer;Database=doc");
    }

I hope it is useful to you.

crateDB read 10k rows without iteration in Npgsql C# client

1 Answers1