How to automatically break down a SQL-like query with many joins into discrete, independent steps?

Question

Note: This is a learning exercise to learn how to implement a SQL-like relational database. This is just one thin slice of a question in the overall grand vision.

I have the following query, given a test database with a few hundred records:

select distinct "companies"."name"
from "companies"
inner join "projects" on "projects"."company_id" = "companies"."id"
inner join "posts" on "posts"."project_id" = "projects"."id"
inner join "comments" on "comments"."post_id" = "posts"."id"
inner join "addresses" on "addresses"."company_id" = "companies"."id"
where "addresses"."name" = 'Address Foo'
and "comments"."message" = 'Comment 3/3/2/1';

Here, the query is kind of unrealistic, but it demonstrates the point which I am trying to make. The point is to have a query with a few joins, so that I can figure out how to write this in sequential steps.

The first part of the question is (which I think I've partially figured out), is how do you write these joins as a sequence of independent steps, with the output of one fed into the input of the other? Also, is there more than one way to do it?

// step 1
let companies = select('companies')
// step 2
let projects = join(companies, select('projects'), 'id', 'company_id')
// step 3
let posts = join(projects, select('posts'), 'id', 'project_id')
// step 4
let comments = join(posts, select('comments'), 'id', 'post_id')
// step 5
let finalPosts = posts.filter(post => !!comments.find(comment => comment.post_id === post.id))
// step 6
let finalProjects = projects.filter(project => !!posts.find(post => post.project_id === project.id))

// step 7, could also be run in parallel to step 2 potentially
let addresses = join(companies, select('addresses'), 'id', 'company_id')

// step 8
let finalCompanies = companies.filter(company => {
  return !!posts.find(post => post.company_id === company.id)
    && !!addresses.find(address => address.company_id === company.id)
})

These filters could probably be more optimized using indexes of some sort, but that is beside the point I think. This just demonstrates that there seem to be about 8 steps to find the companies we are looking for.

The main question is, how do you automatically figure out the steps from the SQL query?

I am not asking about how to parse the SQL query into an AST. Assume we have some sort of object structure we are dealing with, like an AST, to start.

How would you have to have the SQL query in structured object form, such that it would lead to these 8 steps? I would like to be able to specify a query (using a custom JSON-like syntax, not SQL), and then have it divide the query into these steps to divide and conquer so to speak and perform the queries in parts (for learning how to implement distributed databases). But I don't see how we go from SQL-like syntax, to 8 steps. Can you show how that might be done?

Here is the full code for the demo, which you can run with psql postgres -f test.sql. The result should be "Company 3".

Basically looking for a high level algorithm (doesn't even need to be code), which describes the key way you would break down some sort of AST-like object representation of a SQL query, into the actual planned steps of the query.

My algorithm looks like this in my head:

1. represent SQL query in object tree.
2. convert object tree to steps.

I am not really sure what (1) should be structured like, and even if we had some sort of structure, I'm not sure how to get that to complete (2). Looking for more details on the implementations of these steps, mainly step (2).

My "object structure" for step 1 would be something like this:

const query = {
  select: {
    distinct: true,
    columns: ['companies.name'],
    from: ['companies'],
  },
  joins: [
    {
      type: 'inner join',
      table: 'projects',
      left: 'projects.company_id',
      right: 'companies.id',
    },
    ...
  ],
  conditions: [
    {
      left: 'addresses.name',
      op: '=',
      right: 'Address Foo'
    },
    ...
  ]
}

I am not sure how useful that is, but it doesn't relate to steps at all. At a high level, what kind of code would I have to write to convert that object sort of structure into steps? Seems like one potential avenue is do a topological sort on the joins. But then you need to combine that with the select and conditions somehow, not sure how you would even begin to programmatically know what step should be before what other step, or even what the steps are. Maybe if I somehow could break it into known "chunks", then it would be simple to apply TOP sort to it after that, but then the question is still, how to get into chunks from the object structure / SQL?

Basically, I have been reading about the theory behind "query planning and optimization", but don't know how to apply it in this regard. How did this site do it?

One aspect is breaking at least the where conditions into CNF.

Answer 1

Implementing joins is a huge topic which is probably out of scope for a StackOverflow answer.

If you're looking for practical information about how joins are implemented, I would suggest...

• The Join Operation section of Use The Index, Luke for different types of join implementation.
• Section 7 of the The SQLite Query Optimizer Overview covers joins. And reading the SQLite source code. It is about as small a practical SQL implementation will get.
• The output of explain in Postgresql gives very detailed information about how it has implemented the query. And they are explained in Operator Optimization Information