3

I have a REST web-service which is expected to expose a paginated GET call.

For eg: I have a list of students( "Name" , "Age" , "Class" ) in my sql table. And I have to expose a paginated API to get all students given a class. So far so good. Just a typical REST api does the job and pagination can be achieved by the sql query.

Now suppose we have the same requirement just that we need to send back students who are from particular state. This information is hosted by a web-service, S2. S2 has an API which given a list of student names and a state "X" returns the students that belong to state X.

Here is where I'm finding it difficult to support pagination.

eg: I get a request with page_size 10, a class C and a state X which results in 10 students from class C from my db. Now I make a call to S2 with these 10 students and state X, in return, the result may include 0 students, all 10 students, or any number students between 0 and 10 from state 'X'.

How do I support pagination in this case?

Brute force would be to make db calls and S2 calls till the page size is met and then only reply. I don't like this approach .

Is there a common practice followed for this, a general rule of thumb, or is this architecture a bad service design?

(EDIT): Also please tell about managing the offset value. if we go with the some approach and get the result set , how can I manage the offset for next page request ?

Thanks for reading :)

k1133
  • 315
  • 3
  • 12
  • Do you just not like the "brute force" approach because it seems to incur network latency? – Will C Apr 20 '16 at 20:58
  • Yes. Apart from network latency it also results in multiple db call and network(S3) calls. If one of these fail the entire call fails . – k1133 Apr 21 '16 at 02:48
  • Probably more discussion than answering the actual question but it seems that it might be better offhand to have a single service to have enough data to serve the request and effectively not making a request to S2. Why doesn't the current service include the data? – Will C Apr 21 '16 at 12:56
  • 1
    The example given is trivial one. But in case of micro services architecture , there are many services involved depending on bounded context. E g : suppose a service has all customer data and the other service has orders data . Now if the requirement is to have a paginated API which gives customers who have purchased something of value between x and y , we will have the same problem . As the exact amount paid is in orders service and customer info in other service. – k1133 Apr 21 '16 at 13:16
  • Orders and customers data is different from the problem that is posed in the question though. Thats a one to one look up eg. if I wanted to return an order from a customer, I can send a request to the order that represents a customer but if I'm looking up by name, it would look up in the customer microservice and then the orders microservice. In this above question, why can't the data about classes and where the student belongs sit in a single service if the intention of the lookup is look up students based on state and class? Why were they broken out in the first place? – Will C Apr 21 '16 at 14:11
  • I agree , in the above example the data pertaining to student should ideally lie within same service. But the point was in microservice architecture where each service hosts its own data , how can a service aggregate and solve pagination problem i.e provide a paginated api for the data . Please think of the given example as just a dummy example . – k1133 Apr 21 '16 at 14:20
  • I guess I'm leaning towards this being an incomplete design which makes it difficult to reason about. There doesn't seem to be a need for S2. If there was a certain need (again hypothetical here), then you can duplicate the data in both the services so that you can eliminate the need to send a call to S2. – Will C Apr 21 '16 at 14:42

2 Answers2

1

Your service should handle the pagination and not hand it off the SQL. Make these steps:

  1. Get all students from S1 (SQL database) where class = C.
  2. Using the result, get all students from S2 that are in the result and where state = X.
  3. Sort the second result in a stable way.
  4. Get the requested page you want from the sorted result.

All this is done in the code that calls both S1 and S2. Only it has the knowledge to build the pages.

  • 1. Isn't it inappropriate to get all students when only a subset is to be send back as response. Also , in case of very large table , won't til fail. 2. Assuming S2 has exposed a paginated call , how would we make all to S2 will all students in one go ? Still thanks for the answer. – k1133 Apr 21 '16 at 07:13
  • If you want to page accross a result that is a combination of results from S1 *and* S2, you can not leave paging to S1 or S2. You can use the result of S1 to make a *filtered* call to S2. Paging must be done in your service. –  Apr 21 '16 at 07:22
  • I get your point . Thanks. But could you tell how will I manage the next page request . Suppose I get the first page result via this approach , how could I manage the next call with page_number 1. – k1133 Apr 21 '16 at 07:47
1

Not doing the pagination with SQL can lead to performance problems in case of large databases.

Some solution in between can be applied. I assume that the pagination parameters (offset, page size) are configurable for both services, yours and the external one.

You can implement prefetch logic for both services, lets say the prefetch chunk size can be 100.

The frontend can be served with required page size 10.

If the prefetched chunks do not result in a frontend page size 10, the backend should prefetch another chunk till the fronend can be served with 10 students.

This approach require more logic in backend to calculate the next offsets for prefetching, but if you want performance and pagination solved you must invest some effort.

S.Stavreva
  • 577
  • 4
  • 8
  • For the next page , as I am not sure about the student sent in first request , I will need to again do the computation and then send second page . Is there any other way ? – k1133 Apr 21 '16 at 07:48