Server Side Pagination in Nodejs MongoDB (Atlas) RestAPIs Example – Express + Mongoose Tutorial (Filtering + Sorting)

Server side Pagination in Nodejs with MongoDB Example

Facing with a large dataset is one of the most user-case in software development, and we need to present it to the user in smaller chunks. To resolve the problem, pagination and sorting is a way to handle it. So in the tutorial, I guide step by step with clearly code to implement a “Server Side Pagination in Nodejs MongoDB Example” with Express RestAPIs, Mongoose CRUD queries with MongoDB Atlas database to provide a set of RestAPIs for pagination filtering and sorting.

– I draw a fullstack overview Diagram Architecture from Restclient requests to MongoDB database through Nodejs RestAPI backend.
– Develop Nodejs Pagination Filtering and Sorting RestAPIs with the supporting of Express Rest Framework and Mongoose ODM.
– I create an testsuite with a number of testcase’s requests from RestClient to MongoDB Atlas through Nodejs RestAPIs Server.

Related posts:


Overview Architecture with Fullstack Diagram – Server Side Pagination in Nodejs MongoDB Example

To handling Nodejs Pagination RestAPI and do Filtering and Sorting queries with MongoDB (Atlas) database, we create a web backend Node.js application with 4 main points:

Nodejs Express MongoDB CRUD RestApis
Nodejs Express MongoDB CRUD RestApis
  • To handle RestAPI requests with Node.js, we use Express.js framework.
  • To do pagination operations with MongoDB database, we use Mongoose ODM queries.
  • We define all Nodejs RestAPI urls in router.js.
  • We implement how to process each RestAPI requests in controller.js file.

Project Goal with Testsuite: Nodejs MongoDB Pagination Rest

We define a set of 3 RestAPIs to do the pagination, filtering and sorting queries with MongoDB database as below list:

  1. Filtering Request – a GET RestAPI /api/customers/filteringbyage is used to fetch all records from MySQL with a filtering by age
  2. Pagination Request – a GET RestAPI /api/customers/pagination is used to fetch data from MySQL with pagination purpose.
  3. Pagination Filtering and Sorting – a GET RestAPI /api/customers/pagefiltersort is defined to fetch data from MySQL with pagination, filtering by age and ordering by 2 fields firstname and lastname

Testcase 1 – Nodejs MongoDB Filtering Request

Nodejs MongoDB Filtering RestAPIs
Nodejs MongoDB Filtering RestAPIs

The request is used to filter all Customer records having age=23. It returns a JSON message with 3 fields:

  • message – an overall descriptive message for filtering request
  • totalItems – total number of returned Customer items after filtering
  • customers – an array of all filtering Customer items

Check the MongoDB database again:

MongoDB Filtering by age=23
MongoDB Filtering by age=23

Testcase 2 – Nodejs MongoDB Pagination Request

Nodejs MongoDB Pagination RestAPI
Nodejs MongoDB Pagination RestAPI

What does it mean? We had done a pagination request to fetch a second page page = 1 with a size of page is 7 (limit=7)

The RestAPI returns a Json result with useful informantion as below:

  1. totalItems describes a total number of matching records in database for the pagination request
  2. totalPages describes the total number of pages matching with requested limit
  3. limit describes the number of items for each fetching page
  4. currentPageNumber is the order number of requested page (currentPageNumber = page + 1)
  5. currentPageSize is the size of the current page (currentPageSize <= limit)
  6. customers is a set of returned items attached with the pagination response

- Check with MongoDB query using skip and limit:

MongoDB Pagination with skip and limit
MongoDB Pagination with skip and limit

Testcase 3 - Nodejs MongoDB Pagination Filtering and Sorting request

Nodejs Pagination Filtering and Sorting
Nodejs Pagination Filtering and Sorting

What does it mean? - The above request had done with 3 proccessing steps:

1. Do the Filtering with age=23, and We just have 12 Customer items in database having age is 23 so returned totalItems is 12. The totalPages is 3 because of 2 reason:
- limit: 5
- and totalPages = Math.ceil(data.count / limit) = Math.ceil(12 / 5)

2. Do the pagination with offset = 5 (limit*page)

3. Finally do the Sorting by firstname with ascending order and lastname with descending order:

MongoDB Pagination Filtering Sorting
MongoDB Pagination Filtering Sorting

How to Create Nodejs MongoDB Pagination Project

Before creating a Nodejs project, we need to confirm that the Nodejs and npm had been installed in your computer development by cmd: node -v and npm -v

Check Nodejs Environment For Development
Check Nodejs Environment For Development

If these commandlines are not recognized by command prompt, it means you need to install them by visit the https://nodejs.org/en/ site and download installed package and do the nodejs setup for development later.

Now starting development! Create a folder and named it as Nodejs-Express-MongoDB-Example, go inside the folder, open a cmd and initiate a Nodejs project by cmd npm init. After all, a package.json file is created as below content:


{
  "name": "nodejs-mongodb-pagination",
  "version": "1.0.0",
  "description": "server side pagination in nodejs mongodb",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "repository": {
    "type": "git",
    "url": "https://github.com/loizenjava"
  },
  "keywords": [
    "nodejs",
    "mongodb",
    "pagination"
  ],
  "author": "https://loizenjava.com",
  "license": "ISC",
  "dependencies": {
    "body-parse": "^0.1.0",
    "express": "^4.17.1",
    "mongoose": "^5.10.3"
  }
}

Here is the project structure of Nodejs MongoDB CRUD Example:

Nodejs MongoDB Pagination Project Structure
Nodejs MongoDB Pagination Project Structure

- mongodb.config.js is used to define MongoDB database's URL configuration
- customer.model.js is used to define a Mongoose model mapped with corresponding MongoDB database document schema.
- router.js is used to define all Nodejs Express RestAPI urls.
- controller.js is used to implement detail logic code to process each incoming request.
- server.js is used to implement a Nodejs Web server.

Install Nodejs Dependencies: Express, Body Parse, Mongoose

To development a 'Node.js MongoDB Pagination Example with Mongoose and Express RestAPIs', we need a set of packages to handle the full stack of the web backend proccessing, they includes Express framework, Body Parse, Mongoose packages.

  1. Express is a minimal and flexible Node.js web application framework that provides a robust set of features for web and mobile applications.
    $ npm install express
    
  2. Body-parser is the Node.js body parsing middleware. It is responsible for parsing incoming request bodies in a middleware before your handlers, available under the req.body property.
    $ npm install body-parser
    
  3. Mongoose is a promise-based Node.js ODM for MongoDB.
    $ npm install --save mongoose

We can install all the packages by one cmd:

$npm install --save express body-parser mongoose

Create Nodejs Mongoose Model

In the tutorial, 'Node.js Express MongoDB RestAPIs Example', We need define a Mongoose ODM model to represent a document in the customer collection, see details of coding in file customer.model.js:

const mongoose = require('mongoose');
 
const CustomerSchema = mongoose.Schema({
    firstname: String,
    lastname: String,
    address: String,
    age: { 
      type: Number, 
      min: 18, 
      max: 65, 
      required: true 
    },
    copyrightby: {
      type: String,
      default: 'https://loizenjava.com'
    }
});

module.exports = mongoose.model('Customer', CustomerSchema);

See the mapping document in MongoDB:

MongoDB document
MongoDB document

Steps to Build Node.js Express RestAPI Example

For building the Nodejs Express RestAPIs Example project with MongoDB, we do 4 step for development:

  1. Create Express WEB Application Server
  2. Define All RestAPI URLs in router.js
  3. Configure MongoDB Database with Mongoose ODM
  4. Implement All RESTAPIs in controllers.js file

Create Express Application Server with MongoDB Connection

To implement an Express RestAPIs Application, firstly we need create a server with express app:

- server.js:

const express = require('express');
const app = express();
...
const server = app.listen(8080, function () {
 
  let host = server.address().address
  let port = server.address().port
 
  console.log("App listening at http://%s:%s", host, port); 
})

- For parsing body of requests, we need use body-parser dependency, add more code on server.js file:


...
var bodyParser = require('body-parser');
...
app.use(bodyParser.json());
...
const server = app.listen(8080, function () {
...
>

- To connect with MongoDB, we use below segment code:

mongoose.connect(dbConfig.url, { useNewUrlParser: true, useUnifiedTopology: true })
    .then(() => {
        console.log("Successfully connected to MongoDB.");    
    }).catch(err => {
        console.log('Could not connect to MongoDB.');
        process.exit();
    });

Here is all coding in the server.js file:

var express = require('express');
var app = express();
var bodyParser = require('body-parser');
app.use(bodyParser.json())

// Configuring the database
const dbConfig = require('./app/config/mongodb.config.js');
const mongoose = require('mongoose');
 
mongoose.Promise = global.Promise;
 
// Connecting to the database
mongoose.connect(dbConfig.url, { useNewUrlParser: true, useUnifiedTopology: true })
    .then(() => {
        console.log("Successfully connected to MongoDB.");    
    }).catch(err => {
        console.log('Could not connect to MongoDB.');
        process.exit();
    });

require('./app/routes/customer.router.js')(app);
// Create a Server
var server = app.listen(8080, function () { 
  var host = server.address().address
  var port = server.address().port
 
  console.log("App listening at http://%s:%s", host, port) 
})

Define All Nodejs Express RestAPIs URLs in router.js

For making a routing with Express framework, we define all URLs in customer.router.js file:


module.exports = function(app) {

  var customers = require('../controllers/customer.controller.js');

  // Create a new Customer
  app.post('/api/customer/create', customers.create);

  app.get('/api/customer/filteringbyage', customers.filteringByAge);

  app.get('/api/customer/pagination', customers.pagination);

  app.get('/api/customer/pagefiltersort', customers.paginationfilterandsort);
}

Configure MongoDB Atlas Database with URL

We create a file mongodb.config.js as below:

module.exports = {
  url: 'mongodb+srv://loizenjava:loizenjava@cluster0.esvi3.mongodb.net/loizenjavadb'
}

Implement All RESTAPIs in controllers.js file

Nodejs Controller RestAPIs
Nodejs Controller RestAPIs

For handling Nodejs MongoDB CRUD RestAPIs' processing that been defined in router.js file, we implement all working functions in controller.js file:

  1. filteringByAge = (req, res) function is used to handle a GET request at the endpoint /api/customer/filteringbyage to filter all documents in MongoDB by a given age value and return back all results to client in a Json message format
  2. pagination = async (req, res) function is used to handle a GET request at the endpoint /api/customer/pagination with 2 query parameters limit and page to get a batch of document from MongoDB database like as a pagination query
  3. pagingfilteringsorting = async (req, res) function is used to handle a GET request at the endpoint /api/customers/pagefiltersort to do an association quering operation: pagination, plus filtering by age and sorting by firstname and lastname from MongoDB database with 3 request query parameters limit, page, age

Implement Filtering RestApi in Nodejs MongoDB

Now we create an filtering restapi to get data from MongoDB database with some condition by Nodejs Express and MongoDB Queries:

filteringByAge = (req, res) function is used to handle a GET request at the endpoint /api/customer/filteringbyage to filter all document in MongoDB by a given age value and return back all results to client in a Json message format.

How to implement it? Just follow simple steps:

  • Get an age value from request query for using to do the filter later
  • Using the .find({}) method of Mongoose model to retrieve all document from MongoDB database with a filter by age condition: {age: age}
  • Build a Json message with informative selected fields and return it back to client side.
  • Do not forget to handle any exception when processing by using try-catch statement!

Coding:


exports.filteringByAge = (req, res) => {
  const age = parseInt(req.query.age);  

  Customer.find({age:age}).select("-__v")
    .then(results => {
      res.status(200).json({
        "message": "Get all Customers with age = " + age,
        "size": results.length,
        "customers": results
      });  
    }).catch(err => {
      console.log(err);
      res.status(500).json({
        message: "Error!",
        error: err
      });
    });
}

Implement Pagination RestAPI in Nodejs MongoDB

pagination = (req, res) function is used to handle a GET request at the endpoint /api/customer/pagination with 2 query parameters limit and page to get a batch of documents from MongoDB database for the pagination query.

To do the pagination with MongoDB database by Mongoose ODM, we use the .find({}) method with skip and limit.

Simple steps to implement the pagination request:

  • Get the page and limit parameters from the request query
  • Calulate an offset(skip) from page and limit parameters
    const offset = page ? page * limit : 0;
  • Do the pagination query to MongDB database using the find({}) method of Mongoose model as blow format:
    let results = await Customer.find({})  // You may want to add a query
                      .skip(offset) // Always apply 'skip' before 'limit'
                      .limit(limit)
                      .select("-__v"); // This is your 'page size'
  • Construct a JSON message with a set of informative fields and return back to client side
  • Remember to handle the un-expected error by using try-catch statement

Detail Coding:

exports.pagination = async (req, res) => {

  try {

    const page = parseInt(req.query.page);
    const limit = parseInt(req.query.limit); // Make sure to parse the limit to number
    
    const offset = page ? page * limit : 0;
  
    // We are using the '3 layer' architecture explored on the 'bulletproof node.js architecture'
    // Basically, it's just a class where we have our business logic
    
    let results = await Customer.find({})  // You may want to add a query
                      .skip(offset) // Always apply 'skip' before 'limit'
                      .limit(limit)
                      .select("-__v"); // This is your 'page size'
        
    let numOfCustomer = await Customer.countDocuments({});
  
    res.status(200).json({
      "message": "Paginating is completed! Query parameters: page = " + page + ", limit = " + limit,
      "totalPages": Math.ceil(numOfCustomer / limit),
      "totalItems": numOfCustomer,
      "limit": limit,
      "currentPageSize": results.length,
      "customers": results
    });      
  } catch (error) {
    res.status(500).send({
      message: "Error -> Can NOT complete a paging request!",
      error: error.message,
    });    
  }

}

- PS: we can use Mongoose aggregate([...]) function to implement Nodejs MongoDB pagination function:


let results = await Customer.aggregate([
                      { $match: {} },    // This is your query
                      { $skip: offset },   // Always apply 'skip' before 'limit'
                      { $limit: limit }, // This is your 'page size'
                  ]).select("-__v")  

Implement Pagination Filtering Sorting RestAPI in Nodejs MongoDB

paginationfilterandsort = (req, res) function is used to handle a GET request at the endpoint /api/customer/pagefiltersort to do an association quering operation: pagination, plus filtering by age and sorting by firstname and lastname from MongoDB database with 3 request query parameters limit, page, age.

We continue to use the method .find({}) to do the associated operation: pagination plus filtering and sorting query. So we add more conditions with .find({}) method:

  • filtering condition: {age: age} - .find({age:age})
  • sorting clause – using to sort documents by 2 fields firstname with ascending direction and lastname with descending direction:
    .sort({"firstname": 1, "lastname": -1})

Straightforward steps to implement the Pagination + Filtering and Sorting function:

  • Retrieve 3 parameters from incoming request query: limit, page for pagination and age for filtering.
  • Calulate an offset for Mongoose pagining query later:
    const offset = page ? page * limit : 0;
  • Do the Mongoose pagination filtering and sorting with .find({}) method:
    let results = await Customer.find({age: age})  // You may want to add a query
                      .skip(offset) // Always apply 'skip' before 'limit'
                      .limit(limit)
                      .sort({"firstname": 1, "lastname": -1})
                      .select("-__v"); // This is your 'page size'
  • Construct a JSON message with informative fields and return back to client side:
    res.status(200).json({
      "message": "Paginating is completed! Query parameters: page = " + page + ", limit = " + limit,
      "totalPages": Math.ceil(numOfCustomer / limit),
      "totalItems": numOfCustomer,
      "limit": limit,
      "age-filtering": age,
      "currentPageSize": results.length,
      "customers": results
    });    
  • Do NOT forget to handle an un-expected errors with try-catch statement.
    try{
        //...
    }catch(error) {
        res.status(500).send({
          message: "Error -> Can NOT complete a paging request!",
          error: error.message,
        });
    }   

– Coding here:

exports.paginationfilterandsort = async (req, res) => {
  try {
    const page = parseInt(req.query.page);
    const limit = parseInt(req.query.limit); // Make sure to parse the limit to number
    const age = parseInt(req.query.age);
    
    const offset = page ? page * limit : 0;
    
    let results = await Customer.find({age: age})  // You may want to add a query
                      .skip(offset) // Always apply 'skip' before 'limit'
                      .limit(limit)
                      .sort({"firstname": 1, "lastname": -1})
                      .select("-__v"); // This is your 'page size'
        
    let numOfCustomer = await Customer.countDocuments({age: age});

    res.status(200).json({
      "message": "Paginating is completed! Query parameters: page = " + page + ", limit = " + limit,
      "totalPages": Math.ceil(numOfCustomer / limit),
      "totalItems": numOfCustomer,
      "limit": limit,
      "age-filtering": age,
      "currentPageSize": results.length,
      "customers": results
    });    
  } catch (error) {
    res.status(500).send({
      message: "Error -> Can NOT complete a paging + filtering + sorting request!",
      error: error.message,
    });
  }
};

Sourcecode

– Here is a full sourcecode for “Nodejs MongoDB Pagination Example”:

Nodejs-MongoDB-Pagination-Example

– Github Sourcecode:

Nodejs MongoDB Pagination Example

Further Reading

Related posts:


2 thoughts on “Server Side Pagination in Nodejs MongoDB (Atlas) RestAPIs Example – Express + Mongoose Tutorial (Filtering + Sorting)”

Leave a Reply

Your email address will not be published. Required fields are marked *